Prostate cancer (PCa) is the second deadliest form of cancer in males. PCa severity can be clinically graded by examining the structural representations of Gleason tissues. The paper proposes a framework for segmenting Gleason tissues and grading PCa using Whole Slide Images (WSI). Our approach encompasses two main contributions: 1) An asymmetric dilated residual segmentation model integrating a novel hierarchical decomposition scheme to extract textured Gleason tissues. 2) A three-tiered loss function to ensure accurate recognition of the cluttered regions in the cancerous tissues. The proposed framework has been extensively evaluated on a large-scale PCa dataset containing 10,516 whole slide scans (with around 71.7M patches), where it outperforms state-of-the-art schemes in several metrics for extracting the Gleason tissues and grading the progression of PCa.