Prostate cancer (PCa) is the second deadliest form of cancer in males. PCa severity can be clinically graded by examining the structural representations of Gleason tissues. This paper presents an asymmetric encoder-decoder that integrates a novel hierarchical decomposition block to exploit the feature representations pooled across various scales and then fuses them together to extract the Gleason tissue patterns from the patched whole slide images. Furthermore, the proposed network is penalized through the three-tiered loss function, which ensures that it accurately recognizes the cluttered regions of the cancerous tissues (according to their severity grade), despite having similar contextual and textural characteristics, leading towards robust grading of PCa progression. The proposed framework has been extensively evaluated on a large-scale PCa dataset containing 10,516 whole slide scans (with around 71.7M patches), where it outperforms state-of-the-art schemes in several metrics for extracting the Gleason tissues and grading the progression of PCa.