Self-attention and channel attention, modelling thesemantic interdependencies in spatial and channel dimensionsrespectively, have recently been widely used for semantic seg-mentation. However, computing spatial-attention and channelattention separately and then fusing them directly can causeconflicting feature representations. In this paper, we proposethe Channelized Axial Attention (CAA) to seamlessly integratechannel attention and axial attention with reduced computationalcomplexity. After computing axial attention maps, we propose tochannelize the intermediate results obtained from the transposeddot-product so that the channel importance of each axial repre-sentation is optimized across the whole receptive field. We furtherdevelop grouped vectorization, which allows our model to be runwith very little memory consumption at a speed comparableto the full vectorization. Comparative experiments conductedon multiple benchmark datasets, including Cityscapes, PASCALContext and COCO-Stuff, demonstrate that our CAA not onlyrequires much less computation resources compared with otherdual attention models such as DANet, but also outperformsthe state-of-the-art ResNet-101-based segmentation models on alltested datasets.