We propose spectral analysis to investigate the correlation between the accuracy and the resolution of segmentation maps for semantic segmentation. The current networks predict segmentation maps on the down-sampled grid of images to alleviate the computational cost. Moreover, these networks can be trained by weak annotations that utilize only the coarse contour of segmentation maps. Despite the successful achievement of these works utilizing the low-frequency information of segmentation maps, however, the accuracy of resultant segmentation maps may also be degraded in the regions near object boundaries. It is yet unclear for a theoretical guideline to determine an optimal down-sampled grid to strike the balance between the cost and the accuracy of segmentation. We analyze the objective function (cross-entropy) and network back-propagation process in frequency domain. We discover that cross-entropy and key features of CNN are mainly contributed by the low-frequency components of segmentation maps. This further provides us quantitative results to determine the efficacy of down-sampled grid of segmentation maps. The analysis is then validated on the two applications: the feature truncation method and the block-wise annotation that limit the high-frequency components of the CNN features and annotation, respectively. The results agree with our analysis. Thus the success of the existing work utilizing low-frequency information of segmentation maps now has theoretical foundation.