Object detection in aerial images is a fundamental research task in the domain of geoscience and remote sensing. However, the advanced progress on this topic mainly focuses on designing progressive backbone architectures or head networks but ignores the neck network. In this letter, we first analyze the importance of the neck network in object detection from the perspective of information bottleneck. Then, to alleviate the information deficiency problem in the current neck networks, we propose a Global Semantic Network (GSNet), which acts as a bridge from the backbone to the head network in a bidirectional global pattern. Compared to the existing neck networks, our model can capture rich and detailed image features with less computational costs. Besides, we further propose a feature Fusion Refinement Module (FRM) for different levels of feature maps, which are suffering from a big information gap. To demonstrate the effectiveness and efficiency of our approach, experiments are carried out on two challenging datasets (i.e., DOTA and HRSC2016). Experimental results in terms of recognition accuracy and computational complexity validate the superiority of our method. The code has been open-sourced at GSNet.