In video data, busy motion details from moving regions are conveyed within a specific frequency bandwidth in the frequency domain. Meanwhile, the rest of the frequencies of video data are encoded with quiet information containing substantial redundancy, which causes low efficiency for video models that take as input raw RGB frames. In this paper, we consider that busy and quiet spatio-temporal regions require different computational resources. We design a trainable Motion Band-Pass Module (MBPM) for separating busy information from quiet information in raw video data. By representing the quiet information with lower resolution, we can increase the efficiency of video data processing. By embedding the MBPM into a two-pathway CNN architecture, we define a Busy-Quiet Net (BQN). The efficiency of BQN is determined by avoiding redundancy in the feature space processed by the two pathways: one operates on quiet features of low-resolution, while the other operates on busy features. The proposed BQN outperforms many recent video processing models on Something-Something V1, Kinetics400, UCF101 and HMDB51.