Facial age estimation is an important and challenging problem in computer vision. Existing approaches usually employ deep neural networks (DNNs) to fit the mapping from facial features to age, even though there exist some noisy and confusing samples. We argue that it is more desirable to distinguish noisy and confusing facial images from regular ones, and alleviate the interference arising from them. To this end, we propose self-paced deep regression forests (SP-DRFs) -- a gradual learning DNNs framework for age estimation. As the model is learned gradually, from simplicity to complexity, it tends to emphasize more on reliable samples and avoid bad local minima. Moreover, the proposed capped-likelihood function helps to exclude noisy samples in training, rendering our SP-DRFs significantly more robust. We demonstrate the efficacy of SP-DRFs on Morph II and FG-NET datasets, where our model achieves state-of-the-art performance.