The advanced performance of depth estimation is achieved by the employment of large and complex neural networks. While the performance is still being continuously improved, we argue that the depth estimation has to be efficient as well since it is a preliminary requirement for real-world applications. However, fast depth estimation tends to lower the performance as the trade-off between the model's capacity and accuracy. In this paper, we aim to achieve accurate depth estimation with a light-weight network. To this end, we first introduce a highly compact network that can estimate a depth map in real-time. We then develop a knowledge distillation paradigm to further improve the performance. We observe that many scenarios have the same scene scales in real-world, yielding similar depth histograms, thus they are potentially valuable and applicable to develop a better learning strategy. Therefore, we propose to employ auxiliary unlabeled/labeled data to improve knowledge distillation. Through extensive and rigorous experiments, we show that our method can achieve comparable performance against state-of-the-of-art methods with only 1% parameters, and outperforms previous light-weight methods in terms of inference accuracy, computational efficiency and generalizability.