Deep learning provides a powerful new approach to many computer vision tasks. Height prediction from aerial images is one of those tasks that benefited greatly from the deployment of deep learning which replaced old multi-view geometry techniques. This letter proposes a two-stage approach, where first a multi-task neural network is used to predict the height map resulting from a single RGB aerial input image. We also include a second refinement step, where a denoising autoencoder is used to produce higher quality height maps. Experiments on two publicly available datasets show that our method is capable of producing state-of-the-art results. Code is available at https://github.com/melhousni/DSMNet.