Light field (LF) depth estimation plays a key role in many LF-based applications. Existing LF depth estimation methods generally consider depth estimation as a regression problem, supervised by a pixel-wise L1 loss between the regressed disparity map and the groundtruth one. However, the disparity map is only a sub-space projection (i.e., an expectation) of the disparity distribution, which is essential for models to learn. In this paper, we propose a simple yet effective method to learn the sub-pixel disparity distribution by fully utilizing the power of deep networks, especially for LF of narrow baselines. We construct the cost volume at the sub-pixel level to produce a finer disparity distribution and design an uncertainty-aware focal loss to supervise the predicted disparity distribution to be close to the groundtruth one. Extensive experimental results demonstrate the effectiveness of our method. Our method ranks the first place over 105 submitted algorithms on the HCI 4D LF Benchmark in terms of all the four accuracy metrics (i.e., BadPix 0.01, BadPix 0.03, BadPix 0.07, and MSE). The Code and model are available at https://github.com/chaowentao/SubFocal.