Recent research on the application of remote sensing and deep learning-based analysis in precision agriculture demonstrated a potential for improved crop management and reduced environmental impacts of agricultural production. Despite the promising results, the practical relevance of these technologies for actual field deployment requires novel algorithms that are customized for analysis of agricultural images and robust to implementation on natural field imagery. The paper presents an approach for analyzing aerial images of a potato crop using deep neural networks. The main objective is to demonstrate automated spatial recognition of a healthy versus stressed crop at a plant level. Specifically, we examine premature plant senescence resulting in drought stress on Russet Burbank potato plants. The proposed deep learning model, named Retina-UNet-Ag, is a variant of Retina-UNet (Jaeger et al., 2018) and includes connections from low-level semantic dense representation maps to the feature pyramid network. The paper also introduces a dataset of field images acquired with a Parrot Sequoia camera carried by a Solo unmanned aerial vehicle. Experimental validation demonstrated the ability for distinguishing healthy and stressed plants in field images, achieving an average Dice score coefficient of 0.74. A comparison to related state-of-the-art deep learning models for object detection revealed that the presented approach is effective for the task at hand. The method applied here is conducive toward the assessment and recognition of potato crop stress (early plant senescence resulting from drought stress in this case) in natural aerial field images collected under real conditions.