Diabetes impacts the quality of life of millions of people. However, diabetes diagnosis is still an arduous process, given that the disease develops and gets treated outside the clinic. The emergence of wearable medical sensors (WMSs) and machine learning points to a way forward to address this challenge. WMSs enable a continuous mechanism to collect and analyze physiological signals. However, disease diagnosis based on WMS data and its effective deployment on resource-constrained edge devices remain challenging due to inefficient feature extraction and vast computation cost. In this work, we propose a framework called DiabDeep that combines efficient neural networks (called DiabNNs) with WMSs for pervasive diabetes diagnosis. DiabDeep bypasses the feature extraction stage and acts directly on WMS data. It enables both an (i) accurate inference on the server, e.g., a desktop, and (ii) efficient inference on an edge device, e.g., a smartphone, based on varying design goals and resource budgets. On the server, we stack sparsely connected layers to deliver high accuracy. On the edge, we use a hidden-layer long short-term memory based recurrent layer to cut down on computation and storage. At the core of DiabDeep lies a grow-and-prune training flow: it leverages gradient-based growth and magnitude-based pruning algorithms to learn both weights and connections for DiabNNs. We demonstrate the effectiveness of DiabDeep through analyzing data from 52 participants. For server (edge) side inference, we achieve a 96.3% (95.3%) accuracy in classifying diabetics against healthy individuals, and a 95.7% (94.6%) accuracy in distinguishing among type-1/type-2 diabetic, and healthy individuals. Against conventional baselines, DiabNNs achieve higher accuracy, while reducing the model size (FLOPs) by up to 454.5x (8.9x). Therefore, the system can be viewed as pervasive and efficient, yet very accurate.