Capillaries are the smallest vessels in the body responsible for the delivery of oxygen and nutrients to the surrounding cells. Various diseases have been shown to alter the density of nutritive capillaries and the flow velocity of erythrocytes. In previous studies, capillary density and flow velocity have been assessed manually by trained specialists. Manual analysis of a standard 20-second long microvascular video takes on average 20 minutes and requires extensive training. Several studies have reported that manual analysis hinders the application of microvascular microscopy in a clinical setting. In this paper, we present a fully automated state-of-the-art system, called CapillaryNet, that can quantify skin nutritive capillary density and red blood cell velocity from handheld microscopy videos. Moreover, CapillaryNet measures several novel microvascular parameters that researchers were previously unable to quantify, i.e. capillary hematocrit and Intra-capillary flow velocity heterogeneity. Our system has been used to analyze skin microcirculation videos from various patient groups (COVID-19, pancreatitis, and acute heart diseases). Our proposed system excels from existing capillary detection systems as it combines the speed of traditional computer vision algorithms and the accuracy of convolutional neural networks.