Peripancreatic vessel segmentation and anatomical labeling play extremely important roles to assist the early diagnosis, surgery planning and prognosis for patients with pancreatic tumors. However, most current techniques cannot achieve satisfactory segmentation performance for peripancreatic veins and usually make predictions with poor integrity and connectivity. Besides, unsupervised labeling algorithms cannot deal with complex anatomical variation while fully supervised methods require a large number of voxel-wise annotations for training, which is very labor-intensive and time-consuming. To address these problems, we propose our Automated Peripancreatic vEssel Segmentation and lAbeling (APESA) framework, to not only highly improve the segmentation performance for peripancreatic veins, but also efficiently identify the peripancreatic artery branches. There are two core modules in our proposed APESA framework: iterative trunk growth module (ITGM) for vein segmentation and weakly supervised labeling mechanism (WSLM) for artery branch identification. Our proposed ITGM is composed of a series of trunk growth modules, each of which chooses the most reliable trunk of a basic vessel prediction by the largest connected constraint, and seeks for the possible growth branches by branch proposal network. Our designed iterative process guides the raw trunk to be more complete and fully connected. Our proposed WSLM consists of an unsupervised rule-based preprocessing for generating pseudo branch annotations, and an anatomical labeling network to learn the branch distribution voxel by voxel. We achieve Dice of 94.01% for vein segmentation on our collected dataset, which boosts the accuracy by nearly 10% compared with the state-of-the-art methods. Additionally, we also achieve Dice of 97.01% on segmentation and competitive performance on anatomical labeling for peripancreatic arteries.