This paper focuses on multi-person 3D mesh recovery from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline, that detects bounding boxes and then regresses the body from bounding-box-level features. However, multi-person occlusion and truncation can make these features ambiguous, which results in the failure of recovery. To deal with this problem, we present a novel bottom-up single-shot method, named "Center-based Human Mesh Recovery network (CenterHMR)". The key idea is to develop an explicit center-based representation for bottom-up pixel-level estimation. Guided by the body centers, our model effectively locates every person and learns robust and discriminative features under occlusion. In an end-to-end manner, the model is trained to estimate multiple differentiable maps that contain the information of multi-person 3D body meshes and their locations.Furthermore, when encountering severe multi-person occlusion, the body centers may be very close or even overlapping. A collision-aware center representation is developed to ensure a distinguishable distance between body centers. Our proposed CenterHMR achieves state-of-the-art performance on four challenging multi-person/occlusion benchmarks (3DPW, CMU Panoptic, MuPoTs-3D, and 3DOH50K). Experiments on crowded/occluded datasets demonstrate the stability under various types of occlusion. Due to the concise bottom-up single-shot design, our released demo code (https://github.com/Arthur151/CenterHMR) is the first open-source real-time (over 30 FPS) implementation of monocular multi-person 3D mesh recovery.