We present the first learning-based framework for category-level 3D object detection and implicit shape estimation based on a pair of stereo RGB images in the wild. Traditional stereo 3D object detection approaches describe the detected objects only with 3D bounding boxes and cannot infer their full surface geometry, which makes creating a realistic outdoor immersive experience difficult. In contrast, we propose a new model S-3D-RCNN that can perform precise localization as well as provide a complete and resolution-agnostic shape description for the detected objects. We first decouple the estimation of object coordinate systems from shape reconstruction using a global-local framework. We then propose a new instance-level network that addresses the unseen surface hallucination problem by extracting point-based representations from stereo region-of-interests, and infers implicit shape codes with predicted complete surface geometry. Extensive experiments validate our approach's superior performance using existing and new metrics on the KITTI benchmark. Code and pre-trained models will be available at this https URL.