We present the first learning-based framework for category-level 3D object detection and implicit shape estimation based on a pair of stereo RGB images in the wild. Previous stereo 3D object detection approaches cannot describe the complete shape details of the detected objects and often fails for the small objects. In contrast, we propose a new progressive approach that can (1) perform precise localization as well as provide a complete and resolution-agnostic shape description for the detected objects and (2) produce significantly more accurate orientation predictions for the tiny instances. This approach features a new instance-level network that explicitly models the unseen surface hallucination problem using point-based representations and uses a new geometric representation for orientation refinement. Extensive experiments show that our approach achieves state-of-the-art performance using various metrics on the KITTI benchmark. Code and pre-trained models will be available at this https URL.