Monocular 3D reconstruction is to reconstruct the shape of object and its other information from a single RGB image. In 3D reconstruction, polygon mesh, with detailed surface information and low computational cost, is the most prevalent expression form obtained from deep learning models. However, the state-of-the-art schemes fail to directly generate well-structured meshes, and most of meshes have two severe problems Vertices Clustering (VC) and Illegal Twist (IT). By diving into the mesh deformation process, we pinpoint that the inappropriate usage of Chamfer Distance (CD) loss is the root causes of VC and IT problems in the training of deep learning model. In this paper, we initially demonstrate these two problems induced by CD loss with visual examples and quantitative analyses. Then, we propose a fine-grained reconstruction method CD^2 by employing Chamfer distance twice to perform a plausible and adaptive deformation. Extensive experiments on two 3D datasets and comparisons with five latest schemes demonstrate that our CD^2 directly generates well-structured meshes and outperforms others by alleviating VC and IT problems.