Voxel-based 3D object classification has been thoroughly studied in recent years. Most previous methods convert the classic 2D convolution into a 3D form that will be further applied to objects with binary voxel representation for classification. However, the binary voxel representation is not very effective for 3D convolution in many cases. In this paper, we propose a hybrid cascade architecture for voxel-based 3D object classification. It consists of three stages composed of fully connected and convolutional layers, dealing with easy, moderate, and hard 3D models respectively. Both accuracy and speed can be balanced in our proposed method. By giving each voxel a signed distance value, an obvious gain regarding the accuracy can be observed. Besides, the mean inference time can be speeded up hugely compared with the state-of-the-art point cloud and voxel based methods.