The ongoing global pandemic of Coronavirus Disease 2019 (COVID-19) has posed serious threat to public health and the economy. Rapid and accurate diagnosis of COVID-19 is crucial to prevent the further spread of the disease and reduce its mortality. Chest computed tomography (CT) is an effective tool for the early diagnosis of lung diseases including pneumonia. However, detecting COVID-19 from CT is demanding and prone to human errors as some early-stage patients may have negative findings on images. In this study, we propose a novel residual network to automatically identify COVID-19 from other common pneumonia and normal people using CT images. Specifically, we employ the modified 3D ResNet18 as the backbone network, which is equipped with both channel-wise attention (CA) and depth-wise attention (DA) modules to further improve the diagnostic performance. Experimental results on the large open-source dataset show that our method can differentiate COVID-19 from the other two classes with 94.7% accuracy, 93.73% sensitivity, 98.28% specificity, 95.26% F1-score, and an area under the receiver operating characteristic curve (AUC) of 0.99, outperforming baseline methods. These results demonstrate that the proposed method could potentially assist the clinicians in performing a quick diagnosis to fight COVID-19.