The outbreak of COVID-19 has resulted in over 67 million infections with over 1.5 million deaths worldwide so far. Both computer tomography (CT) diagnosis and nucleic acid test (NAT) have their pros and cons. Here we present a multitask-learning (MTL) framework, termed COVID-MTL, which is capable of simultaneously detecting COVID-19 against both radiology and NAT as well as assessing infection severity. We proposed an active-contour based method to refine lung segmentation results on COVID-19 CT scans and a Shift3D real-time 3D augmentation algorithm to improve the convergence and accuracy of state-of-the-art 3D CNNs. A random-weighted multitask loss function was then proposed, which made simultaneous learning of different COVID-19 tasks more stable and accurate. By only using CT data and extracting lung imaging features, COVID-MTL was trained on 930 CT scans and tested on another 399 cases, which yielded AUCs of 0.939 and 0.846, and accuracies of 90.23% and 79.20% for detection of COVID-19 against radiology and NAT, respectively, and outperformed state-of-the-art models. COVID-MTL yielded AUC of 0.800 ± 0.020 and 0.813 ± 0.021 (with transfer learning) for classifying control/suspected (AUC of 0.841), mild/regular (AUC of 0.808), and severe/critically-ill (AUC of 0.789) cases. Besides, we identified top imaging biomarkers that are significantly related (P < 0.001) to the positivity and severity of COVID-19.