Both radiology and nucleic acid test (NAT) have their pros and cons for assessment of COVID-19. Here we present a 3D CNN-based multitask learning (MTL) framework, termed COVID-MTL, which is capable of simultaneously detecting COVID-19 against both radiology and NAT as well as assessing infection severity. A real-time 3D augmentation algorithm (Shift3D) was proposed to introduce space variances by shifting low-level feature representations of volumetric inputs in three dimensions, which boosted the convergence and accuracy of state-of-the-art 3D CNNs. A random-weighted loss was proposed to assign learning weights to different COVID-19 tasks under Dirichlet distribution, which prevented task dominance and improved joint performance. By only using CT data, COVID-MTL was trained on 930 CT scans and tested on another 399 cases, which yielded AUCs of 0.939 and 0.846, and accuracies of 90.23% and 79.20% for detection of COVID-19 against radiology and NAT, respectively, and outperformed state-of-the-art models. COVID-MTL yielded AUC of 0.800 ± 0.020 and 0.813 ± 0.021 (with transfer learning) for classifying control/suspected, mild/regular, and severe/critically-ill cases. To decipher the recognition mechanism, we identified high-throughput lung features, which are significantly related (P < 0.001) to the positivity and severity of COVID-19.