Quick and accurate diagnosis is of paramount importance to mitigate the effects of COVID-19 infection, particularly for severe cases. Enormous effort has been put towards developing deep learning methods to classify and detect COVID-19 infections from chest radiography images. However, recently some questions have been raised surrounding the clinical viability and effectiveness of such methods. In this work, we investigate the impact of multi-task learning (classification and segmentation) on the ability of CNNs to differentiate between various appearances of COVID-19 infections in the lung. We also employ self-supervised pre-training approaches, namely MoCo and inpainting-CXR, to eliminate the dependence on expensive ground truth annotations for COVID-19 classification. Finally, we conduct a critical evaluation of the models to assess their deploy-readiness and provide insights into the difficulties of fine-grained COVID-19 multi-class classification from chest X-rays.