The goal of active learning algorithms is to judiciously select subsets of unlabeled samples to be labeled by an oracle, in order to reduce the time and cost associated with supervised learning. Previously, active learning techniques for deep neural networks have used the same network for the task at hand (e.g., classification) as well as sample selection, which can be conflicting goals. To address this issue, we use a separate sample scoring network to capture the relevant information about the distribution of the labeled samples, and use it to assess the novelty of unlabeled samples. Specifically, we propose to efficiently train the scoring network using a self-supervised learning (pretext) task on the labeled samples. To make the scoring network more robust, we added to it another head, which is trained using the supervised (task) objective itself. The scoring network was paired with a scoring function that allows an appropriate trade-off between the two heads. We also ensure that the selected samples are diverse by selectively fine-tuning the scoring network in sub-rounds of each query round. The resulting scheme performs competitively with the state-of-the-art on benchmark datasets. More importantly, in realistic scenarios when some labels are erroneous and new classes are introduced on the fly, the performance of the proposed method remains strong.