Children learn continually by asking questions about the concepts they are most curious about. With robots becoming an integral part of our society, they must also learn unknown concepts continually by asking humans questions. This paper presents a novel framework for curiosity-driven online learning of objects. The paper utilizes a recent state-of-the-art approach for continual learning and adapts it for online learning of objects. The paper further develops a self-supervised technique to find most of the uncertain objects in an environment by utilizing an internal representation of previously learned classes. We test our approach on a benchmark dataset for continual learning on robots. Our results show that our curiosity-driven online learning approach beats random sampling and softmax-based uncertainty sampling in terms of classification accuracy and the total number of classes learned.