Despite being studied extensively for a few decades, handwritten character recognition (HCR) is considered a challenging learning problem in pattern recognition and there is very limited research on script independent models. This is mainly because of diversity of scripts, focus of the conventional research on handcrafted feature extraction techniques, and unavailability of public datasets and codes to reproduce the results. On the other hand, deep learning has witnessed huge success in different areas of pattern recognition, including HCR, and provides end-to-end learning but it has been studied for specific scripts only. In this paper, we have proposed a novel deep learning architecture which exploits transfer learning and image-augmentation for end-to-end learning for script independent handwritten character recognition, called HCR-Net. HCR-Net is based on a novel transfer learning approach for HCR, where some of lower layers of a pre-trained network are utilized. Due to transfer learning and image-augmentation, HCR-Net provides faster training, better performance and better generalizations, and can achieve up to 99% results of its final accuracy in just first epoch. The experimental results on publicly available datasets of Bangla, Punjabi, Hindi, English, Swedish, Urdu, Farsi, Tibetan, Kannada, Malayalam, Telugu, Marathi, Nepali and Arabic languages prove the efficacy of HCR-Net and establishes several new benchmarks. For reproducibility of the results and for the advancements of the HCR research, complete code is publicly released at https://github.com/jmdvinodjmd/HCR-Net.