With the increasing maturity of the human pose estimation domain, its applications have become more and more broaden. Yet, the state-of-the-art pose estimation models performance degrades significantly in the applications that include novel subjects or poses, such as infants with their unique movements. Infant motion analysis is a topic with critical importance in early developmental studies. However, models trained on large-scale adult pose datasets are barely successful in estimating infant poses due to the significant differences in their body ratio and the versatility of poses they can take compared to adults. Moreover, the privacy and security considerations hinder the availability of adequate infant images required for training of a robust pose inference model from scratch. Here, we present an invariant representation learning strategy that allows us to augment the limited available real infant pose data by incorporating the knowledge from the adjacent domains of adult poses as well as synthetic infant models. We introduce a multi-stage training strategy to gradually transfer these knowledge into our fine-tuned domain-adapted infant pose (FiDIP) estimation model. In developing FiDIP, we also built and publicly released a synthetic and real infant pose (SyRIP) dataset with small yet diverse real infant images as well as generated synthetic infant data. We demonstrated that our FiDIP model outperforms state-of-the-art human pose estimation model for the infant pose estimation, with the mean average precision (AP) as high as 90.1.