Images in aerial datasets are very large in resolution, and each frame contains many dense and small objects. State-of-the-art detection methods fail to capture small objects, local features, and region proposals for densely overlapped objects in aerial imagery due to the high variation of object sizes in satellite imagery with respect to the image size and high variation of content. Aerial imagery content varies greatly within the dataset due to the large change in lighting conditions, and the type of ground imagery captures from high altitudes. The variation is even higher between different datasets as object sizes, class distributions, image acquisition, and weather conditions can vary even more drastically. Thus, Domain Adaptation (DA) has been introduced as a band-aid to alleviate the degradation of object identification in previously unseen datasets. In this paper, we propose a small object detection pipeline that improves the feature extraction process by spatial pyramid pooling, cross-stage partial networks, heat-map-based region proposal network, and objects localization and identification through a novel image difficulty score that adapts the overall focal loss measure based on the image difficulty. Next, we propose novel contrastive learning with progressive domain adaptation to produce domain-invariant features across aerial datasets using local and global features. Effective analysis and illustration of different performance metrics and challenges show that our proposed method is comparable to the current State-of-Art models and creates a first-ever Domain Adaptation benchmark for the object detection task in highly imbalanced satellite datasets with large domain gaps and dominant small objects.