Image retrieval under generalized test scenarios has gained significant momentum in literature, and the recently proposed protocol of Universal Cross-domain Retrieval is a pioneer in this direction. A common practice in any such generalized classification or retrieval algorithm is to exploit samples from multiple domains during training to learn a domain-invariant representation of data. Such criterion is often restrictive, and thus in this work, for the first time, we explore the challenges associated with generalized retrieval problems under a low-data regime, which is quite relevant in many real-world scenarios. We attempt to make any retrieval model trained on a small cross-domain dataset (containing just two training domains) more generalizable towards any unknown query domain or category by quickly adapting it to the test data during inference. This form of test-time training or adaptation of the retrieval model is explored by means of a number of self-supervision-based loss functions, for example, Rotnet, Jigsaw-puzzle, Barlow twins, etc., in this work. Extensive experiments on multiple large-scale datasets demonstrate the effectiveness of the proposed approach.