Unsupervised representation learning (URL) that learns compact embeddings of high-dimensional data without supervision has achieved remarkable progress recently. Although the ultimate goal of URLs is similar across various scenarios, the related algorithms differ widely in different tasks because they were separately designed according to a specific URL task or data. For example, dimension reduction methods, t-SNE, and UMAP, optimize pair-wise data relationships by preserving the global geometric structure, while self-supervised learning, SimCLR, and BYOL, focus on mining the local statistics of instances under specific augmentations. From a general perspective, we summarize and propose a unified similarity-based URL framework, GenURL, which can adapt to various URL tasks smoothly and efficiently. Based on the manifold assumption, we regard URL tasks as different implicit constraints on the data geometric structure or content that help to seek an optimal low-dimensional representation for the high-dimensional data. Therefore, our method has two key steps to learning task-agnostic representation in URL: (1) data structural modeling and (2) low-dimensional transformation. Specifically, (1) provides a simple yet effective graph-based submodule to model data structures adaptively with predefined or constructed graphs; and based on data-specific pretext tasks, (2) learns compact low-dimensional embeddings. Moreover, (1) and (2) are successfully connected and benefit mutually through our novel objective function. Our comprehensive experiments demonstrate that GenURL achieves consistent state-of-the-art performance in self-supervised visual representation learning, unsupervised knowledge distillation, graph embeddings, and dimension reduction.