Unsupervised representation learning (URL) that learns compact embeddings of complex data without supervision has achieved remarkable progress recently. Although the general goal of URL is similar across various scenarios, the actual algorithms differ widely in different tasks because they were separately designed according to a specific URL task. In this paper, we develop a general and efficient similarity-based URL framework called GenURL, which can adapt to various URL tasks in a unified manner. Based on the manifold assumption, we regard most URL tasks as an embedding problem that seeks an optimal low-dimensional representation for the high-dimensional data. The learning process contains two steps, data structural modeling, and low-dimensional embedding. Specifically, we provide a general method to model data structures by adaptively combining graph distances on the predefined graphs, then propose robust loss functions for the low-dimensional embedding objective. Combined with a specific pretext task, GenURL achieves state-of-the-art or competitive performance in self-supervised visual representation learning, unsupervised knowledge distillation, graph embeddings, and dimension reduction.