The standard contrastive learning acts on the extracted features with ℓ_2 normalization. For domain adaptation tasks, however, we find that contrastive learning with the standard paradigm does not perform well. The reason is mainly that the class weights (weights of the final fully connected layer) are not involved during optimization, which does not guarantee the produced features to be clustered around the class weights learned from source data. To tackle this issue, we propose a simple yet powerful probabilistic contrastive learning (PCL) in this paper, which not only produces compact features but also enforces them to be distributed around the class weights. Specifically, we break the traditional contrastive learning paradigm (feature+ℓ_2 normalization) by replacing the features with probabilities and removing ℓ_2 normalization. In this way, we can enforce the probability to approximate the one-hot form, thereby narrowing the distance between the features and the class weights. PCL is generic due to conciseness, which can be used for different tasks. In this paper, we conduct extensive experiments on five tasks, i.e., unsupervised domain adaptation (UDA), semi-supervised domain adaptation (SSDA), semi-supervised learning (SSL), UDA detection, and UDA semantic segmentation. The results demonstrate that our PCL can bring significant gains for these tasks. In particular, for segmentation tasks, with the blessing of PCL, our method achieves or even surpasses CPSL-D with a smaller training cost (1*3090, 5 days vs 4*V100, 11 days). Code is available at https://github.com/ljjcoder/Probabilistic-Contrastive-Learning.