CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on the particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from explaining away, that is, it focuses too much on few specific features and/or insufficiently extracts the covariance structure in the data. The former problem of focusing on few features only is caused by a saturation of the InfoNCE objective, which is severe for high mutual information. The latter problem of insufficiently exploiting the covariance structure is caused by a deficiency in extracting feature associations and co-occurrences. We introduce "Contrastive Leave One Out Boost" (CLOOB), which uses the InfoLOOB objective and modern Hopfield networks. In contrast to InfoNCE, the InfoLOOB objective (leave one out bound) does not saturate and works well for high mutual information. Modern Hopfield networks, on the other hand, allow to use retrieved embeddings, which have an enriched covariance structure via co-occurrences of stored features. We compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.