This paper describes PAWS-VMK, a prototypical deep learning approach that obtains state-of-the-art results for image classification tasks in both a semi-supervised learning (SSL) and out-of-distribution (OOD) detection context. We consider developments in the fields of SSL, OOD detection, and computer vision foundation models to introduce a number of innovations that connect the key ideas within these works to create PAWS-VMK. These innovations include (1) parametric von Mises-Fisher Stochastic Neighbour Embedding (vMF-SNE) to initialise a projection head for SSL using the high-quality embeddings of the foundation model; (2) the PAWS-MixMatch loss, that creates more compact embeddings and obtains higher accuracy in comparison to the consistency loss used in PAWS and (3) simple k-Means prototype selection (SKMPS), a simple technique that obtains competitive performance with more complex unsupervised label selection approaches. PAWS-VMK sets new benchmarks in semi-supervised learning for CIFAR-10 (99.2%) and CIFAR-100 (89.8%) with four labelled instances per class, and Food-101 (90.1%) with two labelled instances per class. We also observe that PAWS-VMK can efficiently detect OOD samples in a manner that is competitive with specialised methods specifically designed for this purpose, achieving 93.1/98.0 and 95.2/96.3 on the CIFAR-10 and CIFAR-100 OpenOOD benchmarks.