We tackle biomedical image segmentation in the scenario of only a few labeled brain MR images. This is an important and challenging task in medical applications, where manual annotations are time-consuming. Current multi-atlas based segmentation methods use image registration to warp segments from labeled images onto a new scan. In a different paradigm, supervised learning-based segmentation strategies have gained popularity. These method consistently use relatively large sets of labeled training data, and their behavior in the regime of a few labeled biomedical images has not been thoroughly evaluated. In this work, we provide two important results for segmentation in the scenario where few labeled images are available. First, we propose a straightforward implementation of efficient semi-supervised learning-based registration method, which we showcase in a multi-atlas segmentation framework. Second, through an extensive empirical study, we evaluate the performance of a supervised segmentation approach, where the training images are augmented via random deformations. Surprisingly, we find that in both paradigms, accurate segmentation is generally possible even in the context of few labeled images.