Deep learning architectures such as convolutional neural networks are the standard in computer vision for image processing tasks. Their accuracy however often comes at the cost of long and computationally expensive training, the need for large annotated datasets, and extensive hyper-parameter searches. On the other hand, a different method known as convolutional extreme learning machine has shown the potential to perform equally with a dramatic decrease in training time. Space imagery, especially about small bodies, could be well suited for this method. In this work, convolutional extreme learning machine architectures are designed and tested against their deep-learning counterparts. Because of the relatively fast training time of the former, convolutional extreme learning machine architectures enable efficient exploration of the architecture design space, which would have been impractical with the latter, introducing a methodology for an efficient design of a neural network architecture for computer vision tasks. Also, the coupling between the image processing method and labeling strategy is investigated and demonstrated to play a major role when considering vision-based navigation around small bodies.