The importance of plastic waste recycling is undeniable. In this respect, computer vision and deep learning enable solutions through the automated analysis of short-wave-infrared hyper-spectral images of plastics. In this paper, we offer an exhaustive empirical study to show the importance of efficient model selection for resolving the task of hyper-spectral image segmentation of various plastic flakes using deep learning. We assess the complexity level of generic and specialized models and infer their performance capacity: generic models are often unnecessarily complex. We introduce two variants of a specialized hyper-spectral architecture, PlasticNet, that outperforms several well-known segmentation architectures in both performance as well as computational complexity. In addition, we shed lights on the significance of signal pre-processing within the realm of hyper-spectral imaging. To complete our contribution, we introduce the largest, most versatile hyper-spectral dataset of plastic flakes of four primary polymer types.