The combination of machine learning models with physical models is a recent research path to learn robust data representations. In this paper, we introduce p^3VAE, a generative model that integrates a physical model which deterministically models some of the true underlying factors of variation in the data. To fully leverage our hybrid design, we enhance an existing semi-supervised optimization technique and introduce a new inference scheme that comes along meaningful uncertainty estimates. We apply p^3VAE to the pixel-wise classification of airborne hyperspectral images. Our experiments on simulated and real data demonstrate the benefits of our hybrid model against conventional machine learning models in terms of extrapolation capabilities and interpretability. In particular, we show that p^3VAE naturally has high disentanglement capabilities. Our code and data have been made publicly available at https://github.com/Romain3Ch216/p3VAE.