Vegetation Indices based on paired images of the visible color spectrum (VIS) and near infrared spectrum (NIR) have been widely used in remote sensing applications. These vegetation indices are extended for their application in autonomous driving in unstructured outdoor environments. In this domain we can combine traditional vegetation indices like the Normalized Difference Vegetation Index (NDVI) and Enhanced Vegetation Index (EVI) with Convolutional Neural Networks (CNNs) pre-trained on available VIS datasets. By laying a focus on learning calibrated CNN outputs, we can provide an approach to fuse known hand-crafted image features with CNN predictions for different domains as well. The method is evaluated on a VIS+NIR dataset of semantically annotated images in unstructured outdoor environments. The dataset is available at mucar3.de/iros2022-ppniv-tas-nir.