Feature-based out-of-distribution (OOD) detectors have received significant attention under the image classification setting lately. However, the practicality of these works in the object detection setting is limited due to the current lack of understanding of the characteristics of the feature space in this setting. Our approach, SAFE (Sensitivity-Aware FEatures), leverages the innate sensitivity of residual networks to detect OOD samples. Key to our method, we build on foundational theory from image classification to identify that shortcut convolutional layers followed immediately by batch normalisation are uniquely powerful at detecting OOD samples. SAFE circumvents the need for realistic OOD training data, expensive generative models and retraining of the base object detector by training a 3-layer multilayer perceptron (MLP) on the surrogate task of distinguishing noise-perturbed and clean in-distribution object detections, using only the concatenated features from the identified most sensitive layers. We show that this MLP can identify OOD object detections more reliably than previous approaches, achieving a new state-of-the-art on multiple benchmarks, e.g. reducing the FPR95 by an absolute 30% from 48.3% to 18.4% on the OpenImages dataset. We provide empirical evidence for our claims through our ablations, demonstrating that the identified critical subset of layers is disproportionately powerful at detecting OOD samples in comparison to the rest of the network.