This work introduces an attention mechanism for image classifiers and the corresponding deep neural network (DNN) architecture, dubbed ISNet. During training, the ISNet uses segmentation targets to learn how to find the image's region of interest and concentrate its attention on it. The proposal is based on a novel concept, background relevance minimization in LRP explanation heatmaps. It can be applied to virtually any classification neural network architecture, without any extra computational cost at run-time. Capable of ignoring the background, the resulting single DNN can substitute the common pipeline of a segmenter followed by a classifier, being faster and lighter. After injecting synthetic bias in images' backgrounds (in diverse applications), we compare the ISNet to multiple state-of-the-art neural networks, and quantitatively demonstrate its superior capacity of minimizing the bias influence over the classifier decisions. The tasks of COVID-19 and tuberculosis detection in chest X-rays commonly employ mixed training databases, which naturally foster background bias and shortcut learning. By focusing on lungs, the ISNet reduced shortcut learning, leading to significantly superior generalization to external (out-of-distribution) test datasets. ISNet presents an accurate, fast, and light methodology to ignore backgrounds and improve generalization.