This work introduces an attention mechanism for image classifiers and the corresponding deep neural network (DNN) architecture, dubbed ISNet. During training, the ISNet uses segmentation targets to learn how to find the image's region of interest and concentrate its attention on it. The proposal is based on a novel concept, background relevance minimization in explanation heatmaps. It can be applied to virtually any classification neural network architecture, without any extra computational cost at run-time. Capable of ignoring the background, the resulting single DNN can substitute the common pipeline of a segmenter followed by a classifier, being faster and lighter. We tested the ISNet with three applications: COVID-19 and tuberculosis detection in chest X-rays, and facial attribute estimation. The first two tasks employed mixed training databases, and fostered shortcut learning. By focusing on lungs and ignoring sources of bias in the background, the ISNet reduced the problem. Thus, it improved generalization to external (out-of-distribution) test datasets in the biomedical classification problems, surpassing a standard classifier, a multi-task DNN (performing classification and segmentation), an attention-gated neural network, and the standard segmentation-classification pipeline. Facial attribute estimation demonstrated that ISNet could precisely focus on faces, being also applicable to natural images. ISNet presents an accurate, fast, and light methodology to ignore backgrounds and improve generalization in diverse domains.