This work introduces an attention mechanism for image classifiers and the corresponding deep neural network (DNN) architecture, dubbed ISNet. During training, the ISNet uses segmentation targets to learn how to find the image's region of interest and concentrate its attention on it. The proposal is based on a novel concept, background relevance minimization in LRP explanation heatmaps. It can be applied to virtually any classification neural network architecture, without any extra computational cost at run-time. Capable of ignoring the background, the resulting single DNN can substitute the common pipeline of a segmenter followed by a classifier, being faster and lighter. We tested the ISNet with three applications: COVID-19 and tuberculosis detection in chest X-rays, and facial attribute estimation. The first two tasks employed mixed training databases, which fostered background bias and shortcut learning. By focusing on lungs, the ISNet reduced shortcut learning, improving generalization to external (out-of-distribution) test datasets. When training data presented background bias, the ISNet's test performance significantly surpassed standard classifiers, multi-task DNNs (performing classification and segmentation), attention-gated neural networks, Guided Attention Inference Networks, and the standard segmentation-classification pipeline. Facial attribute estimation demonstrated that ISNet could precisely focus on faces, being also applicable to natural images. ISNet presents an accurate, fast, and light methodology to ignore backgrounds and improve generalization, especially when background bias is a concern.