Data hiding is the procedure of encoding desired information into an image to resist potential noises while ensuring the embedded image has little perceptual perturbations from the original image. Recently, with the tremendous successes gained by deep neural networks in various fields, data hiding areas have attracted increasing number of attentions. The neglect of considering the pixel sensitivity within the cover image of deep neural methods will inevitably affect the model robustness for information hiding. Targeting at the problem, in this paper, we propose a novel deep data hiding scheme with Inverse Gradient Attention (IGA), combing the ideas of adversarial learning and attention mechanism to endow different sensitivity to different pixels. With the proposed component, the model can spotlight pixels with more robustness for embedding data. Empirically, extensive experiments show that the proposed model outperforms the state-of-the-art methods on two prevalent datasets under multiple settings. Besides, we further identify and discuss the connections between the proposed inverse gradient attention and high-frequency regions within images.