The vanilla GAN [5] suffers from mode collapse deeply, which usually manifests as that the images generated by generators tend to have a high similarity amongst them, even though their corresponding latent vectors have been very different. In this paper, we introduce a pluggable block called diversity penalty (dp) to alleviate mode collapse of GANs. It is used to reduce the similarity of image pairs in feature space, i.e., if two latent vectors are different, then we enforce the generator to generate two images with different features. The normalized Gram Matrix is used to measure the similarity. We compare the proposed method with Unrolled GAN [17], BourGAN [26], PacGAN [14], VEEGAN [23] and ALI [4] on 2D synthetic dataset, and results show that our proposed method can help GAN capture more modes of the data distribution. Further, we apply this penalty term into image data augmentation on MNIST, Fashion-MNIST and CIFAR-10, and the testing accuracy is improved by 0.24%, 1.34% and 0.52% compared with WGAN GP [6], respectively. Finally, we quantitatively evaluate the proposed method with IS and FID on CelebA, CIFAR-10, MNIST and Fashion-MNIST. Results show that our method gets much higher IS and lower FID compared with some current GAN architectures.