Learning socially-aware motion representations is at the core of recent advances in human trajectory forecasting and robot navigation in crowded spaces. Despite promising progress, existing neural motion models often struggle to generalize in closed-loop operations (e.g., output colliding trajectories), when the training set lacks examples collected from dangerous scenarios. In this work, we propose to address this issue via contrastive learning with negative data augmentation. Concretely, we introduce a social contrastive loss that encourages the encoded motion representation to preserve sufficient information for distinguishing a positive future event from a set of negative ones. We explicitly draw these negative samples based on our domain knowledge of unfavorable circumstances in the multi-agent context. Experimental results show that the proposed method dramatically reduces the collision rates of recent trajectory forecasting, behavioral cloning and reinforcement learning algorithms, outperforming current state-of-the-art models on several benchmarks. Our method makes few assumptions about neural architecture designs, and hence can be used as a generic way to promote the robustness of neural motion models.