The 2D object detection in clean images has been a well studied topic, but its vulnerability against adversarial attack is still worrying. Existing work has improved robustness of object detectors by adversarial training, at the same time, the average precision (AP) on clean images drops significantly. In this paper, we propose that using feature alignment of intermediate layer can improve clean AP and robustness in object detection. Further, on the basis of adversarial training, we present two feature alignment modules: Knowledge-Distilled Feature Alignment (KDFA) module and Self-Supervised Feature Alignment (SSFA) module, which can guide the network to generate more effective features. We conduct extensive experiments on PASCAL VOC and MS-COCO datasets to verify the effectiveness of our proposed approach. The code of our experiments is available at https://github.com/grispeut/Feature-Alignment.git.