Previously, in the field of image registration, there are mainly two paradigms, the traditional optimization-based methods, and the deep-learning-based methods. We designed a robust training architecture that is simple and generalizable. We present Optron, a general training architecture incorporating the idea of optimizing-in-the-loop. By iteratively optimizing the prediction result of a deep learning model through a plug-and-play optimizer module in the training loop, Optron introduces pseudo ground truth to an unsupervised training process. This pseudo supervision provides more direct guidance towards model training compared with unsupervised methods. Utilizing this advantage, Optron can consistently improve the models' performance and convergence speed. We evaluated our method on various combinations of models and datasets, and we have achieved state-of-the-art performance on the IXI dataset, improving the previous state-of-the-art method TransMorph by a significant margin of +1.6% DSC. Moreover, Optron also consistently achieved positive results with other models and datasets. It increases the validation DSC on IXI for VoxelMorph and ViT-V-Net by +2.3% and +2.2% respectively, demonstrating our method's generalizability. Our implementation is publicly available at https://github.com/miraclefactory/optron