For infrastructure inspections, damage representation does not constantly match the predefined classes of damage grade, resulting in detailed clusters of unseen damages or more complex clusters from overlapped space between two grades. The damage representation has fundamentally complex features; consequently, not all the damage classes can be perfectly predefined. The proposed MN-pair contrastive learning method helps to explore an embedding damage representation beyond the predefined classes by including more detailed clusters. It maximizes both the similarity of M-1 positive images close to an anchor and dissimilarity of N-1 negative images using both weighting loss functions. It learns faster than the N-pair algorithm using one positive image. We proposed a pipeline to obtain the damage representation and used a density-based clustering on a 2-D reduction space to automate finer cluster discrimination. We also visualized the explanation of the damage feature using Grad-CAM for MN-pair damage metric learning. We demonstrated our method in three experimental studies: steel product defect, concrete crack, and the effectiveness of our method and discuss future works.