Coronary artery disease (CAD) is one of the primary causes leading deaths worldwide. The presence of atherosclerotic lesions in coronary arteries is the underlying pathophysiological basis of CAD, and accurate extraction of individual arterial branches using invasive coronary angiography (ICA) is crucial for stenosis detection and CAD diagnosis. We propose an innovative approach called the Edge Attention Graph Matching Network (EAGMN) for coronary artery semantic labeling. By converting the coronary artery semantic segmentation task into a graph node similarity comparison task, identifying the node-to-node correspondence would assign semantic labels for each arterial branch. More specifically, The EAGMN utilizes the association graph constructed from the two individual graphs as input. Experimental results indicate the EAGMN achieved a weighted accuracy of 0.8653, a weighted precision of 0.8656, a weighted recall of 0.8653 and a weighted F1-score of 0.8643. Furthermore, we employ ZORRO to provide interpretability and explainability of the graph matching for artery semantic labeling. These findings highlight the potential of the EAGMN for accurate and efficient coronary artery semantic labeling using ICAs. By leveraging the inherent characteristics of ICAs and incorporating graph matching techniques, our proposed model provides a promising solution for improving CAD diagnosis and treatment