Most existing re-identification methods focus on learning ro-bust and discriminative features with deep convolution net-works. However, many of them consider content similarityseparately and fail to utilize the context information of thequery and gallery sets, e.g. probe-gallery and gallery-galleryrelations, thus hard samples may not be well solved due tothe limited or even misleading information. In this paper,we present a novel Context-Aware Graph Convolution Net-work (CAGCN), where the probe-gallery relations are en-coded into the graph nodes and the graph edge connectionsare well controlled by the gallery-gallery relations. In thisway, hard samples can be addressed with the context infor-mation flows among other easy samples during the graph rea-soning. Specifically, we adopt an effective hard gallery sam-pler to obtain high recall for positive samples while keeping areasonable graph size, which can also weaken the imbalancedproblem in training process with low computation complex-ity. Experiments show that the proposed method achievesstate-of-the-art performance on both person and vehicle re-identification datasets in a plug and play fashion with limitedoverhead.