Graph Convolutional Networks (GCNs) have been widely studied for compact data representation and semi-supervised learning tasks. However, existing GCNs usually use a fixed neighborhood graph which is not guaranteed to be optimal for semi-supervised learning tasks. In this paper, we first re-interpret graph convolution operation in GCNs as a composition of feature propagation and (non-linear) transformation. Based on this observation, we then propose a unified adaptive neighborhood feature propagation model and derive a novel Adaptive Neighborhood Graph Propagation Network (ANGPN) for data representation and semi-supervised learning. The aim of ANGPN is to conduct both graph construction and graph convolution simultaneously and cooperatively in a unified formulation and thus can learn an optimal neighborhood graph that best serves graph convolution for data representation and semi-supervised learning. One main benefit of ANGPN is that the learned (convolutional) representation can provide useful weakly supervised information for constructing a better neighborhood graph which meanwhile facilitates data representation and learning. Experimental results on four benchmark datasets demonstrate the effectiveness and benefit of the proposed ANGPN.