We propose a new method to tackle the mapping challenge from time-series data to spatial image in the field of seismic exploration, i.e., reconstructing the velocity model directly from seismic data by deep neural networks (DNNs). The conventional way of addressing this ill-posed inversion problem is through iterative algorithms, which suffer from poor nonlinear mapping and strong nonuniqueness. Other attempts may either import human intervention errors or underuse seismic data. The challenge for DNNs mainly lies in the weak spatial correspondence, the uncertain reflection-reception relationship between seismic data and velocity model, as well as the time-varying property of seismic data. To tackle these challenges, we propose end-to-end seismic inversion networks (SeisInvNets) with novel components to make the best use of all seismic data. Specifically, we start with every seismic trace and enhance it with its neighborhood information, its observation setup, and the global context of its corresponding seismic profile. From the enhanced seismic traces, the spatially aligned feature maps can be learned and further concatenated to reconstruct a velocity model. In general, we let every seismic trace contribute to the reconstruction of the whole velocity model by finding spatial correspondence. The proposed SeisInvNet consistently produces improvements over the baselines and achieves promising performance on our synthesized and proposed SeisInv data set according to various evaluation metrics. The inversion results are more consistent with the target from the aspects of velocity values, subsurface structures, and geological interfaces. Moreover, the mechanism and the generalization of the proposed method are discussed and verified. Nevertheless, the generalization of deep-learning-based inversion methods on real data is still challenging and considering physics may be one potential solution.