Metric learning especially deep metric learning has been widely developed for large-scale image inputs data. However, in many real-world applications, we can only have access to vectorized inputs data. Moreover, on one hand, well-labeled data is usually limited due to the high annotation cost. On the other hand, the real data is commonly streaming data, which requires to be processed online. In these scenarios, the fashionable deep metric learning is not suitable anymore. To this end, we reconsider the traditional shallow online metric learning and newly develop an online progressive deep metric learning (ODML) framework to construct a metric-algorithm-based deep network. Specifically, we take an online metric learning algorithm as a metric-algorithm-based layer (i.e., metric layer), followed by a nonlinear layer, and then stack these layers in a fashion similar to deep learning. Different from the shallow online metric learning, which can only learn one metric space (feature transformation), the proposed ODML is able to learn multiple hierarchical metric spaces. Furthermore, in a progressively and nonlinearly learning way, ODML has a stronger learning ability than traditional shallow online metric learning in the case of limited available training data. To make the learning process more explainable and theoretically guaranteed, we also provide theoretical analysis. The proposed ODML enjoys several nice properties and can indeed learn a metric progressively and performs better on the benchmark datasets. Extensive experiments with different settings have been conducted to verify these properties of the proposed ODML.