In this paper, we first tackle the problem of pedestrian attribute recognition by video-based approach. The challenge mainly lies in spatial and temporal modeling and how to integrating them for effective and dynamic pedestrian representation. To solve this problem, a novel multi-task model based on the conventional neural network and temporal attention strategy is proposed. Since publicly available dataset is rare, two new large-scale video datasets with expanded attribute definition are presented, on which the effectiveness of both video-based pedestrian attribute recognition methods and the proposed new network architecture is well demonstrated. The two datasets are published on http://irip.buaa.edu.cn/mars_duke_attributes/index.html.