Pedestrian detection is a research hotspot and a difficult issue in the computer vision such as the Intelligent Surveillance System, the Intelligent Transport System, robotics, and automotive safety. However, the human body's position, angle, and dress in a video scene are complicated and changeable, which have a great influence on the detection accuracy. In this paper, through the analysis on the pros and cons of Census Transform Histogram (CENTRIST), a novel feature is presented for human detection Ternary CENTRIST (T-CENTRIST). The T-CENTRIST feature takes the relationship between each pixel and its neighborhood pixels into account. Meanwhile, it also considers the relevancy among these neighborhood pixels. Therefore, the proposed feature description method can reflect the silhouette of pedestrian more adequately and accurately than that of CENTRIST. Second, we propose a fast pedestrian detection framework based on T-CENTRIST in infrared image, which introduces the idea of extended blocks and the integral image. Finally, experimental results verify the effectiveness of the proposed pedestrian detection method.