Temporal action detection (TAD) aims to determine the semantic label and the boundaries of every action instance in an untrimmed video. Previous methods tackle this task with complex pipelines. In this paper, we propose an end-to-end temporal action detection Transformer (TadTR) with a simple set prediction pipeline. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context from the video for each query and directly predicts action instances. To adapt Transformer for TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. TadTR requires lower computation cost than previous detectors while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Our code is available at https://github.com/xlliu7/TadTR.