Temporal action detection (TAD) aims to determine the semantic label and the boundaries of every action instance in an untrimmed video. It is a fundamental  task in video understanding and significant progress has been made in TAD . Previous methods involve multiple stages or networks and hand-designed rules or operations, which fall short in efficiency and flexibility. Here, we construct an end-to-end framework for TAD upon Transformer, termed TadTR, which simultaneously predicts all action instances as a set of labels and temporal locations in parallel. TadTR is able to adaptively extract temporal context information needed for making action predictions, by selectively attending to a number of snippets in a video. It greatly simplifies the pipeline of TAD and runs much faster than previous detectors . Our method achieves state-of-the-art performance on HACS Segments and THUMOS14 and competitive performance on  ActivityNet-1.3. Our code will be made available at URL
Temporal action detection (TAD) aims to determine the semantic label and the boundaries of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding and significant progress has been made . Previous methods involve multiple stages or networks and hand-designed rules or operations, which fall short in efficiency and flexibility. In this paper, we propose an end-to-end framework for TAD upon Transformer, termed TadTR, which maps a set of learnable embeddings to action instances in parallel. TadTR is able to adaptively extract temporal context information required for making action predictions, by selectively attending to a sparse set of snippets in a video. As a result, it simplifies the pipeline of TAD and requires lower computation cost than previous detectors , while preserving remarkable detection performance. TadTR achieves state-of-the-art performance on HACS Segments (+3.35\% average mAP). As a single-network detector, TadTR runs 10\times faster than its comparable competitor. It outperforms existing single-network detectors by a large margin on THUMOS14 (+5.0\% average mAP) and ActivityNet (+7.53\% average mAP). When combined with other detectors, it reports 54.1\% mAP at IoU=0.5 on THUMOS14, and 34.55\% average mAP on ActivityNet-1.3. Our code will be released at URL
A
and challenging
D
in TAD
R
Here, we construct
In this paper, we propose
R
simultaneously predicts all action instances as
maps
R
labels and temporal locations
learnable embeddings to action instances
R
needed
required
R
number
sparse set
R
It greatly
As a result, it
R
runs much faster
requires lower computation cost
R
. Our method
, while preserving remarkable detection performance. TadTR
R
and
(+3.35\% average mAP). As a single-network detector, TadTR runs 10\times faster than its comparable competitor. It outperforms existing single-network detectors by a large margin on
R
and competitive performance
(+5.0\% average mAP) and ActivityNet (+7.53\% average mAP). When combined with other detectors, it reports 54.1\% mAP at IoU=0.5
A
THUMOS14, and 34.55\% average mAP on
R
made available
released