Regression Before Classification For Temporal Action Detection

Action classification combined with location regression is a widely-utilized mechanism in existing temporal action detection methods. However, there exists an inconsistency problem between locations and categories of action instances in this mechanism. More specifically, while the location of the proposal has been refined by the regressor, the action classifier still uses input and loss corresponding to the outdated unrefined proposal to predict category. In this paper, we propose to eliminate this inconsistency by making two modifications to the action classifier: 1) redirecting the classification loss to the refined proposal, and 2) rearranging the location regressor before the action classifier so that the feature of the refined proposal is fed to the classifier. Extensive experiments show that eliminating the inconsistency problem can significantly promote the detection performance. Our method achieves state-of-the-art performance for temporal action detection on the challenging THUMOS'14 dataset.