Human Action Recognition by Discriminative Feature Pooling and Video Segmentation Attention Model


We introduce a simple yet effective network that embeds a novel Discriminative Feature Pooling (DFP) mechanism and a novel Video Segment Attention Model (VSAM), for video-based human action recognition from both trimmed and untrimmed videos. Our DFP module introduces an attentional pooling mechanism for 3D Convolutional Neural Networks that attentionally pools 3D convolutional feature maps to emphasize the most critical spatial, temporal, and channel-wise features related to the actions within a video segment, while our VSAM ensembles these most critical features from all video segments and learns (1) class-specific attention weights to classify the video segments into the corresponding action categories, and (2) class-agnostic attention weights to rank the video segments based on their relevance to the action class. Our action recognition network can be trained from both trimmed videos in a fully-supervised way and untrimmed videos in a weakly-supervised way. For untrimmed videos with weak labels, our network learns attention weights without the requirement of precise temporal annotations of action occurrences in videos. Evaluated on the untrimmed video datasets of THUMOS14 and ActivityNet1.2, and trimmed video datasets of HMDB51, UCF101, and HOLLYWOOD2, our network achieves superior performance, compared to the latest state-of-the-art methods


Mechanical and Aerospace Engineering

Research Center/Lab(s)

Intelligent Systems Center

Publication Status

Early Access


First Published 09 Feb 2021

This research work is supported by the National Science Foundation via CPS Synergy project CMMI-1646162 and National Robotics Initiative project NRI-1830479.

Keywords and Phrases

Action Recognition; Annotations; Attentional Pooling; Discriminative Features; Feature Extraction; Fullysupervised; Image Recognition; Task Analysis; Three-Dimensional Displays; Training; Two Dimensional Displays; Weakly-Supervised

International Standard Serial Number (ISSN)

1520-9210; 1941-0077

Document Type

Article - Journal

Document Version


File Type





© 2021 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

09 Feb 2021