Who Missed the Class? -- Unifying Multi-Face Detection, Tracking and Recognition in Videos


We investigate the problem of checking class attendance by detecting, tracking and recognizing multiple student faces in classroom videos taken by instructors. Instead of recognizing each individual face independently, first, we perform multi-object tracking to associate detected faces (including false positives) into face tracklets (each tracklet contains multiple instances of the same individual with variations in pose, illumination etc.) and then we cluster the face instances in each tracklet into a small number of clusters, achieving sparse face representation with less redundancy. Then, we formulate a unified optimization problem to (a) identify false positive face tracklets; (b) link broken face tracklets belonging to the same person due to long occlusion; and (c) recognize the group of faces simultaneously with spatial and temporal context constraints in the video. We test the proposed method on Honda/UCSD database and real classroom scenarios. The high recognition performance achieved by recognizing a group of multi-instance tracklets simultaneously demonstrates that multi-face recognition is more accurate than recognizing each individual face independently.

Meeting Name

IEEE International Conference on Multimedia and Expo (2014: Jul. 14-18, Chengdu, China)


Computer Science

Keywords and Phrases

Object Recognition; Optimization; Students; Tracking (Position); Context Constraint; Face Representations; Face Tracking; Multi-Object Tracking; Multiple Instances; Multiple Object Tracking; Number of Clusters; Unified Optimizations; Face Recognition; Face Detection; multiple Object Recognition

International Standard Serial Number (ISSN)


Document Type

Article - Conference proceedings

Document Version


File Type





© 2014 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

01 Sep 2014