Abstract
This paper presents a new thermal empowered multi-task network (TEMT-Net) to improve facial action unit detection. Our primary goal is to leverage the situation that the training set has multi-modality data while the application scenario only has one modality. Thermal images are robust to illumination and face color. In the proposed multi-task framework, we utilize both modality data. Action unit detection and facial landmark detection are correlated tasks. To utilize the advantage and the correlation of different modalities and different tasks, we propose a novel thermal empowered multi-task deep neural network learning approach for action unit detection, facial landmark detection and thermal image reconstruction simultaneously. The thermal image generator and facial landmark detection provide regularization on the learned features with shared factors as the input color images. Extensive experiments are conducted on the BP4D and MMSE databases, with the comparison to the state-of-the-art methods. The experiments show that the multi-modality framework improves the AU detection significantly.
Recommended Citation
P. Liu et al., "Multi-modality Empowered Network For Facial Action Unit Detection," Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019, pp. 2175 - 2184, article no. 8659257, Institute of Electrical and Electronics Engineers, Mar 2019.
The definitive version is available at https://doi.org/10.1109/WACV.2019.00235
Department(s)
Computer Science
International Standard Book Number (ISBN)
978-172811975-5
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2023 Institute of Electrical and Electronics Engineers, All rights reserved.
Publication Date
04 Mar 2019
Comments
National Science Foundation, Grant CNS-1629898