基于特权信息和面部先验知识的表情类别和动作单元识别研究

In this project, we propose to address several issues related to facial action unit and expression recognition. First, we propose to recognize expression from visible images using thermal infrared images as privileged information. Second, we propose to combine facial prior knowledge with facial images to improve facial action unit and expression recognition. Finally, we propose a multi-expression recognition by exploiting dependencies among expressions. First, facial thermal images are used as privileged information. During the training of the expression recognition model, the privileged information is used to help select more effective visible image features and to help establish a better mapping relationship between the visible image features and its expression labels. During testing, using the learned mapping function, the expressions can be directly inferred from the visible images using selected visible features without privileged information. The proposed expression recognition method learning from privileged information provides a novel and potential solution to enhance the robustness of visible facial expression recognition to illuminant changes. Second, we combine the facial prior knowledge on action unit and expressions with image-driven recognition methods, and introduce a facial prior-based expression and action unit recognition. Since the generic knowledge on facial action and expression is independent on the imaging conditions, the proposed facial prior-based expression and action unit recognition may greatly advance the application process of expression and action unit recognition. Third, we analyze and capture the global concurrent and mutual exclusive dependences among expressions, and propose a multi-label expression classification method. It provides a new method for expression analysis and modeling.

本项目提出以红外热像为特权信息的可见光表情识别研究；在此基础上，研究融合面部先验知识和面部图像数据的可见光表情类别和动作单元识别；并探讨多标签的表情类别分类方法。 本项目以面部红外热图作为特权信息，在表情识别模型的训练阶段，辅助可见图像，更好地建立可见图像特征到表情之间的映射关系；在表情识别模型的测试阶段，只需通过可见光图像识别用户表情，提出基于特权信息的可见光表情分析和识别研究，为提高可见光表情识别对光照的鲁棒性提出了一种新的、可行的解决途径。本项目以面部动作单元之间以及表情类别和动作单元之间的时空概率关系为先验知识，结合特征驱动的表情识别方法，提出融合面部先验知识和面部图像数据的表情识别，为推高表情识别算法的扩展性能提供新的思路，必将推动表情识别的实用化进程。本项目对各种表情类别的共生和互斥关系进行分析建模，提出多标签的表情类别识别，为表情识别研究提供了新的方法。

面部表情是人类情感交流的一个重要途径，因此，表情识别已经成为情感人机交互研究中的关键课题之一。本项目提出基于特权信息的表情识别，以红外热图为特权信息，辅助可见光图像，建立可见光特征到表情之间的映射关系，实现基于特权信息的可见光表情分类识别，为提高可见光表情识别对光照的鲁棒性提出了一种新的、可行的解决途径；提出融合面部先验知识和面部图像数据的可见光表情类别和动作单元识别，分析表情变化时，面部肌肉运动的时空关系，采用概率图模型和对抗学习对面部肌肉运动的时空关系建模，融合图像特征，实现表情类别和面部动作单元识别，为推高表情识别算法的扩展性能提供新的思路；提出多标签的表情识别，采用概率图模型对多标签之间的依赖关系进行建模，实现多标签表情类别分类，为表情识别研究提供了新的方法。在国内外高水平期刊和重要会议上共发表论文47篇，包括SCI期刊论文15篇，CCF A类会议论文13篇。培养硕士10名，博士1名。参与组织相关国际会议3次。获得情感计算领域国际比赛亚军一次。