Human Face Emotion Recognition in Thermal Images Using cGAN and EfficientNet with Attention Mechanisms

Abstract

Human use emotions to express their feelings and effectively interact with others. Human express emotions through hands, voice, gestures, and most importantly, facial expressions. Facial emotion recognition widely used in human-computer interaction, security, and healthcare. Traditionally based on posed visible images, such methods fail to capture natural emotions and are prone to lighting conditions. Visible spectrum imaging has several limitations, including sensitivity to lighting conditions, facial obstructions, and potential variations in facial expressions. Thermal imaging is less affected by obstructions of the face and changes in illumination. Thermal facial emotion recognition offers a robust alternative to conventional visible-spectrum methods by mitigating the effects of lighting variability and deceptive facial expressions. However, the use of thermal images for emotion recognition has not been extensively explored, primarily due to the limited availability of thermal image datasets. This research investigated the impact of enhanced preprocessing, synthetic data generation, and advanced deep learning architectures on thermal emotion classification. In this paper we used bilateral filters, Gaussian Blur and CLAHE for image preprocessing, data augmentation, cGANs for synthetic data generation, and EfficientNet-B5with CBAM for feature extraction. We utilized CBAM on the EfficientNet-B5 blocks. Our proposed method passed the extracted features to ResNet-18 for the classification of human facial emotions of thermal images into five emotional expressions such as happy, sad, angry, natural, and surprise. The proposed method used comprehensive facial thermal datasets for training and testing to compare its performance on baseline work. We employed the Fréchet Inception Distance to evaluate the realism and quality of the generated images. Accuracy, precision and F1-score metrics are utilized to assess the performance of the model. Experimental results demonstrated a significant improvement in recognition performance, achieving an accuracy of 98.81%. These findings highlight the effectiveness of the proposed pipeline for thermal facial emotion recognition and suggest promising potential for deployment in real-world conditions.