Keyframe-Based Saliency Detection For Human Action Recognition Using Deep Learning

Abstract

Human action recognition is an active research area in the computer vision and it is widely applied to video processing, surveillance, home automation, healthcare, human behavior analysis, etc. Problems such as the inclusiveness of irrelevant features and redundant video frames affects the performance of action recognition. To overcome such challenges, traditional methods for recognizing human actions where the main focus was on body motions were studied. With the recent advances in deep learning, techniques that integrate spatial and temporal features with reduced dimension have been developed with marked success over the traditional methods. However, solely extracting the informative frames for representing action, and including only relevant information for fast processing remains a challenge. Therefore, in this study, keyframe-based saliency detection using deep learning was developed to overcome the stated problems. This study is aimed at developing keyframebased saliency detection to recognize human actions using deep learning. Keyframe selection using color difference of consecutive frames compared to local maximum, saliency detection using pyramidal attention guided network, and VGG-16+Bi-directional LSTM for action recognition were used. UCF101 dataset of 20 classes was used for training and validating the network using 70%, 30% train-test split approach. By applying the developed KFSD HAR model, the overall training accuracy of 86% and 61sec time was achieved. Generally, the developed KFSD HAR outperforms existing VGG-Bi-LSTM, VGG-LSTM and CNN models having training accuracy (4.023%, 53.2%, 83%) and time(329 sec, 175 sec., 66 sec.) respectively due to potential keyframes selected from video and detected using saliency method for action recognition.