The Interplay of Hand Gestures and Facial Expressions in Conveying Emotions A CNN-BASED APPROACH

Emotion recognition is the process of identifying human emotions. Identifying the emotion being conveyed by facial expressions is a well-studied area as is the processing of formal hand gestures (mudras, semaphores, sign language etc..) as a means of communication. Our study focuses on two observations:(1) hand gestures, which have no reference to any formal system, can convey meanings even without accompanying facial expressions and (2) facial expressions and 'informal' hand gestures can nontrivially combine to convey messages with altogether new meanings. We present visuals to illustrate these observations. Experimentally, we present an image classification algorithm using Convolutional Neural Networks and TensorFlow library and OpenCV technology; with suitable datasets, we were able to train our system to recognize emotions conveyed by a limited set of hand gestures with no support from facial expressions (observation 1 above). We also indicate how the work ought to be extended to handle cases where hand gestures and facial expressions combine to convey interesting emotional signals.