Multimodal Emotional Analysis using XAI for Psychotherapy

A person’s mental well-being can be perceived by the emotions that they express. What a person feels can be observed by various physical and physiological cues. But people aren’t all the same, some are capable of expressing what they truly feel, others might not and there are certain scenarios where the person who is expressing those emotions isn’t completely aware of the emotional state they are in. Such scenarios are where even a trained professional isn't always one- hundred percent right. This raises the need for a solution that can observe a person’s behavioral traits and guess their emotional state. Currently we have various deep learning approaches that can tackle the problem in hand. One of the widely used approaches is making use of a Unimodal system that predicts a person’s emotional state by processing information that is collected in the form of a single modality. But using a single channel to perform such a complex classification task is often inefficient. To make more appropriate classifications, this study proposes a multimodal approach that incorporates eXplainable Artificial Intelligence (XAI) methodologies, and hence improving psychotherapeutic outcomes. The multimodal emotion recognition approach integrates multiple information channels of physical cues of a person, like speech and facial expressions. A more accurate prediction can be arrived at with various complementary channels backing it up. The addition of XAI algorithms make it clearer as to how the model arrived at its conclusion. Overall, this system provides a solution that can be personalized for each client and allows us to have a proper data-driven tool for emotional analysis, which can help the practitioners to design appropriate treatment plans for their client. By adding this state-of-the-art technology as a supplement to conventional psychotherapy techniques, we can yield more successful treatments.