Multimodal Affective Computing: Affective Information Representation, Modelling, and Analysis

With the invention of high-power computing systems, machines are expected to show intelligence at par with human beings. A machine must be able to analyze and interpret emotions to demonstrate intelligent behavior. Affective computing not only helps computers to improve performance intelligently but also helps in decision-making. This chapter introduces affective computing and related issues that influence emotions. This study also provides an overview of humancomputer interaction (HCI) and the possible use of different modalities for HCI. Further, challenges in affective computing are also discussed, along with the application of affective computing in various areas.

This chapter presents a brief overview of Affective computing and a formal definition of emotion given by various researchers. Human-computer interaction aims to enhance communication between man and machine so that machines can acquire, analyze, interpret and act on par with human beings. At the same time, Affective human-computer interaction focuses on enhancing communication between man and machines using affective information. Moreover, this chapter deals with Human emotional expression and perception through various modalities such as speech, facial expressions, physiological signals, etc. It also detailed the overview of Action Units and Techniques for classifying facial expressions as reported in the literature.

This chapter presents a state-of-the-art review of existing emotion theory, modeling approaches, and affective information extraction and processing methods. The basic theory of emotions deals with Darwin's evolutionary theory, Schechter's theory of emotion, and James–Lange's theory. These theories are fundamental building blocks of Affective Computing research. Emotion modeling approaches can be categorized into categorical, appraisal, and dimensional models. Noticeable contributions to Affect recognition systems in terms of modality, database, and dimensionality are also discussed in this chapter.

This chapter presents a state-of-the-art review of existing affective information extraction and processing approaches. Various evaluation criteria, such as Evaluation matrices like ROC, F1 measure, Mean Square Error, Mean Average Error, Threshold criteria, and Performance criteria are also reported in this chapter.

The multimodal information can be assimilated at three levels 1) early fusion, 2) intermediate fusion, and 3) late fusion. Early fusion can be performed at the sensor or signal level. Intermediate fusion can be at the feature level, and late fusion may be done at the decision level. Apart from that, some more fusion techniques are rank-based, adaptive, etc. This chapter provides an extensive review of studies based on fusion and reported noticeable work herewith. Eventually, we discussed the challenges associated with multimodal fusion. 

This chapter presents a multi-modal fusion framework for emotion recognition using multiresolution analysis. The proposed framework consists of three significant steps: (1) feature extraction and selection, (2) feature level fusion, and (3) mapping of emotions in three-dimensional VAD space. The proposed framework considers subject-independent features that can incorporate many more emotions. It is possible to handle many channel features, especially synchronous EEG channels and feature-level fusion works. This framework of representing emotions in 3D space can be extended for mapping emotion in three-dimensional spaces with three specific coordinates for a particular emotion. In addition to the fusion framework, we have explained multiresolution approaches, such as wavelet and curvelet transform, to classify and predict emotions.

This study presents emotion recognition from facial expressions in a noisy environment. The challenges addressed in this study are noise in the images and illumination changes. Wavelets have been extensively used for noise reduction; therefore, we have applied wavelet and curvelet analysis from noisy images. The experiments are performed with different values of Gaussian noise (mean: 0.01, 0.03) and (variance: 0.01, 0.03). Similarly, for experimentation with illumination changes, we have considered different dynamic ranges (0.1, 0.9). Three benchmark databases, Cohn-Kanade, JAFFE, and In-house, are used for all experimentation. The five best machine learning algorithms are used for classification purposes. Experimental results show that SVM and MLP classifiers with wavelet and curvelet-based coefficients yield better results for emotion recognition. We can conclude that Wavelet coefficients-based features perform well, especially in the presence of Gaussian noise and illumination changes for facial expression recognition.

This chapter introduces an emotion recognition system based on audio and video cues. For audio-based emotion recognition, we have explored various aspects of feature extraction and classification strategy and found that wavelet analysis is sound. We have shown comparative results for discriminating capabilities of various combinations of features using the Fisher Discriminant Analysis (FDA). Finally, we have combined the audio and video features using a feature-level fusion approach. All the experiments are performed with eNTERFACE and RML databases. Though we have applied multiple classifiers, SVM shows significantly improved performance with a single modality and fusion. The results obtained using fusion outperformed in contrast results based on a single modality of audio or video. We can conclude that fusion approaches are best as it is using complementary information from multiple modalities.

This study presents a multimodal fusion framework for emotion recognition from physiological signals. In contrast to emotion recognition through facial expression, a large number of emotions can be recognized accurately through physiological signals. The DEAP database, a benchmark multimodal database with many collections of EEG and peripheral signals, is employed for experimentation. The proposed method takes into account those features that are subject-independent and can incorporate many more emotions. As it is possible to handle many channel features, especially synchronous EEG channels, feature-level fusion is applied in this study. The features extracted from EEG and peripheral signals include relative, logarithmic, and absolute power energy of Alpha, Beta, Gamma, Delta, and Theta. Experimental results demonstrate that physiological signals' Theta and Beta bands are the most significant contributor to the performance. On the other hand, SVM performs outstandingly.

In this study, we have discussed emotion representation in two and threedimensional space. The three-dimensional space is based on the three emotion primitives, i.e., valence, arousal, and dominance. The multimodal cues used in this study are EEG, Physiological signals, and video (under limitations). Due to the limited emotional content in videos from the DEAP database, we have considered only three classes of emotions, i.e., happy, sad, and terrible. The wavelet transforms, a classical transform, were employed for multi-resolution analysis of signals to extract features. We have evaluated the proposed emotion model with standard multimodal datasets, DEAP. The experimental results show that SVM and MLP can predict emotions in single and multimodal cues.