Diagnosis of emotional user states for affect-adaptive systems in safety-critical environments

Affect-adaptive systems detect the current emotional state (consisting of two dimensions, valence and arousal) of the user and are capable of adequately responding by adapting the interaction. Emotional user states such as anger or anxiety can influence performance in human-machine systems, which may have fatal consequences in safety-critical environments. The aim of this project is to develop emotional state diagnostics for this domain, that allows for continuous discrimination of “critical emotional states” and “uncritical emotional states”. The first step in this project was to develop an experimental environment that simulates safety-critical environments and to connect sensors that assess the emotional user state. Emotional valence is detected using facial expression analysis and pupil diameters, heart rate, heart rate variability and respiration rate are used to indicate emotional arousal. An initial laboratory experiment investigated the relationship between emotion and performance and interindividual differences were found. A future affect-adaptive system should take these differences into account in order to adapt interaction appropriately. Therefore, the concept of Affective Response Categories (ARCs) that categorizes users according to their emotion-performance relationship was developed and experimentally validated. Further steps towards a real time diagnosis and classification of the emotional user state require an algorithm that assigns users to ARCs and the conceptual design of a system, which is able to perform a continuous classification into "critical" and "non-critical" based on the respective ARC and the current emotional state. In a final step, the diagnosis system will be experimentally validated. This project is a cooperation with and partially funded by Fraunhofer FKIE.