Statistical modelling of combined eye movements and EEG

A Digital Human Model is the vision of a simulated human which can be used for medical or scientific questions. Clearly, such a simulation requires models of cognitive and perceptual processes in the brain. To gain fundamental knowledge to inform such models, we commonly measure brain responses using electroencephalography (EEG). However,  in order to allow a Digital Human Model to exhibit natural behavior in a natural world, we foremost need to investigate real human behavior and brain responses in their own natural environments. Currently, such experiments are rarely possible, because the necessary analysis tools are missing. Here, we develop these highly needed tools by using a new unified statistical framework based on deconvolution and Generalized Additive Models. The resulting models represent important steps towards multi-x modelling (FC2) in the Digital Human (FC1) and address challenges of ML4Sim (PN6).

Already within the project network, PN7.5 investigates connections of data-driven models based on eye-gaze data, with modular simulations of cognition which include attention, perception and decision-making modules. These models will be able to mimic behavior, but they are uninformed about any actual brain processes. We will bridge this gap by combining brain activity measures and statistical and cognitive models. In addition, we will connect with PN7.5 by focusing on the combination of gaze-evoked brain activity. In broader terms, we want to build a fundament for investigating the human brain detached from experimental control, under fully natural conditions. The outcomes of this project can be used to build parts of a Digital Human Model applicable to natural experiments which require unrestricted gaze, and which need concurrent analysis of brain activity: for example, revealing fatigue at demanding workplaces, detecting omissions of important information in visualization or medical imaging, or improving human-computer interactions.