Bridging Natruralistic Stimuli, Eye Movement And Brain Activity Via Cca And Locality Preserving Projection

Naturalistic stimuli provide more life-like multimodal inputs and promise to investigate brain activity and cognition. Many studies have succeeded in bridging low-level naturalistic stimuli features and high-level semantics derived from brain responses. In this paradigm, eye movement was widely used as a behavior measurement, which was also demonstrated to closely related to cognitive processes of individuals. However, there lacks an integrative work to bridge these three modalities ranging from multimedia feature, high-level brain semantics, and behavior. In this work, we adopted canonical correlation analysis (CCA) method with the locality preserving projection (LPP) to correlate functional MRI based brain responses to multimedia feature features under the guidance of eye movement types. Canonical correlation was estimated between brain responses and multimedia feature features while the LPP constraint is used to preserve the temporal relationship by maximizing the discrimination between time points which have different eye movement types. Our method selected brain regions responsive to stimulus features exhibit a gradient on the cortex. Both selected brain regions and multimedia feature have higher distinguishing ability among eye movement types than other unselected ones, suggestive of the validity of the method.