CNS*2026 Halifax

Introduction
Spatial navigation relies on integrating multimodal cues[1], yet both in vivo and in silico hippocampal research overwhelmingly focuses on vision [2,3,6]. While recent work showed mice entorhinal cortex contain both unimodal and multimodal cells[3], how different modalities are weighted and integrated remains poorly understood. We develop a modelling pipeline with an agent traversing a multimodal VR environment. A recurrent neural network was trained to perform a next-state prediction task[2]. We hypothesised that the network would develop place cell-like units that utilise both modalities, and that integrating modalities would result in more robust spatial representations, with each sense contributing additively to the cognitive map.

Methods
An agent traversed a 2D arena with audiovisual cues in Unity3D[4]. Binaural audio broadcasted with head-related transfer functions[5] and visual frames were encoded via autoencoders into low-dimensional embeddings. We trained an RNN[2] to perform next-state prediction using its current sensory states and motion, under three conditions: audiovisual, visually-lesioned and auditorily-lesioned. Latent units were classified into place units using empirical metrics such as spatial information scores. These tunings were used to perform maximum-likelihood decoding of the agent’s position and orientation. The relative contributions of sensory inputs were effectively decomposed using a linearly weighted combination of their unimodal responses.

Results
The audiovisual model produced 127 spatially tuned place cells, significantly more than visually-driven (33 cells) and auditorily-driven (81 cells) ones. Furthermore, the audiovisual model yielded the lowest trajectory decoding error (0.151 m) compared to visual-only (0.879 m) and auditory-only (0.293 m) ones, with the highest spatial information content. Unimodal units that respond to a single modality were identified, as well as multimodal units that remap when both are present. Finally, by approximating multimodal ratemaps as linear combinations of unimodal maps, we found that most place cells integrate modalities additively, exhibiting intermediate visual weightings (μ=0.405) and relying more on auditory cues.

Discussion
While derived in silico, these results offer a framework for biological navigation. The model suggests the hippocampus may additively processes multisensory streams to reduce uncertainty rather than switching between senses. Notably, auditory cues proved dominant in our VR setup, likely because visual landmarks lose salience at a distance or vanish when facing walls. Consequently, multimodal units anchor to the most reliable available cues—in this case, sound. We further hypothesise these units will dynamically remap or reweigh sensory reliance if a primary modality degrades. Ultimately, this model provides a normative theory for multisensory integration, generating precise, testable predictions for planned in vivo ferret recordings.

(a) Virtual environment with visual cues and sound sources; (b) Model architecture and pipeline; (c) Spatial ratemap examples in audiovisual, and lesioned (-) conditions; (d) Distribution of spatial information contents; (e) Number of place units identified; (f) ML decoding of position; (g) ML decoding of head direction; (h) Distribution of visual weights (x) and resulting correlations (y); compar

References

Jeffery, K. J. (2007). Integration of the sensory inputs to place cells: what, where, why, and how?. Hippocampus
[1] Levenstein, D., ..., & Richards, B. (2024). Sequential predictive learning is a unifying theory for hippocampal representation and replay. bioRxiv
[2] Nguyen, D., ... , & Gu, Y. (2024). The medial entorhinal cortex encodes multisensory spatial information. Cell reports
[3] George, T. M., ..., & Barry, C. (2024). RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments. Elife
[4] Cuevas-Rodríguez, ... & Reyes-Lecuona, A. (2019). 3D Tune-In Toolkit: An open-source library for real-time binaural spatialisation. PloS one
[5] Banino, A., ... & Kumaran, D. (2018). Vector-based navigation using grid-like representations in artificial agents. Nature


Acknowledgement
We thank Barry Lab, Bizley La, the Department of UCL Cell and Developmental Biology, the Ear Institute for this work.
This work was supported by the UKRI Biotechnology and Biological Sciences Research Council [grant
number BB/T008709/1].