CNS*2026 Halifax

Introduction
Reinforcement learning (RL) is a learning mechanism that allows animals to acquire behaviors through trial and error, and it is thought to be implemented in the brain's neural circuits. Hierarchical reinforcement learning (HRL), in which multiple RL systems are organized hierarchically, has been hypothesized to improve learning efficiency by decomposing long action sequences into shorter subgoal-directed processes [1]. However, whether such hierarchical organization improves learning efficiency in spiking neural network models has not been fully examined. In this study, we constructed an HRL model based on spiking neurons [2], and examined whether hierarchical organization improves learning efficiency in a two-dimensional maze task.


Methods
The hierarchical model consisted of a meta-controller as the higher-level process and a controller as the lower-level process. The meta-controller selected a subgoal from multiple candidates, whereas the controller generated actions to reach it. These two processes were associated with the basal ganglia and cerebellum, respectively, based on their functional roles in action selection and motor control. External and internal rewards were given for reaching the final goal and subgoal, respectively. To evaluate the effect of hierarchy, we compared this model with a non-hierarchical condition in which only the controller learned to reach the final goal.

Results
The hierarchical model solved the continuous two-dimensional maze task more efficiently than the non-hierarchical model. In the hierarchical condition, the agent gradually acquired subgoal-directed behavior and reduced the number of steps required to reach the final goal. In contrast, in the non-hierarchical condition, the controller alone had to learn actions toward the final goal from sparse external rewards, resulting in inefficient exploration and slower improvement. These results indicate that introducing hierarchy enabled the agent to learn an effective route to the goal more efficiently.


Discussion
The improved performance of the hierarchical model suggests that decomposing a long navigation task into shorter subgoal-directed processes can facilitate learning in a spiking RL framework. By selecting intermediate subgoals, the higher-level process reduced the difficulty of learning long action sequences from sparse rewards, while the lower-level process learned actions for reaching each subgoal. This supports the hypothesis that hierarchical organization may contribute to efficient behavior acquisition in the brain. The model also provides a computational framework for examining how brain-inspired action-selection and control processes can support hierarchical learning.


References
[1] Kulkarni, T. D., Narasimhan, K. R., Saeedi, A., & Tenenbaum, J. B. (2016). Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (arXiv:1604.06057). arXiv. https://doi.org/10.48550/arXiv.1604.06057

[2] Frémaux, N., Sprekeler, H., & Gerstner, W. (2013). Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons. PLoS Computational Biology, 9(4), e1003024. https://doi.org/10.1371/journal.pcbi.1003024

Acknowledgement
This study was supported by MEXT/JSPS KAKENHI Grant Number 22H05161.