Introduction Learning through trial and error (reinforcement learning, RL) enables animals to adapt their behavior in dynamic environments. Updating behavior based on reward prediction errors requires balancing stability and flexibility: learners should avoid overreacting to noise while remaining sensitive to genuine environ- mental changes [1]. Here, we investigated how mice adjust their learning parameters under uncertainty and developed a meta-reinforcement learning framework to account for this adaptation [2].
Methods We manipulated two sources of environmental uncertainty: reward probabilities, which determine out- come stochasticity, and the frequency of contingency changes, which determines volatility [3]. We first de- rived theoretical predictions from a standard RL model by systematically varying stochasticity and volatility to identify reward-maximizing parameter values. We then compared these predictions with mouse behavior in a binary operant task using intracranial self-stimulation [4], ensuring stable motivation across animals, sessions, and thousands of trials. Behavioral data were fitted with a classical RL model and used to constrain a meta-learning procedure.
Results Simulations predicted that the optimal learning rate should increase with environmental volatility but decrease with stochasticity, while the optimal decision parameter (exploitation/exploration trade-off) should decrease with both factors. Consistent with these predictions, fitted learning rates in mice varied with both volatility and stochasticity. In contrast, decision parameter remained stable across conditions.
To account for these results, we developed a meta-RL model in which mice estimate stochasticity from reward prediction errors and track volatility using a simple heuristic inspired by inference models. This model provided the best explanation of behavioral data.
Discussion Together, these results indicate that mice dynamically adjust learning rates in response to environmental uncertainty using computationally simple estimates of volatility and stochasticity. This framework provides a tractable approach for investigating the neural mechanisms underlying adaptive learning
References [1] Kenji Doya. Modulators of decision making. Nature neuroscience, 11(4):410–416, 2008. [2] Nathaniel D Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroscience, 8(12):1704–1711, 2005. [3] P Piray and ND Daw. A model for learning based on the joint estimation of stochasticity and volatility. Nature Communication, 1(12):6587, 2021. [4] William A Carlezon Jr and Elena H Chartoff. Intracranial self-stimulation (icss) in rodents to study the neurobiology of motivation. Nature protocols, 2(11):2987–2995, 2007
Acknowledgement The authors acknowledge the support of La Fondation pour la Recherche Médicale. We also thank Jacques Gautrais (CBI, Toulouse) for his valuable advice and discussions regarding analysis codes.
