Unsupervised Learning and Clustered Connectivity Enhance Reinforcement Learning in Spiking Neural Networks

Frontiers in Computational Neuroscience, vol. 15, pp. 543872

Abstract

Reinforcement learning is a paradigm that can account for how organisms learn to adapt their behavior in complex environments with sparse rewards. To partition an environment into discrete states, implementations in spiking neuronal networks typically rely on input architectures involving place cells or receptive fields specified ad hoc by the researcher. This is problematic as a model for how an organism can learn appropriate behavioral sequences in unknown environments, as it fails to account for the unsupervised and self-organized nature of the required representations. Additionally, this approach presupposes knowledge on the part of the researcher on how the environment should be partitioned and represented and scales poorly with the size or complexity of the environment. To address these issues and gain insights into how the brain generates its own task-relevant mappings, we propose a learning architecture that combines unsupervised learning on the input projections with biologically motivated clustered connectivity within the representation layer. This combination allows input features to be mapped to clusters; thus the network self-organizes to produce clearly distinguishable activity patterns that can serve as the basis for reinforcement learning on the output projections. On the basis of the MNIST and Mountain Car tasks, we show that our proposed model performs better than either a comparable unclustered network or a clustered network with static input projections. We conclude that the combination of unsupervised learning and clustered connectivity provides a generic representational substrate suitable for further computation.

Authors 3

1. Philipp Weidel Aachen Department of Computer Science 3-Software Engineering

   RWTH Aachen University · Forschungszentrum Jülich · Jülich Aachen Research Alliance

   Affiliation as printed

   Department of Computer Science 3 - Software Engineering, RWTH Aachen University, Aachen, Germany

   Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure-Function Relationship (JBI-1 / INM-10), Research Centre Jülich, Jülich, Germany

2. Renato Duarte corresponding

   Forschungszentrum Jülich · Jülich Aachen Research Alliance

   Affiliation as printed

   Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure-Function Relationship (JBI-1 / INM-10), Research Centre Jülich, Jülich, Germany

3. Abigail Morrison Aachen Department of Computer Science 3-Software Engineering

   RWTH Aachen University · Forschungszentrum Jülich · Jülich Aachen Research Alliance

   Affiliation as printed

   Department of Computer Science 3 - Software Engineering, RWTH Aachen University, Aachen, Germany

   Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure-Function Relationship (JBI-1 / INM-10), Research Centre Jülich, Jülich, Germany