On-Chip Learning via Transformer In-Context Learning

arXiv (Cornell University)

Abstract

Autoregressive decoder-only transformers have become key components for scalable sequence processing and generation models. However, the transformer's self-attention mechanism requires transferring prior token projections from the main memory at each time step (token), thus severely limiting their performance on conventional processors. Self-attention can be viewed as a dynamic feed-forward layer, whose matrix is input sequence-dependent similarly to the result of local synaptic plasticity. Using this insight, we present a neuromorphic decoder-only transformer model that utilizes an on-chip plasticity processor to compute self-attention. Interestingly, the training of transformers enables them to ``learn'' the input context during inference. We demonstrate this in-context learning ability of transformers on the Loihi 2 processor by solving a few-shot classification problem. With this we emphasize the importance of pretrained models especially their ability to find simple, local, backpropagation free, learning rules enabling on-chip learning and adaptation in a hardware friendly manner.

Authors 2

1. Jan Finkbeiner Aachen Fakultät für Elektrotechnik und Informationstechnik

   RWTH Aachen University · Forschungszentrum Jülich

   Affiliation as printed

   Fakultät für Elektrotechnik und Informationstechnik , RWTH Aachen , Aachen , 52074 , Germany

   Peter Grünberg Institut , Forschungszentrum Jülich GmbH , Jülich , 52425 , Germany

2. Emre Neftci Aachen Fakultät für Elektrotechnik und Informationstechnik

   RWTH Aachen University · Forschungszentrum Jülich

   Affiliation as printed

   Fakultät für Elektrotechnik und Informationstechnik , RWTH Aachen , Aachen , 52074 , Germany

   Peter Grünberg Institut , Forschungszentrum Jülich GmbH , Jülich , 52425 , Germany