On-Chip Learning via Transformer In-Context Learning

IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp. 1–5

Abstract

Autoregressive decoder-only transformers have become key components for scalable sequence processing and generation models. However, the transformer’s self-attention mechanism requires transferring prior token projections from the main memory at each time step (token), thus severely limiting their performance on conventional processors. Self-attention can be viewed as a dynamic feed-forward layer whose matrix is input sequence-dependent, similar to the result of local synaptic plasticity. Using this insight, we present a neuromorphic decoder-only transformer model that utilizes an on-chip plasticity processor to compute self-attention. Interestingly, the training of transformers enables them to "learn" the input context during inference. We demonstrate this in-context learning ability of transformers on the Loihi 2 processor by solving a few-shot classification problem. With this, we emphasize the importance of pretrained models, especially their ability to find simple, local, backpropagation-free learning rules enabling on-chip learning and adaptation in a hardware-friendly manner.

Authors 2

1. Jan Finkbeiner Aachen Fakultät fr Elektrotechnik und Informationstechnik

   RWTH Aachen University

   Affiliation as printed

   Fakultät fr Elektrotechnik und Informationstechnik, RWTH Aachen,Aachen,Germany,52074

2. Emre Neftci Aachen Fakultät fr Elektrotechnik und Informationstechnik

   RWTH Aachen University

   Affiliation as printed

   Fakultät fr Elektrotechnik und Informationstechnik, RWTH Aachen,Aachen,Germany,52074