Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search

Nature Communications, vol. 13, pp. 6284

Abstract

Lifelong on-device learning is a key challenge for machine intelligence, and this requires learning from few, often single, samples. Memory-augmented neural networks have been proposed to achieve the goal, but the memory module must be stored in off-chip memory, heavily limiting the practical use. In this work, we experimentally validated that all different structures in the memory-augmented neural network can be implemented in a fully integrated memristive crossbar platform with an accuracy that closely matches digital hardware. The successful demonstration is supported by implementing new functions in crossbars, including the crossbar-based content-addressable memory and locality sensitive hashing exploiting the intrinsic stochasticity of memristor devices. Simulations show that such an implementation can be efficiently scaled up for one-shot learning on more complex tasks. The successful demonstration paves the way for practical on-device lifelong learning and opens possibilities for novel attention-based algorithms that were not possible in conventional hardware.

Authors 12

1. Ruibin Mao

   University of Hong Kong

   Affiliation as printed

   Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

2. Bo Wen

   University of Hong Kong

   Affiliation as printed

   Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

3. Arman Kazemi

   University of Notre Dame · Hewlett-Packard (United States) · Hewlett Packard Enterprise (United States)

   Affiliation as printed

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

   Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, CA, USA

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA

   Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, USA

4. Yahui Zhao

   University of Hong Kong

   Affiliation as printed

   Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

5. Ann Franchesca Laguna

   University of Notre Dame · De La Salle University

   Affiliation as printed

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

   Department of Computer Technology, De La Salle University, Manila, Philippines

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA

6. Ngai Wong

   University of Hong Kong

   Affiliation as printed

   Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

7. Michael Thaddeus Niemier

   University of Notre Dame

   Affiliation as printed

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA

8. Xiaobo Sharon Hu

   University of Notre Dame

   Affiliation as printed

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

   Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA

9. Xia Sheng

   Hewlett-Packard (United States) · Hewlett Packard Enterprise (United States)

   Affiliation as printed

   Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, CA, USA

   Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, USA

10. Catherine E. Graves corresponding

    Hewlett-Packard (United States) · Hewlett Packard Enterprise (United States)

    Affiliation as printed

    Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, CA, USA. catherine.graves@hpe.com

    Hewlett Packard Labs, Hewlett Packard Enterprise, Milpitas, USA

11. John Paul Strachan corresponding Aachen

    Forschungszentrum Jülich · RWTH Aachen University

    Affiliation as printed

    Peter Grünberg Institut (PGI-14), Forschungszentrum Jülich GmbH, Jülich, Germany. j.strachan@fz-juelich.de

    RWTH Aachen University, Aachen, Germany. j.strachan@fz-juelich.de

    Peter Grünberg Institut (PGI-14), Forschungszentrum Jülich GmbH, Jülich, Germany

    RWTH Aachen University, Aachen, Germany

12. Can Li corresponding

    University of Hong Kong

    Affiliation as printed

    Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China. canl@hku.hk

    Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China