Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition

arXiv (Cornell University)

Abstract

Memristors provide a new chance for resource-efficient computation of neural models for natural language processing by enabling analog execution of vector-matrix-multiplication. Yet, computations on these devices are currently subject to larger distortion, both in weight programming and execution. In this work, we identify large output values of transformed positional encodings to cause major degradation within analog-to-digital conversion (ADC) as part of memristor-based computation. By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, we reduce the degradation of the execution by ~50% relative, while keeping the estimated energy consumption stable. Additionally, we investigate scenarios where the ADC cannot be modified. In that case the degradation can be reduced by ~30% relative after removing encoding-related linear transformations.

Authors 3

1. Benedikt Hilmes
   Affiliation as printed

   Apptek GmbH , Aachen , Germany

2. Nick Rossenbach Aachen Machine Learning and Human Language Technology Group Faculty of Computer Science

   RWTH Aachen University

   Affiliation as printed

   Machine Learning and Human Language Technology Group , Faculty of Computer Science , RWTH Aachen University , Aachen , Germany

3. Ralf Schlüter Aachen Machine Learning and Human Language Technology Group Faculty of Computer Science

   RWTH Aachen University

   Affiliation as printed

   Machine Learning and Human Language Technology Group , Faculty of Computer Science , RWTH Aachen University , Aachen , Germany