ConfASR: A Conformer Block Accelerator for Speech Recognition Optimized for Edge Devices

Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 147–153

Abstract

Attention-based neural networks, like transformers, have significantly improved automatic speech recognition (ASR). Adding convolution operations to transformers results in conformers, which enable better learning of local dependencies and reduce word error rate. We introduce ConfASR, the first conformer block accelerator designed for efficient ASR inference on edge devices. Our system is optimized both in terms of algorithms and hardware to support all transformer operations and additional features required for conformers, including depthwise-separable convolution and learned positional encoding. We propose a hardware-friendly normalization, shared scaling factors for non-linear functions, and an efficient dataflow with a shared MAC array that keeps all activations on chip. Implemented in a 22 nm FDSOI technology, ConfASR operates at 250 MHz with a power consumption of 359 mW, and a die area of $1.19 \mathrm{~mm}^{2}$. It performs over 900 times faster than necessary for real-time streaming requirements. This makes the architecture suitable not only for ASR but also for other transformer-based applications. ConfASR reduces latency by over $4 \times$ and power consumption by $16 \times$ during real-time use compared to previous solutions, while supporting more functionality.

Authors 7

1. Malte Wabnitz Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

2. M. Nilovic Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

3. Finn Scholz Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

4. Dominik Friedrich Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

5. Christian Lanius Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

6. Jie Lou Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany

7. Tobias Gemmeke Aachen Chair of Integrated Digital Systems and Circuit Design (IDS)

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Integrated Digital Systems and Circuit Design,Aachen,Germany