Accelerating Deep Learning Inference in Constrained Embedded Devices Using Hardware Loops and a Dot Product Unit

IEEE Access, vol. 8, pp. 165913–165926

Abstract

Deep learning algorithms have seen success in a wide variety of applications, such as machine translation, image and speech recognition, and self-driving cars. However, these algorithms have only recently gained a foothold in the embedded systems domain. Most embedded systems are based on cheap microcontrollers with limited memory capacity, and, thus, are typically seen as not capable of running deep learning algorithms. Nevertheless, we consider that advancements in compression of neural networks and neural network architecture, coupled with an optimized instruction set architecture, could make microcontroller-grade processors suitable for specific low-intensity deep learning applications. We propose a simple instruction set extension with two main components-hardware loops and dot product instructions. To evaluate the effectiveness of the extension, we developed optimized assembly functions for the fully connected and convolutional neural network layers. When using the extensions and the optimized assembly functions, we achieve an average clock cycle count decrease of 73% for a small scale convolutional neural network. On a per layer base, our optimizations decrease the clock cycle count for fully connected layers and convolutional layers by 72% and 78%, respectively. The average energy consumption per inference decreases by 73%. We have shown that adding just hardware loops and dot product instructions has a significant positive effect on processor efficiency in computing neural network functions.

Authors 7

1. Jure Vreča

   University of Maribor

   Affiliation as printed

   Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia

2. Karl J. X. Sturm Aachen Institute for Communication Technologies and Embedded Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Communication Technologies and Embedded Systems, RWTH Aachen University, Aachen, Germany

3. Ernest Gungl

   University of Maribor

   Affiliation as printed

   Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia

4. Farhad Merchant Aachen Institute for Communication Technologies and Embedded Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Communication Technologies and Embedded Systems, RWTH Aachen University, Aachen, Germany

5. Paolo Bientinesi

   Umeå University

   Affiliation as printed

   Department of Computing Science, Umeå University, Umeå, Sweden

6. Rainer Leupers Aachen Institute for Communication Technologies and Embedded Systems

   RWTH Aachen University

   Affiliation as printed

   Institute for Communication Technologies and Embedded Systems, RWTH Aachen University, Aachen, Germany

7. Zmago Brezočnik

   University of Maribor

   Affiliation as printed

   Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia