EASTER: Learning to Split Transformers at the Edge Robustly

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 43, pp. 3626–3637

Abstract

Prevalent large transformer models present significant computational challenges for resource-constrained devices at the Edge. While distributing the workload of deep learning models across multiple edge devices has been extensively studied, these works typically overlook the impact of failures of edge devices. Unpredictable failures, due to, e.g., connectivity issues or discharged batteries, can compromise the reliability of inference serving at the Edge. In this article, we introduce a novel methodology, called EASTER, designed to learn robust distribution strategies for transformer models against device failures that consider the tradeoff between robustness (i.e., maintaining model functionality against failures) and resource utilization (considering memory usage and computations). We evaluate EASTER with three representative transformers—ViT, GPT-2, and Vicuna—under device failures. Our results demonstrate EASTER’s efficiency in memory usage, and possible end-to-end latency improvement for inference across multiple edge devices while preserving model accuracy as much as possible under device failures.

Authors 5

1. Xiaotian Guo Aachen Leiden Institute of Advanced Computer Science

   Leiden University · University of Amsterdam

   Affiliation as printed

   Informatics Institute, University of Amsterdam, Amsterdam, XH, The Netherlands

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, CA, The Netherlands

2. Quan Jiang

   Nanjing Agricultural University

   Affiliation as printed

   Computer Science and Technology Department, Nanjing Agricultural University, Nanjing, China

3. Yixian Shen

   University of Amsterdam

   Affiliation as printed

   Informatics Institute, University of Amsterdam, Amsterdam, XH, The Netherlands

4. Andy D. Pimentel

   University of Amsterdam

   Affiliation as printed

   Informatics Institute, University of Amsterdam, Amsterdam, XH, The Netherlands

5. Todor P. Stefanov Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, CA, The Netherlands