Shifting-based Optimizable Linear Relaxations for General Activation Functions

arXiv (Cornell University)

Abstract

The use of neural networks (NNs) is rapidly increasing, including in safety- and security-critical domains. To provide formal guarantees about NN behavior, many verification methods rely on optimizable linear relaxations of activation functions. However, existing techniques depend on hand-crafted relaxations for each activation function. Extension to state-of-the-art activation functions therefore requires substantial manual effort. In contrast, our approach SLiR (Shifting-based Linear Relaxations) is broadly applicable, requiring only a Lipschitz constant or a set of critical points. SLiR parameterizes relaxations by their slope and computes the corresponding offset via a shifting procedure that ensures sound upper and lower bounds over the input domain, enabling efficient optimization while maintaining correctness. Our experiments show that SLiR produces tight relaxations across a wide range of practical activation functions and enables verification of up to 7.8x more properties compared to state-of-the-art methods.

Authors 4

1. Philipp Kern

   Karlsruhe Institute of Technology

   Affiliation as printed

   Karlsruhe Institute of Technology

2. László Antal Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

3. Erika Ábrahám Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

4. Carsten Sinz

   Karlsruhe University of Applied Sciences

   Affiliation as printed

   Karlsruhe University of Applied Sciences