Contractivity of neural ODEs: An eigenvalue optimization problem

Mathematics of Computation, vol. 95, pp. 293–319

Abstract

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ordinary differential equations (ODEs). When looking at contractivity properties of a one-layer weight-tied neural ODE u ˙ ( t ) = σ ( A u ( t ) + b ) \dot {u}(t)=\sigma (Au(t)+b) (with u , b ∈ R n u,b \in \mathbb {R}^n , A A is a given n × n n \times n matrix, σ : R → R \sigma : \mathbb {R}\to \mathbb {R} denotes an activation function and for a vector z ∈ R n z \in \mathbb {R}^n , σ ( z ) ∈ R n \sigma (z) \in \mathbb {R}^n has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type D A D A , where D D is a diagonal matrix such that

Authors 4

1. Nicola Guglielmi

   Gran Sasso Science Institute

   Affiliation as printed

   Division of Mathematics, Gran Sasso Science Institute, viale Rendina 26-28, 67100, L’Aquila, Italy

2. Arturo De Marinis

   Gran Sasso Science Institute

   Affiliation as printed

   Division of Mathematics, Gran Sasso Science Institute, viale Rendina 26-28, 67100, L’Aquila, Italy

3. Anton Savostianov Aachen Computational Network Science

   Gran Sasso Science Institute · RWTH Aachen University

   Affiliation as printed

   Division of Mathematics, Gran Sasso Science Institute, viale Rendina 26-28, 67100, L’Aquila, Italy; Computational Network Science, RWTH Aachen University, Aachen, Germany

4. Francesco Tudisco

   Gran Sasso Science Institute · University of Edinburgh

   Affiliation as printed

   School of Mathematics, University of Edinburgh, James Clerk Maxwell Building, Peter Guthrie Tait Road, Edinburgh, EH9 3FD, UK; and Division of Mathematics, Gran Sasso Science Institute, viale Rendina 26-28, 67100, L’Aquila, Italy