Residual bayesian attention networks for uncertainty quantification in regression tasks

Scientific Reports, vol. 15, pp. 38279

Abstract

The demand for uncertainty quantification in modern sequence modeling tasks has prompted researchers to explore deep integration between Bayesian inference and Transformer architectures, but existing methods still face systematic engineering challenges in key technical aspects such as attention mechanism probabilization, residual connection uncertainty propagation, and epistemic-aleatoric uncertainty decoupling. This study proposes the Residual Bayesian Attention (RBA) framework, which achieves end-to-end probabilistic inference capabilities through three tightly coupled core components: Bayesian feedforward layers establish differentiable propagation mechanisms for parameter-level uncertainty, multi-layer residual Bayesian attention embeds radial basis function kernels into attention computation and introduces adaptive residual weights modeled by Beta distributions, and the Bayesian covariance construction module generates mathematically rigorous covariance representations through outer product operations and eigenvalue correction. Systematic evaluation on benchmark datasets covering six domains including engineering optimization, time series forecasting, and spatial modeling demonstrates that RBA achieves stable uncertainty quantification performance in medium-scale structured data scenarios, particularly exhibiting technical advantages in prediction interval calibration quality. Notably, through objective evaluation of challenging tasks such as complex physical systems, this study identifies the common technical boundaries of current deep learning methods in multi-physics coupled system modeling, providing important empirical insights for the development direction of this field. Therefore, RBA, as a systematic engineering integration framework of Bayesian inference and Transformer architecture, provides a methodological contribution with clearly defined applicability boundaries for principled uncertainty quantification in deep sequence modeling.

Authors 3

1. Youliang Chen

   UNSW Sydney · University of Shanghai for Science and Technology

   Affiliation as printed

   Department of Civil Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, 516 Jungong Rd, PR China

   Department of Engineering Geology, School of Civil and Environmental Engineering, University of New South Wales, Kensington, NSW, 2052, Australia

2. Wencan Guan corresponding Aachen Department of Engineering Geology and Hydrogeology

   RWTH Aachen University · University of Shanghai for Science and Technology

   Affiliation as printed

   Department of Civil Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, 516 Jungong Rd, PR China. guanwencan123@outlook.com

   Department of Engineering Geology and Hydrogeology, RWTH Aachen University, D-52064, Aachen, Lochnerstr. 4-20 Haus A, Germany. guanwencan123@outlook.com

   European Scoiety for Fuzzy Logic and Technology, Mieres, 33600, Manual Llaneza 32, Spain. guanwencan123@outlook.com

   Department of Civil Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, 516 Jungong Rd, PR China

3. Rafig Azzam Aachen Department of Engineering Geology and Hydrogeology

   RWTH Aachen University

   Affiliation as printed

   Department of Engineering Geology and Hydrogeology, RWTH Aachen University, D-52064, Aachen, Lochnerstr. 4-20 Haus A, Germany

   Department of Engineering Geology and Hydrogeology, RWTH Aachen University, D-52064, Aachen, Lochnerstr. 4–20 Haus A, Germany