Vision-Conditioned Variational Bayesian Last Layer Dynamics Models

arXiv (Cornell University)

Abstract

Agile control of robotic systems often requires anticipating how the environment affects system behavior. For example, a driver must perceive the road ahead to anticipate available friction and plan actions accordingly. Achieving such proactive adaptation within autonomous frameworks remains a challenge, particularly under rapidly changing conditions. Traditional modeling approaches often struggle to capture abrupt variations in system behavior, while adaptive methods are inherently reactive and may adapt too late to ensure safety. We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment. The model first learns nominal vehicle dynamics and is then fine-tuned with feature-wise affine transformations of latent features, enabling context-aware dynamics prediction. The resulting model is integrated into an optimal controller for vehicle racing. We validate our method on a Lexus LC500 racing through water puddles. With vision-conditioning, the system completed all 12 attempted laps under varying conditions. In contrast, all baselines without visual context consistently lost control, demonstrating the importance of proactive dynamics adaptation in high-performance applications.

Authors 6

1. Paul Brunzema Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   department of Mechanical Engineering , Aachen University , Germany

2. Thomas W. K. Lew Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   Toyota Research Institute , 4440 El Camino Real , CA , USA

   department of Mechanical Engineering , Aachen University , Germany

3. Ray Zirui Zhang Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   department of Mechanical Engineering , Aachen University , Germany

4. Takeru Shirasawa Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   department of Mechanical Engineering , Aachen University , Germany

5. John K. Subosits Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   department of Mechanical Engineering , Aachen University , Germany

6. Marcus Greiff Aachen Department of Mechanical Engineering

   FH Aachen

   Affiliation as printed

   department of Mechanical Engineering , Aachen University , Germany