Toward Standardized Benchmarking of Search-Based Scenario Selection Methods in Autonomous System Validation

Journal of Marine Science and Engineering, vol. 14, pp. 1386

Abstract

Autonomous systems must demonstrate reliable safety and robustness even under rare, safety-critical conditions. Because the vast range of possible operating situations cannot be exhaustively tested, scenario-based testing has emerged as a structured approach to expose systems to representative and challenging situations. Search-based scenario selection methods (SBSSMs) algorithmically explore the scenario space to identify critical and informative cases. Yet, existing implementations are often tightly coupled to specific domains and simulation environments, hindering reproducibility, comparability, and generalizability across application areas. This paper introduces a benchmarking framework that addresses these limitations by providing a domain-independent environment for the systematic evaluation of SBSSMs. Building on empirical observations from maritime simulation studies and the literature, the framework generates synthetic test instances that mimic the structural patterns found there while abstracting away application-specific semantics. It supplies standardized performance metrics and reference implementations of key method classes, enabling direct comparison of new approaches with established ones under controlled, repeatable conditions. Illustrative demonstrations show how the benchmark reveals characteristic strengths and weaknesses of different methods across diverse structural settings. As a reproducible and extensible baseline, the framework promotes standardized performance assessment and the transfer of methodological advances from maritime navigation to domains such as automotive safety.

Authors 4

1. Daniel Autenrieth corresponding

   Atlas Elektronik (Germany)

   Affiliation as printed

   TKMS ATLAS ELEKTRONIK GmbH, Sebaldsbrücker Heerstraße 235, 28309 Bremen, Germany

2. Daniel Paland

   Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR)

   Affiliation as printed

   German Aerospace Center (DLR), Institute of Systems Engineering for Future Mobility, Escherweg 2, 26121 Oldenburg, Germany

3. Jan Mentjes

   Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR)

   Affiliation as printed

   German Aerospace Center (DLR), Institute of Systems Engineering for Future Mobility, Escherweg 2, 26121 Oldenburg, Germany

4. Adrian Zlocki Aachen fka GmbH

   Forschungsgesellschaft Kraftfahrwesen Aachen (Germany)

   Affiliation as printed

   fka GmbH, Steinbachstraße 7, 52074 Aachen, Germany