Both Ends Count! Just How Good are LLM Agents at Text-to-“Big SQL”?

Abstract

Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world, Text-to-SQL systems are often embedded with Big Data workflows, such as large-scale data processing or interactive data analytics. We refer to this as “Text-to-Big SQL”. However, existing text-to-SQL benchmarks remain narrowly scoped and overlook the cost and performance implications that arise at scale. For instance, translation errors that are minor on small datasets lead to substantial cost and latency overheads as data scales, a relevant issue completely ignored by text-to-SQL metrics.

Authors 3

1. Germán T. Eizaguirre

   Universitat Rovira i Virgili

   Affiliation as printed

   Universitat Rovira i Virgili, Tarragona, Spain

2. Lars Tissen Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

3. Marc Sánchez‐Artigas

   Universitat Rovira i Virgili

   Affiliation as printed

   Universitat Rovira i Virgili, Tarragona, Spain