GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Abstract
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
Authors 68
-
Affiliation as printed
Google Research , 12 Grammarly , 13
-
Bangladesh University of Engineering and Technology
Affiliation as printed
Bangladesh University of Engineering and Technology ,
-
Affiliation as printed
Mphasis NEXT Labs ,
-
Affiliation as printed
New York University , 26
-
Affiliation as printed
Amazon Alexa AI ,
-
Affiliation as printed
Carnegie Mellon University ,
-
Affiliation as printed
University of London ,
-
Rensselaer Polytechnic Institute
Affiliation as printed
Rensselaer Polytechnic Institute ,
-
Hong Kong University of Science and Technology
Affiliation as printed
The Hong Kong University of Science and Technology ,
-
Allen Institute for Artificial Intelligence
Affiliation as printed
Allen Institute for AI ,
-
Affiliation as printed
Tianjin University , 42
-
Affiliation as printed
University of Michigan ,
-
Microsoft (United States) · IBM (United States) · Massachusetts Institute of Technology
Affiliation as printed
MIT-IBM Watson AI Lab , 21 Masakhane , 22 Meta AI , 23 Microsoft ,
-
Affiliation as printed
University of Pennsylvania ,
-
Affiliation as printed
Tianjin University , 42
-
Affiliation as printed
Amazon Alexa AI ,
-
Affiliation as printed
Edinburgh Napier University ,
-
Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)
Affiliation as printed
HKUST , 14
-
Affiliation as printed
Stanford University , 35 Symanto Research , 36 Syracuse
-
Affiliation as printed
Columbia University ,
-
Affiliation as printed
University of Turku ,
-
Hong Kong University of Science and Technology
Affiliation as printed
The Hong Kong University of Science and Technology ,
-
Microsoft (United States) · IBM (United States) · Massachusetts Institute of Technology
Affiliation as printed
IBM Research , 17 IIIT Hyderabad ,
MIT-IBM Watson AI Lab , 21 Masakhane , 22 Meta AI , 23 Microsoft ,
-
Affiliation as printed
Winterlight Labs , 51 Yale University , 52 trivago N.V .,
-
Affiliation as printed
University of Turku ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)
Affiliation as printed
HKUST , 14
-
Affiliation as printed
Google Research , 12 Grammarly , 13
-
Affiliation as printed
New York University , 26
-
University of California, Santa Cruz
Affiliation as printed
University of California , Santa Cruz ,
-
Affiliation as printed
Emory University ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Affiliation as printed
University of Edinburgh ,
-
Technische Universität Darmstadt
Affiliation as printed
Technical University of Darmstadt ,
-
Affiliation as printed
University of Pennsylvania ,
-
Affiliation as printed
Google Research , 12 Grammarly , 13
-
Affiliation as printed
University of Helsinki ,
-
Affiliation as printed
The Ohio State University ,
-
Affiliation as printed
Google Research , 12 Grammarly , 13
-
Affiliation as printed
École Polytechnique
-
Nico Daheim Aachen
Affiliation as printed
RWTH Aachen University ,
-
Allen Institute for Artificial Intelligence
Affiliation as printed
Allen Institute for AI ,
-
Affiliation as printed
Charles University , 6 Chattermill ,
-
Affiliation as printed
Carnegie Mellon University ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Affiliation as printed
University of London ,
-
Affiliation as printed
University of Edinburgh ,
-
Affiliation as printed
Johns Hopkins University ,
-
Bangladesh University of Engineering and Technology
Affiliation as printed
Bangladesh University of Engineering and Technology ,
-
Affiliation as printed
University of Edinburgh ,
-
Affiliation as printed
University of London ,
-
Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)
Affiliation as printed
HKUST , 14
-
Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)
Affiliation as printed
HKUST , 14
-
Affiliation as printed
Harvard University , 15 Hugging Face ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Affiliation as printed
University of London ,
-
Affiliation as printed
Pompeu Fabra University , 29 Queen Mary
-
Bangladesh University of Engineering and Technology
Affiliation as printed
Bangladesh University of Engineering and Technology ,
-
Affiliation as printed
Tianjin University , 42
-
Luleå University of Technology
Affiliation as printed
Luleå University of Technology ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Affiliation as printed
University , 37 TU Kaiserslautern ,
-
Affiliation as printed
Google Research , 12 Grammarly , 13
-
Affiliation as printed
University of Michigan ,
-
Affiliation as printed
Harvard University , 15 Hugging Face ,
-
Affiliation as printed
University of London ,
-
Affiliation as printed
IBM Research , 17 IIIT Hyderabad ,
Tsinghua University , 43
Cited by 3 stored of 3
3 results
No patents citing this paper on Lens.org (checked 2026-10-06).