GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pp. 266–281

Abstract

Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2022.

Authors 68

1. Sebastian Gehrmann corresponding

   Google (United States)

   Affiliation as printed

   Google Research , 12 Grammarly , 13

   Google Research, 12 Grammarly,

2. Abhik Bhattacharjee

   Bangladesh University of Engineering and Technology

   Affiliation as printed

   Bangladesh University of Engineering and Technology ,

   Bangladesh University of Engineering and Technology,

3. Abinaya Mahendiran

   Microsoft (United States) · IBM (United States) · Massachusetts Institute of Technology

   Affiliation as printed

   MIT-IBM Watson AI Lab , 21 Masakhane , 22 Meta AI , 23 Microsoft , Mphasis NEXT Labs ,

   Mphasis NEXT Labs,

4. Alex Wang

   Microsoft (United States) · IBM (United States) · New York University · Massachusetts Institute of Technology

   Affiliation as printed

   MIT-IBM Watson AI Lab , 21 Masakhane , 22 Meta AI , 23 Microsoft , Mphasis NEXT Labs ,

   New York University,

5. Alexandros Papangelis

   Amazon (United States)

   Affiliation as printed

   Amazon Alexa AI ,

   Amazon Alexa AI,

6. Aman Madaan

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University ,

   Carnegie Mellon University,

7. Angelina McMillan-Major

   Universitat Pompeu Fabra · International Institute of Information Technology, Hyderabad

   Affiliation as printed

   Pompeu Fabra University , 29 Queen Mary

   IIIT Hyderabad,

8. Bernd Bohnet

   Google (United States)

   Affiliation as printed

   Google Research , 12 Grammarly , 13

   Google Research, 12 Grammarly,

9. Bingsheng Yao Aachen

   Rensselaer Polytechnic Institute · RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University ,

   Rensselaer Polytechnic Institute,

10. Bryan Wilie

    Hong Kong University of Science and Technology · Technische Universität Darmstadt

    Affiliation as printed

    Technical University of Darmstadt ,

    The Hong Kong University of Science and Technology,

11. Chandra Bhagavatula

    Allen Institute for Artificial Intelligence

    Affiliation as printed

    Allen Institute for AI ,

    Allen Institute for AI,

12. Chaobin You

    Tianjin University · The Ohio State University

    Affiliation as printed

    The Ohio State University ,

    Tianjin University,

13. Cristina Gârbacea

    University of Michigan

    Affiliation as printed

    University of Michigan ,

    University of Michigan,

14. Dakuo Wang

    Northeastern University · IBM (United States) · Luleå University of Technology · Massachusetts Institute of Technology

    Affiliation as printed

    Luleå University of Technology ,

    MIT-IBM Watson AI Lab, 21 Masakhane, 22 Meta AI, 23 Microsoft,

    Northeastern University,

15. Daniel Deutsch

    University of Pennsylvania

    Affiliation as printed

    University of Pennsylvania ,

    University of Pennsylvania,

16. Deyi Xiong

    Tianjin University · The Ohio State University

    Affiliation as printed

    The Ohio State University ,

    Tianjin University,

17. Di Jin

    Amazon (United States)

    Affiliation as printed

    Amazon Alexa AI ,

    Amazon Alexa AI,

18. Dimitra Gkatzia

    Edinburgh Napier University

    Affiliation as printed

    Edinburgh Napier University ,

    Edinburgh Napier University,

19. Dragomir Radev

    Hong Kong University of Science and Technology · Yale University · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

    Yale University, 51 trivago N.V.,

20. Esin Durmus

    Robert Gordon University · Stanford University

    Affiliation as printed

    Robert Gordon University , 33 Salesforce Research ,

    Stanford University,

21. Faisal Ladhak

    Columbia University

    Affiliation as printed

    Columbia University ,

    Columbia University,

22. Filip Ginter

    University of Turku

    Affiliation as printed

    University of Turku ,

    University of Turku,

23. Genta Indra Winata

    Hong Kong University of Science and Technology · Technische Universität Darmstadt

    Affiliation as printed

    Technical University of Darmstadt ,

    The Hong Kong University of Science and Technology,

24. Hendrik Strobelt

    IBM (United States) · Harvard University · Luleå University of Technology · Massachusetts Institute of Technology

    Affiliation as printed

    Harvard University , 15 Hugging Face ,

    Luleå University of Technology ,

    IBM Research,

    MIT-IBM Watson AI Lab, 21 Masakhane, 22 Meta AI, 23 Microsoft,

25. Jekaterina Novikova

    Yale University

    Affiliation as printed

    Winterlight Labs , 50 Yale University , 51 trivago N.V .,

    Winterlight Labs,

26. Jenna Kanerva

    University of Turku

    Affiliation as printed

    University of Turku ,

    University of Turku,

27. Jenny Chim

    Queen Mary University of London · Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

    Queen Mary University of London,

28. Jiawei Zhou

    Harvard University · Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

    Harvard University, 15 Hugging Face,

29. Jordan Clive

    Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

    Chattermill,

30. Joshua Maynez

    Google (United States)

    Affiliation as printed

    Google Research , 12 Grammarly , 13

    Google Research, 12 Grammarly,

31. João Sedoc

    Microsoft (United States) · IBM (United States) · New York University · Massachusetts Institute of Technology

    Affiliation as printed

    MIT-IBM Watson AI Lab , 21 Masakhane , 22 Meta AI , 23 Microsoft , Mphasis NEXT Labs ,

    New York University,

32. Juraj Juraska

    University of California, Santa Cruz · University of Aberdeen

    Affiliation as printed

    University of Aberdeen ,

    University of California, Santa Cruz,

33. Kaustubh Dhole

    Emory University

    Affiliation as printed

    Emory University ,

    Emory University,

34. Khyathi Raghavi Chandu

    Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

35. Laura Perez Beltrachini

    University of Edinburgh

    Affiliation as printed

    University of Edinburgh,

36. Leonardo Ribeiro

    University of Kaiserslautern · Technische Universität Darmstadt

    Affiliation as printed

    University , 36 TU Kaiserslautern ,

    Technical University of Darmstadt,

37. Zhang Li

    University of Pennsylvania

    Affiliation as printed

    University of Pennsylvania ,

    University of Pennsylvania,

38. Mahim Pushkarna

    Google (United States)

    Affiliation as printed

    Google Research , 12 Grammarly , 13

    Google Research, 12 Grammarly,

39. Mathias Creutz

    University of Helsinki

    Affiliation as printed

    University of Helsinki ,

    University of Helsinki,

40. Mihir Kale

    Google (United States)

    Affiliation as printed

    Google Research , 12 Grammarly , 13

    Google Research, 12 Grammarly,

41. Moussa Kamal Eddine

    École Polytechnique

    Affiliation as printed

    École Polytechnique

42. Nico Daheim Aachen

    University of London · RWTH Aachen University

    Affiliation as printed

    University of London ,

    RWTH Aachen University,

43. Nishant Subramani

    Allen Institute for Artificial Intelligence

    Affiliation as printed

    Allen Institute for AI ,

    Allen Institute for AI,

44. Ondřej Dušek

    Charles University

    Affiliation as printed

    Charles University , 6 Chattermill ,

    Charles University,

45. Paul Pu Liang

    Carnegie Mellon University

    Affiliation as printed

    Carnegie Mellon University ,

    Carnegie Mellon University,

46. Pawan Sasanka Ammanamanchi

    Universitat Pompeu Fabra

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

47. Qi Zhu

    Universitat Pompeu Fabra · Tsinghua University

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

    Tsinghua University,

48. Ratish Puduppully

    University of Edinburgh

    Affiliation as printed

    University of Edinburgh ,

    University of Edinburgh,

49. Reno Kriz

    Johns Hopkins University

    Affiliation as printed

    IBM Research , 17 IIIT Hyderabad ,

    Johns Hopkins University,

50. Rifat Shahriyar

    Bangladesh University of Engineering and Technology

    Affiliation as printed

    Bangladesh University of Engineering and Technology ,

    Bangladesh University of Engineering and Technology,

51. Ronald Cardenas

    University of Edinburgh

    Affiliation as printed

    University of Edinburgh ,

    University of Edinburgh,

52. Saad Mahamood

    Universitat Pompeu Fabra · International Institute of Information Technology, Hyderabad

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

    IIIT Hyderabad,

53. Salomey Osei

    Universitat Pompeu Fabra · International Institute of Information Technology, Hyderabad

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

    IIIT Hyderabad,

54. Samuel Cahyawijaya

    Universitat Pompeu Fabra · Hong Kong University of Science and Technology

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

    HKUST,

55. Sanja Štajner

    Northeastern University · Universitat Pompeu Fabra

    Affiliation as printed

    Northeastern University , 27 Orange Labs ,

    Pompeu Fabra University,

56. Sébastien Montella

    Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

    Affiliation as printed

    HKUST , 14

    Orange Labs,

57. Shailza Jolly

    University of Kaiserslautern · University of California, Santa Cruz

    Affiliation as printed

    University of California , Santa Cruz ,

    TU Kaiserslautern,

58. Simon Mille

    Northeastern University · Universitat Pompeu Fabra

    Affiliation as printed

    Northeastern University , 27 Orange Labs ,

    Pompeu Fabra University,

59. Tahmid Hasan

    Bangladesh University of Engineering and Technology

    Affiliation as printed

    Bangladesh University of Engineering and Technology ,

    Bangladesh University of Engineering and Technology,

60. Tianhao Shen

    Tianjin University · The Ohio State University

    Affiliation as printed

    The Ohio State University ,

    Tianjin University,

61. Tosin Adewumi

    Johns Hopkins University · Luleå University of Technology

    Affiliation as printed

    Johns Hopkins University ,

    Luleå University of Technology,

62. Vikas Raunak

    Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · International Institute of Information Technology, Hyderabad

    Affiliation as printed

    HKUST , 14

    IIIT Hyderabad,

63. Vipul Raheja

    New York University · International Institute of Information Technology, Hyderabad

    Affiliation as printed

    New York University , 26

    IIIT Hyderabad,

64. Vitaly A. Nikolaev

    Google (United States)

    Affiliation as printed

    Google Research , 12 Grammarly , 13

    Google Research, 12 Grammarly,

65. Yacine Jernite

    Universitat Pompeu Fabra · International Institute of Information Technology, Hyderabad

    Affiliation as printed

    Pompeu Fabra University , 29 Queen Mary

    IIIT Hyderabad,

66. Yisi Sang

    Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · Syracuse University

    Affiliation as printed

    HKUST , 14

    Syracuse University,

67. Yixin Liu

    Hong Kong University of Science and Technology · Yale University · The Hong Kong University of Science and Technology (Guangzhou) · Stanford University

    Affiliation as printed

    HKUST , 14

    Stanford University , 35 Syracuse

    Yale University, 51 trivago N.V.,

68. Yufang Hou

    IBM (United States) · Harvard University

    Affiliation as printed

    Harvard University , 15 Hugging Face ,

    IBM Research,