GA: A Comprehensive Survey on LLM-based GUI Agent

Abstract

The Graphical User Interface (GUI) is a visual method that allows users to interact with computers and mobile devices. Nowadays, users rely on GUI for completing some tasks, such as browsing web or using mobile applications. Users often meet some needs such as setting an alarm for 8:00 AM to wake them up and checking the weather for tomorrow. Some commercial agents have been integrated into users personal phones to help the user accomplish a series of basic tasks. Unfortunately, these commercial agents often relied on fixed templates or program scripts to ensure reliability. This also limited their functionality to some basic system applications. Recently large language models (LLMs) have made significant breakthroughs in natural language processing (NLP). Astonishingly, LLMs have demonstrated not only a strong ability to understand and generate text but also planning and reasoning capabilities. Some researchers have considered using LLMs as the agent’s brain, equipping these agents with corresponding capabilities. LLM-based agents are also being applied to help users automate tasks on their personal phones and computers. These agents often can understand the GUI environment on personal phones and computers, allowing them to make decisions to complete tasks. This is also the origin of the term “GUI Agent”. Our review surveys recent research on LLM-based GUI Agents. We summarize the capabilities of existing GUI Agents and also discuss the GUI Agent task automation pipeline. A comprehensive list of studies in this paper will be available at a GitHub repositories.

Authors 13

1. Longzhao Huang

   Beijing University of Posts and Telecommunications

   Affiliation as printed

   School of Artificial Intelligence, Beijing University of Posts and Telecommunications

2. Jun Liu

   Beijing University of Posts and Telecommunications

   Affiliation as printed

   School of Artificial Intelligence, Beijing University of Posts and Telecommunications

3. Changwei Wang

   Qilu University of Technology · Shandong University · Shandong Academy of Sciences · National Supercomputing Center in Jinan

   Affiliation as printed

   Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Key Laboratory of Computing Power Network and Information Security, Qilu University of Technology (Shandong Academy of Sciences)

   Shandong Fundamental Research Center for Computer Science, Shandong Provincial Key Laboratory of Computer Networks

4. Rongtao Xu

   Chinese Academy of Sciences · Institute of Automation

   Affiliation as printed

   Institute of Automation, The State Key Laboratory of Multimodal Artificial Intelligence Systems, Chinese Academy of Sciences

5. Wenhao Xu

   Beijing University of Posts and Telecommunications

   Affiliation as printed

   School of Artificial Intelligence, Beijing University of Posts and Telecommunications

6. Zhiwei Xu

   Shandong University

   Affiliation as printed

   School of Artificial Intelligence, Shandong University

7. Qi Zhang

   Tongji University

   Affiliation as printed

   Tongji University

8. Yu Zhang

   Tongji University

   Affiliation as printed

   Tongji University

9. Kexue Fu

   Tongji University · Qilu University of Technology · Shandong Academy of Sciences · National Supercomputing Center in Jinan

   Affiliation as printed

   Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Key Laboratory of Computing Power Network and Information Security, Qilu University of Technology (Shandong Academy of Sciences)

   Tongji University

10. Longxiang Gao

    Qilu University of Technology · Shandong Academy of Sciences · National Supercomputing Center in Jinan

    Affiliation as printed

    Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Key Laboratory of Computing Power Network and Information Security, Qilu University of Technology (Shandong Academy of Sciences)

11. Lei Zhang

    Chinese Academy of Sciences · Institute of Computing Technology

    Affiliation as printed

    Institute of Computing Technology, Chinese Academy of Sciences

12. Li Guo

    Beijing University of Posts and Telecommunications

    Affiliation as printed

    School of Artificial Intelligence, Beijing University of Posts and Telecommunications

13. Shibiao Xu

    Beijing University of Posts and Telecommunications

    Affiliation as printed

    School of Artificial Intelligence, Beijing University of Posts and Telecommunications