ScreenAgent

niuzaisheng/ScreenAgent is ranked 19,164 among these 8 peers; iagooar/qqqa is ranked 19,160.

ScreenAgent : A Computer Control Agent Driven by Visual Language Large Model View ScreenAgent Paper arxiv:2402. (2024-5-19) ScreenAgent Web Client released, a simpler way to experience controlling a desktop with a large model. We have built the ScreenAgent project, creating an environment for Visual Language Model agents (VLM Agent) to interact with real computer screens. In this environment, the agent can observe screenshots and manipulate the GUI by outputting mouse and keyboard operations. We have also designed an

automatic control process, which includes planning, action, and reflection stages, guiding the agent to continuously interact with the environment and complete multi-step tasks. In addition, we have built the ScreenAgent dataset, which collects screenshots and action sequences when completing various daily computer tasks. ScreenAgent Design Motivation To guide the VLM Agent to interact continuously with the computer screen, we have built a process that includes "planning-execution-reflection". In the planning phase, the Agent is asked to break down

the user task into subtasks. In the execution phase, the Agent will observe the screenshot and give specific mouse and keyboard actions to execute the subtasks. The controller will execute these actions and feed back the execution results to the Agent. In the reflection phase, the Agent will observe the execution results, judge the current state, and choose to continue execution, retry, or adjust the plan. This process will continue until the task is completed. Running Process We referred to

the VNC remote desktop connection protocol to design the action space of the Agent, which are all the most basic mouse and keyboard operations. Most of the mouse click operations require the Agent to give the exact screen coordinate position. Compared with calling specific APIs to complete tasks, this method is more universal and can be applied to various desktop operating systems and applications. Supported Action Types and Action Attributes Teaching the Agent to use a computer is not a

simple matter. It requires the Agent to have multiple comprehensive abilities such as task planning, image understanding, visual positioning, and tool use. For this reason, we manually annotated the ScreenAgent dataset. This dataset covers a variety of daily computer tasks, including file operations, web browsing, gaming entertainment and other scenarios. We build a session according to the above "planning-execution-reflection" process. Dataset Task Type Distribution The project mainly includes the following parts: Preparation Step 1, Prepare the desktop to

be controlled First, you need to prepare the desktop operating system to be controlled, where the VNC Server is installed, such as TightVNC. Or you can use a Docker container with a GUI. You can use the following command to pull and start this container: Please replace with your new VNC password, and with your clipboard service password. Since keyboard input of long strings of text or unicode characters relies on the clipboard, if the clipboard service is not enabled

, you can only input ASCII strings by pressing the keyboard in sequence, and you cannot input Chinese and other unicode