pi-computer-use
pi-computer-use lets AI agents use desktop apps on macOS, Windows, and Linux.
The macOS helper requires macOS 14 or newer; the macOS reference maps each platform claim to Apple documentation and the implementation.
An agent can look at an app window, understand the buttons and text inside it, and perform actions like clicking, typing, scrolling, and waiting for something to change. This is useful when the agent needs to work with a normal desktop app instead of an API, a terminal command, or a file.
New to computer use? Start with: Wait, what exactly is Computer Use?
What this package does
This is a Pi extension. After installation, Pi agents get tools for:
- finding open apps and windows
- observing what is visible in a window
- searching the visible interface for text, buttons, and controls
- inspecting parts of the interface in more detail
- clicking, typing, scrolling, and pressing UI controls
- waiting for UI changes
In short: it gives an agent a controlled way to operate desktop software.
What this package is not
pi-computer-use is not a replacement for app APIs or MCP servers. If an app has a reliable direct integration, use that first.
Computer use is most helpful when the only available interface is the app on screen.
Install
pi install npm:@injaneity/pi-computer-use
Start Pi and complete the platform setup flow.
On macOS, the runtime installs the helper per user at ~/Applications/pi-computer-use.app and requests Accessibility and Screen Recording. Existing writable system-wide installs remain at /Applications/pi-computer-use.app. Enable both grants for that app in System Settings, then choose Recheck. The macOS reference documents the exact APIs and avoids relying on version-specific pane labels.
On Windows, use an interactive desktop session. Windows support uses the platform accessibility APIs and does not use the macOS helper app.
On Linux, run Pi inside the target user's graphical session with a working AT-SPI2 accessibility bus. AT-SPI semantic operations remain background-first. X11 additionally supports EWMH window metadata/focus, window capture, and policy-gated XTEST physical input; strict headless/background policies never use focus or XTEST. Native Wayland remains semantic-only; diagnostics reads portal capability properties without creating a session, and interactive portal use is disabled. See Linux support for the exact capability matrix and portal status.
Use /computer-use inside Pi to show the active configuration and where it came from.
Main tools
find_rootsobserve_uisearch_uiexpand_uiinspect_uiact_uiread_textwait_for
Follow the getting-started tutorial, then use the generated agent state and action contract for operating behavior or the generated tool schema reference for exact parameters.
Documentation
The docs follow Diátaxis and render through the Next.js/Fumadocs app in docs/. Run npm install --prefix docs once, then npm run docs:dev and open http://localhost:8080/docs.
Development status
The architecture is centered on immutable, state-scoped observations. Desktop surfaces and CDP pages form one multi-root forest; progressive outline queries remain cached, while live work is ordered per physical resource so independent roots can run in parallel. act_ui accepts one or more intent steps, preserves focus across dependent input, verifies delivery, recovers safely, stores one complete successor state, and returns a compact diff when identity confidence allows. Older direct tools such as screenshot, click, set_text, and computer_actions are no longer part of the public extension surface.
License
MIT