Vercel's agent-desktop lets AI agents drive Mac, Windows and Linux apps through the accessibility tree, not screenshots

agent-desktop, an open-source tool from Vercel Labs, gives AI agents a way to operate ordinary native desktop apps, plus system surfaces like the macOS Dock and menu bar, by reading the operating system's accessibility tree instead of guessing from screenshots. It topped a daily GitHub trending tracker on November 3 with 2,275 new stars in a day, and the repo page showed about 10.9K stars, 726 forks and an Apache-2.0 license when we checked.

Why it matters: most "computer use" agents today take a screenshot, ask a vision model where to click, and hope the pixels haven't moved. That is slow, expensive in tokens and brittle. agent-desktop takes the approach browser automation tools like Playwright take for the web: expose the app's real structure, let the agent act on named elements, then verify. If you're building an agent that has to touch native software, this is a cheaper, more deterministic starting point.

How it works

Every call is a one-shot CLI command that returns JSON, according to the README:

1. snapshot reads the app's accessibility tree and returns each actionable element with a ref tied to that snapshot (for example @s8f3k2p9:e8). A bare @e8 is rejected on purpose, so an agent can't act on an element from an old view.
2. Act on a ref with commands such as click, type, set-value, select, toggle or scroll.
3. Snapshot again to confirm the result. If the UI changed underneath, actions fail with STALE_REF or AMBIGUOUS_TARGET rather than clicking the wrong thing.

Errors carry a machine-readable code, a disposition and often a suggestion, and the README tells agents to retry only when the disposition says a retry is safe. The planning loop stays in your code: as a DEV Community write-up puts it, the tool doesn't plan, keep state between calls or embed a model.

The README lists 60 command names, 56 of them working, grouped into observe, act, keyboard and mouse, apps and windows, notifications, clipboard, waits and sessions. Each release also ships a C library (libagent_desktop_ffi) so programs can call it in-process; the README includes a Python ctypes example.

Headless by default

Actions on refs go through accessibility APIs, so by default they don't move your mouse pointer or steal window focus, and headless calls refuse synthesized keyboard and mouse input. You opt into physical input with a global --headed flag. That matters if an agent is working on the same machine you're using.

Platform support

Builds cover x64 and ARM64. Cross-platform support is the big change: when the project was posted to Hacker News about six months ago, its author said macOS was ready and Windows and Linux were on the roadmap, and commenters criticised the "cross-platform" label.

Limits to know

• It is only as good as the app's accessibility tree. The DEV write-up notes that many Win32 apps, custom-drawn controls, games and some Electron apps expose thin or missing trees. Hacker News commenters raised the same problem with custom-rendered dialogs and lazy-loaded content. A screenshot command exists as a fallback.
• Token savings are a claim. The project says its "skeleton then drill down" pattern uses 78–96% fewer tokens on dense apps; the DEV author flags that this hasn't been independently verified and depends on each app.
• Privacy. The README warns that snapshots, clipboard reads, screenshots and logs can contain private data and should be kept out of public logs and uploads. An optional helper, separate from the native CLI, sends screen descriptions to a model provider you configure.
• Packaging. Homebrew and Scoop packages are still being submitted; npm, Bun, Yarn, pnpm and GitHub release archives work now.

Try it

npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide   # macOS only

Who should care: anyone wiring Claude Code, Codex or a custom agent into desktop workflows, and QA teams who want scriptable, structured UI automation across all three desktop operating systems. Start with a well-behaved native app, keep --headed off until you trust the loop, and don't pipe snapshots into anything you'd mind leaking.