# Vercel's agent-desktop lets AI agents drive Mac, Windows and Linux apps through the accessibility tree, not screenshots

> agent-desktop, an Apache-2.0 Rust CLI from Vercel Labs, gives coding agents structured, ref-based control of native desktop apps without a vision model. It added about 2,300 stars in a day and was GitHub's top trending repo on November 3.

Published Nov 3, 2026 by TechMeasured Staff · Repos · Canonical: https://techmeasured.com/repos/vercel-agent-desktop-accessibility-tree-computer-use-cli

**agent-desktop**, an open-source tool from Vercel Labs, gives AI agents a way to operate ordinary native desktop apps, plus system surfaces like the macOS Dock and menu bar, by reading the operating system's accessibility tree instead of guessing from screenshots. It topped a daily GitHub trending tracker on November 3 with 2,275 new stars in a day, and the repo page showed about 10.9K stars, 726 forks and an Apache-2.0 license when we checked.

Why it matters: most "computer use" agents today take a screenshot, ask a vision model where to click, and hope the pixels haven't moved. That is slow, expensive in tokens and brittle. agent-desktop takes the approach browser automation tools like Playwright take for the web: expose the app's real structure, let the agent act on named elements, then verify. If you're building an agent that has to touch native software, this is a cheaper, more deterministic starting point.

## How it works

Every call is a one-shot CLI command that returns JSON, according to the README:

1. **`snapshot`** reads the app's accessibility tree and returns each actionable element with a ref tied to that snapshot (for example `@s8f3k2p9:e8`). A bare `@e8` is rejected on purpose, so an agent can't act on an element from an old view.
2. **Act on a ref** with commands such as `click`, `type`, `set-value`, `select`, `toggle` or `scroll`.
3. **Snapshot again** to confirm the result. If the UI changed underneath, actions fail with `STALE_REF` or `AMBIGUOUS_TARGET` rather than clicking the wrong thing.

Errors carry a machine-readable `code`, a `disposition` and often a `suggestion`, and the README tells agents to retry only when the disposition says a retry is safe. The planning loop stays in your code: as a DEV Community write-up puts it, the tool doesn't plan, keep state between calls or embed a model.

The README lists 60 command names, 56 of them working, grouped into observe, act, keyboard and mouse, apps and windows, notifications, clipboard, waits and sessions. Each release also ships a C library (`libagent_desktop_ffi`) so programs can call it in-process; the README includes a Python `ctypes` example.

## Headless by default

Actions on refs go through accessibility APIs, so by default they don't move your mouse pointer or steal window focus, and headless calls refuse synthesized keyboard and mouse input. You opt into physical input with a global `--headed` flag. That matters if an agent is working on the same machine you're using.

## Platform support

| Platform | Backend | Caveats (per README) |
|---|---|---|
| macOS | Accessibility API | Needs Accessibility permission (`agent-desktop permissions --guide`) |
| Windows | UI Automation | No grant needed for apps at the terminal's integrity level |
| Linux | AT-SPI | Needs a graphical session and `at-spi2-core`; mouse input needs X11 or `--headed`; Wayland clipboard needs `wl-clipboard` |

Builds cover x64 and ARM64. Cross-platform support is the big change: when the project was posted to Hacker News about six months ago, its author said macOS was ready and Windows and Linux were on the roadmap, and commenters criticised the "cross-platform" label.

## Limits to know

- **It is only as good as the app's accessibility tree.** The DEV write-up notes that many Win32 apps, custom-drawn controls, games and some Electron apps expose thin or missing trees. Hacker News commenters raised the same problem with custom-rendered dialogs and lazy-loaded content. A `screenshot` command exists as a fallback.
- **Token savings are a claim.** The project says its "skeleton then drill down" pattern uses 78–96% fewer tokens on dense apps; the DEV author flags that this hasn't been independently verified and depends on each app.
- **Privacy.** The README warns that snapshots, clipboard reads, screenshots and logs can contain private data and should be kept out of public logs and uploads. An optional helper, separate from the native CLI, sends screen descriptions to a model provider you configure.
- **Packaging.** Homebrew and Scoop packages are still being submitted; npm, Bun, Yarn, pnpm and GitHub release archives work now.

## Try it

```bash
npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide   # macOS only
```

**Who should care:** anyone wiring Claude Code, Codex or a custom agent into desktop workflows, and QA teams who want scriptable, structured UI automation across all three desktop operating systems. Start with a well-behaved native app, keep `--headed` off until you trust the loop, and don't pipe snapshots into anything you'd mind leaking.


## Sources

1. [GitHub — vercel-labs/agent-desktop](https://github.com/vercel-labs/agent-desktop)
2. [GitHub — Trending 2026-11-03 (caige6666/github-trending-daily)](https://github.com/caige6666/github-trending-daily/issues/26)
3. [DEV Community — agent-desktop: observe-act-verify loops via Windows UI Automation instead of screenshot guessing](https://dev.to/madnessbrains_i/agent-desktop-10-observe-act-verify-loops-via-windows-ui-automation-instead-of-screenshot-guessing-52ei)
4. [Hacker News (mirror) — Show HN: Agent-desktop, native desktop automation CLI for AI agents](https://hn.jakechampion.name/item/47982708)