मुख्य कंटेंट तक स्किप करें

Desktop Agents Without Pixels

उन्नत
What you'll learn
  • Understand what an accessibility-tree agent reads, and how it differs from a screenshot agent at the protocol level
  • Learn the retry rule that most computer-use harnesses get wrong — and why 'did it fail?' is the wrong question
  • See why refs are scoped to a snapshot, and what a harness should do instead of guessing when one goes stale
  • Know the three-state property model that stops an agent acting on a failed read
  • Spot the two failure modes this approach does NOT fix: truncated trees and apps with no accessibility surface
  • Understand the security surface: everything on screen becomes agent input

On 28 October 2026 Vercel Labs tagged v1.0.0 of agent-desktop, and by 31 October it was at the top of GitHub's trending list. It is a Rust CLI and C library, Apache-2.0, that gives an agent computer use on macOS, Windows and Linux by reading the operating system's accessibility tree — the same structure a screen reader consumes: AX API on macOS, UI Automation on Windows, AT-SPI on Linux. Vercel's own framing is "agent-browser, for your whole computer."

Every write-up so far stops at the headline: trees instead of pixels, so clicks land where they should. That is true and it is the least interesting half.

The interesting half is that the repository ships a CONCEPTS.md which is, in effect, a specification for how a computer-use harness should handle uncertainty — and it answers questions that the mainstream computer-use APIs leave entirely to you. If you build agent loops, read this page for the model even if you never install the tool. Our computer-use agents compared page covers the screenshot-based APIs from Anthropic, OpenAI and Google; this is the other architecture.

What changes when you read the tree instead of the screen​

A screenshot agent gets an image and returns coordinates. The contract is: here are some pixels, tell me where to click. Everything the agent knows about the UI it has to infer visually, and every action is a bet on a coordinate.

A tree agent gets structure. A snapshot returns JSON with a ref for every element the agent can act on, and actions name a ref rather than a point.

Guided walkthrough1 of 4
  1. Snapshot the accessibility tree of a window, app or surface. You get back JSON — roles, names, states — with a ref per actionable node, and a snapshot ID that the refs belong to.

The practical differences that follow from this are larger than they look:

  • Refs survive layout changes that break coordinates. A button that moves 200 px down the window is the same node in the tree. For a screenshot agent it is a new bet.
  • It is headless by default. Input goes through the accessibility API without moving the system pointer or stealing focus, so the agent can drive apps while you keep using the machine. A --headed mode exists for physical input synthesis when an app ignores programmatic input. This is a genuinely different operating model from screenshot agents, which fight you for the cursor.
  • Snapshots are text, not images. That matters for your bill and your context window — see why agents burn tokens. But it is not a free win, because dense trees get large; the repo's own answer is to map coarsely first (--skeleton --compact) and then drill into a region with --root.
  • You still need screenshots sometimes. The tool ships screenshot alongside snapshot. Canvas-rendered apps, custom-drawn UIs and games have little or no accessibility surface, and no amount of tree-reading invents one.

The minimal observe / act / verify loop

npm install -g agent-desktop
agent-desktop version

# macOS only: accessibility permission must be granted first
agent-desktop permissions --guide

# 1. observe — -i limits output to interactive elements
agent-desktop snapshot --app Finder -i

# 2. act — the ref MUST carry its snapshot ID; a bare @e8 is rejected
agent-desktop click @s8f3k2p9:e8

# 3. verify — re-snapshot and confirm the change landed
agent-desktop snapshot --app Finder -i

# dense UI? map coarsely, then drill into one region
agent-desktop snapshot --app Slack --skeleton --compact
agent-desktop snapshot --root @s8f3k2p9:e42

Output is JSON with an ok boolean and either data or an error carrying a code, a disposition and a suggestion. The commands group into observe (snapshot, find, get, screenshot, list-surfaces), act-on-ref (click, type, set-value, scroll, toggle, expand), raw keyboard and mouse (press, hover, drag, key-down), window and app management (launch, focus-window, resize-window, list-apps), and system (permissions, status, batch, skills).

The idea worth stealing: delivered, not successful​

Here is the rule, and it is the single most useful thing in the repository:

The distinction that matters is not success versus failure. It is delivered versus not delivered.

Think about what an agent does when a type command returns an error. The instinctive harness behaviour — retry on error — is actively dangerous, because "error" conflates two completely different situations:

  • The keystrokes never reached the field. Retrying is free.
  • The keystrokes may have landed and the confirmation read failed. Retrying types everything twice.

A harness that cannot tell these apart will silently double-submit forms, double-click Send, and duplicate rows. So agent-desktop classifies every mutation into three delivery states instead of a boolean:

Delivery stateMeaningRetry?
Not deliveredArgument error, access denial, or a read-only command — the action demonstrably never reached the targetSafe. Retry freely
Delivered unverifiedInput was posted but the effect is unconfirmed — the typical outcome of physical input synthesisUnsafe. Re-reading state is the only correct move
DeliveredEffect confirmed by re-reading the elementDone; nothing to retry

Reads get a parallel four-way classification, and the first one is the subtle one:

  • Settled absence — the provider was asked and the target genuinely lacks that property or pattern. A final answer, never retried.
  • Retryable failure — a transport fault such as a timeout or dropped connection. Worth repeating within a budget.
  • Unavailable element — it disappeared. Settles as a stale ref if the element had been resolved; retryable if hit mid-walk.
  • Terminal failure — a denial or an unclassified code. Final, surfaced as-is.

The tool exposes this directly: you retry only when error.disposition.retry says safe.

Key takeaways
  • If you are writing your own computer-use loop today, port this one idea even if you port nothing else. Classify every action by whether it could have landed, and make 'may have landed' a distinct branch that re-reads state instead of retrying. Idempotency is not a property of your retry code — it is a property of each individual action, and only the tool layer knows which is which.

Why refs are scoped to a snapshot​

In agent-desktop a ref looks like @e8, but you may never use it bare — a bare ref fails with INVALID_ARGS. It must be qualified by its snapshot ID: @s8f3k2p9:e8.

That looks like friction. It is a deliberate type system. The stated reason: refs are deterministic inside one snapshot but are not stable across UI changes. The snapshot ID is a namespace, and every lookup is confined to the namespace it was created in — a snapshot taken under a session needs that same session scope later.

So the awkward syntax makes a real category error unrepresentable. You cannot accidentally carry an identifier from an observation three actions ago and have it silently resolve to whatever now occupies that slot. Compare the screenshot world, where a coordinate from a stale screenshot is syntactically indistinguishable from a fresh one and resolves to something every time.

Fail-closed resolution: refusing to guess​

When the action fires, the ref is resolved against the live tree through a graded sequence, and the ordering encodes a judgement about which evidence is trustworthy:

Guided walkthrough1 of 4
  1. Follow the path recorded at snapshot time down the current tree.

The design commitment is in what happens when none of that produces a confident match: the resolution fails closed. Missing, stale and ambiguous matches are rejected — STALE_REF, AMBIGUOUS_TARGET — rather than resolved to a best guess. On macOS the implementation retains the native accessibility objects specifically so an element cannot be silently misidentified when tree ordering shifts underneath it.

This is the opposite of how a pixel agent degrades. A stale coordinate always hits something; a stale ref returns an error. Trading a silent wrong action for a loud failure is the entire reliability claim, and it is the right trade for anything an agent does unsupervised.

What you'll learn
  • The same reasoning shows up in the repo's two-sided treatment of partial results: if a snapshot runs out of its time budget mid-walk, it returns what it observed rather than discarding the walk — but it reports completeness per node. A harness that ignores that field will confidently reason over a truncated tree and conclude a button does not exist. Check completeness before you conclude absence.

Tri-state properties: Absent, Unknown, Known​

A smaller mechanism with outsized consequences. Properties are never collapsed to a boolean. Three states stay distinct:

  • Known — a value was read.
  • Absent — the provider was asked and genuinely has no value.
  • Unknown — the read itself failed.

Most harnesses flatten Unknown into false or null, and an agent then acts on a failed read as if it were a negative result: the checkbox "isn't checked" (we could not read it), the field "is empty" (the query errored). The repo's phrasing is that uncertainty must travel — when a higher-priority evidence source fails, that failure is carried forward rather than smoothed away. Accessible names are computed in the core from ranked slots rather than in each platform adapter, so shared logic never sees a platform-specific token and the same uncertainty rules apply everywhere.

If you take two things from this page, take delivery semantics and this. They are both cases of the same discipline: do not let a failure to know masquerade as knowledge.

Two limits worth naming​

The interaction lease covers dispatch only. Input exclusivity is machine-wide and held by one process at a time, but the lease deliberately does not cover the waiting that precedes an action, because waiting for an element to become actionable can run long. The practical consequence: an element can become actionable and then change again between your wait and your dispatch. That is exactly why the verify step is not optional.

Platform support is not uniform, and Linux is the ragged edge. Accessibility tree, clicking and typing work across all three OSes, but on Linux clicking and typing run through AT-SPI with X11/GNOME caveats, screenshots need X11 or a Wayland portal, the clipboard needs wl-clipboard under Wayland, app management is launch-and-list only, and notifications are GNOME Shell. macOS needs accessibility permission granted explicitly, plus a TMPDIR that is user-owned with mode 0700. Windows needs no special permission at matching terminal integrity level. Of roughly 60 commands, not all are operational on every platform — check agent-desktop status on the machine you actually plan to run on.

The security surface is the whole screen​

This needs stating plainly, because the convenience is seductive.

An accessibility-tree agent reads the text content of applications, which is a strictly richer capture than a screenshot of the focused window — including, potentially, content that is on screen but not what you were looking at. The repo is explicit that snapshots, screenshots and clipboard contents can contain sensitive data and should be kept out of public logs. It does build in one real mitigation: secure fields. Once a platform marks an element secret, the read path withholds both its value and related text evidence, and the action path may write into it but never echoes the attempted or observed value.

That protects passwords. It does not protect the open email, the Slack DM, or the customer record in the next window. Two consequences:

  1. Transcripts become sensitive artifacts. Agent logs from a desktop session are closer to a screen recording than to a terminal history. Treat them accordingly — see what your agent uploads.
  2. Every readable surface is an injection surface. Any text an app displays becomes agent input, so a crafted calendar invite, filename or chat message is a prompt-injection vector with no network request involved. The same reasoning as the web your agent reads applies to the desktop, and prompt injection is the primer. An agent with machine-wide input exclusivity and no pointer movement is also, by construction, one whose actions you will not see happening.
Watch out
  • Headless-by-default is a usability feature and a supervision hazard at the same time. Because the agent does not move your pointer or steal focus, there is no ambient signal that it is acting. If you run this unattended, the compensating control has to be the log and a scoped-down target app — not your attention. Our hardening-autonomous-runs page covers the general pattern.

When this architecture is the right choice​

SituationReach for
Native desktop app with real accessibility support (Finder, Office, most Cocoa/WinUI apps)Tree agent. Stable refs, cheap text snapshots, no coordinate math
Web app in a browserA browser-specific harness first — the DOM is a better tree than the browser's AX projection. See computer-use agents compared and shared-login browsers
Canvas/WebGL app, game, custom-drawn UI, remote desktopScreenshot agent. There is no accessibility surface to read
Agent must run while you keep workingTree agent, headless mode — no focus stealing, no pointer fight
Unsupervised, repeated, mutating actionsTree agent, and honour the delivery semantics. This is the case the whole design exists for
Visual verification ("does this look right?")Screenshot, regardless of harness. A tree cannot tell you the layout is broken
कार्ड पलटने के लिए Enter या Space दबाएँ। कार्ड बदलने के लिए बाएँ और दाएँ तीर कुंजियों का उपयोग करें।शब्द दिखाया गया।
1 / 10

Check yourself

0/5
  1. Your harness sends a `type` command into a form field and gets back an error with delivery state 'delivered unverified'. What is the correct next move?
  2. Why does agent-desktop reject a bare ref like `@e8` and require `@s8f3k2p9:e8`?
  3. A snapshot returns without the button your agent expected. What must you check before concluding the button does not exist?
  4. Which task is the WORST fit for an accessibility-tree agent?
  5. What is the main NEW security exposure of a desktop tree agent compared with a screenshot agent confined to one browser tab?
Key takeaways
  • Accessibility trees replace a coordinate bet with a named reference, so layout changes that break a screenshot agent are a non-event — but canvas, games and custom-drawn UIs have no tree to read.
  • Steal the delivery model: classify actions by whether they COULD have landed, not by whether they returned an error. 'May have landed' deserves a re-read, never a retry.
  • Scoping refs to a snapshot is a type system, not friction — it makes stale-identifier bugs impossible to express rather than silently wrong.
  • Fail-closed resolution is the reliability claim: a stale coordinate always hits something, a stale ref returns STALE_REF. Prefer the loud failure for anything unsupervised.
  • Keep Unknown distinct from Absent. An agent that reads a failed query as a negative result will act confidently on nothing.
  • Headless operation removes the ambient signal that an agent is working. If you run it unattended, logs and a scoped target app are the control — not your attention.

Sources & further reading​

Next​