Computer-Use Agents Compared
Every major lab now ships a model that can look at a screen and click on it. Anthropic's computer use tool, OpenAI's computer tool, and Google's computer_use tool all solve the same shape of problem — take a screenshot, decide an action, execute it, screenshot again — and all three are wire-incompatible with each other in ways that are not obvious until your clicks start landing 40 pixels off.
This page is the harness-level guide, not the leaderboard. The leaderboard changes monthly; the failure modes below have been the same since the first preview shipped.
:::caution Related security reading When the screen an agent drives is a browser, the agent also lands above a boundary the web has relied on for 30 years — see Agentic Browsers Break the Same-Origin Policy for the UW findings on Atlas, Comet, Gemini in Chrome and Claude for Chrome. :::
:::tip Related pattern For the shared-login variant — an agent that drives a real Chromium session with your cookies inherited, via ego lite and its ego-browser skill — the loop above collapses into a JavaScript-per-turn model and the security posture inverts (the agent has your privileges by design). :::
- Understand the agent loop all three providers share — and the three places they diverge
- Fix the coordinate bug that silently breaks most first attempts (Retina, downscaling, and image size limits)
- Know each provider's safety gate, because it is part of the protocol, not an optional add-on
- Read OSWorld honestly: what the human baseline actually means
- Pick reasoning effort correctly — the cheapest setting is not the one you would guess
The loop everyone shares
Strip away the branding and all three are the same state machine:
- The model gets your instruction and an image of the current screen. Some providers also want the current URL or a short history of recent actions.
- Click at (x, y). Type this. Scroll here. It is a request for your code to do something — the model cannot touch the machine itself.
- You own this. Anthropic states it plainly: your application must run the tool; Claude cannot run it directly. You implement the screenshot capture, the mouse, the keyboard.
- The new screen goes back as a tool result. The model sees the consequence of its own action — this is the entire feedback signal.
- Termination is the absence of a further action request, not a 'done' flag. Your loop needs its own step cap and stall detection.
The consequence people miss: the model's only perception of the world is the image you send it. Every failure below is really a failure of that image, or of the coordinate space it implies.
Where the three diverge
The divergence is not "which one is smarter." It is the contract.
| Anthropic (Claude) | OpenAI (GPT) | Google (Gemini) | |
|---|---|---|---|
| Action space | Raw desktop primitives: screenshot, left_click, type, key, scroll, drag, hold_key, wait, plus zoom on the newest tool version | Structured UI actions: click, double_click, scroll, type, keypress, drag, move, wait, screenshot | Semantic actions: not just click/type but navigate, go_back, go_forward, open_app, long_press — the browser and mobile concepts are first-class |
| Coordinates | You send pixel dimensions; the model returns raw pixel coordinates in that space | Same pixel contract, with recommended desktop sizes | Same, with an intent field explaining the reasoning behind each action |
| Loop plumbing | Tool result blocks carrying an image | computer_call → computer_call_output keyed by call_id; previous_response_id carries history so you don't resend it | Function call → function_result with a fresh screenshot |
| Safety gate | Injection classifiers run automatically on screenshots | pending_safety_checks you must explicitly acknowledge | safety_decision with policy categories |
| Sweet spot | General desktop / OS-level control | Desktop and browser, plus a code-execution path | Browser-first; strong on mobile; explicitly not optimized for desktop OS control |
That last Anthropic/Google row is the one that decides architecture. Google's action space is semantic — go_back is a concept the model can name. Anthropic's is mechanical — going back means the model must find and click the back button, or press the right key combination. Semantic actions are more reliable inside a browser and useless outside one. Mechanical primitives work anywhere and fail more often.
So: porting a computer-use agent between providers is not a model swap. It is rewriting the harness's action executor. Budget accordingly. (Same lesson as Coding Agent CLIs Compared: the harness, not the model, is the thing you're actually married to.)
The coordinate bug that eats your first week
This is the single highest-value thing on this page, because almost everyone hits it and the symptom looks like "the model is bad at clicking."
It is not bad at clicking. Your image and your coordinate space disagree.
Three independent causes, which stack:
1. Retina / HiDPI doubles your image
macOS Retina displays capture screenshots at a device pixel ratio of 2 — the image is twice the resolution of the logical screen coordinates. Send that raw and the model reasons about a 2880-wide image while your click executor thinks in 1440-wide logical points. Every click lands at roughly half the intended position, consistently, in one direction.
Fix: downscale by 2 before sending, or halve the coordinates the model returns. Not both.
2. The API silently downscales oversized images
Models have image size limits — and they differ between models from the same lab. Claude Sonnet 5, Opus 4.8 and Opus 4.7 accept up to 2576 pixels on the long edge; earlier Claude models accept 1568 pixels and roughly 1.15 megapixels total.
Here is the trap: if you send something bigger, the API downscales it for you rather than erroring. The model then returns coordinates in the space of the image it saw — and you never learned the scale factor, because the resize happened server-side. Only genuinely enormous images (over ~8,000 px on a side) get rejected outright with a validation error.
So the "helpful" behavior is the bug. Always resize client-side, set display_width_px/display_height_px to the dimensions you actually sent, and scale returned coordinates back up yourself.
3. Detail settings and aspect ratio
On OpenAI's tool, screenshots should use detail: "original" — both "high" and "low" degrade click accuracy for this task specifically. And if you resize without preserving aspect ratio, clicks land in the right region and miss the target.
Coordinate scaling — the shape of the fix
# Before sending: shrink to fit the model's image limit, remember the factor. LONG_EDGE_LIMIT = 2576 # check YOUR model's limit; older Claude models: 1568 scale = min(1.0, LONG_EDGE_LIMIT / max(width, height)) sent_w, sent_h = round(width * scale), round(height * scale) # -> send the resized image, and declare display_width_px=sent_w, display_height_px=sent_h # After the model replies: map its coordinates back to the real screen. real_x, real_y = model_x / scale, model_y / scale # On a Retina capture you did NOT pre-downscale, divide by the device pixel ratio too.
Symptom → cause cheat sheet
| Symptom | Almost certainly |
|---|---|
| Clicks consistently offset in one direction | Your declared display_width_px/display_height_px don't match the image you actually sent |
| Clicks land in the right area but miss small targets | Detail lost to downscaling, or aspect ratio distorted on resize |
| Accuracy poor everywhere | Resolution too low — try 1280x720 as a floor |
| Model misreads tiny text (tab titles, filenames, line numbers) | It needs to zoom, and you didn't enable it |
Resolution guidance, from the vendors themselves: Anthropic suggests 1024x768 or 1280x720 for general desktop, 1280x800 or 1366x768 for web apps, and explicitly says to avoid going above 1920x1080. OpenAI recommends 1440x900 or 1600x900. Bigger is not better — you pay tokens for pixels and then throw the detail away in the downscale.
The zoom escape hatch
Claude's newest computer tool version (computer_20251124) adds a zoom action, off by default — you must set enable_zoom: true. With it on, Claude zooms into a region when it needs to read small text that isn't legible at the screenshot's base resolution: sidebar filenames, tab titles, status-bar text, line numbers, button labels.
Non-obvious operational note: if Claude isn't zooming when you expect it to, the fix is usually to ask about a specific region or element rather than about the screen as a whole.
Safety gates are part of the protocol
Do not treat this as a bolt-on. In all three APIs, the safety mechanism changes the shape of your loop.
Anthropic trained the model to resist prompt injection and runs classifiers over your prompts automatically when the computer-use tools are in play. When a classifier spots a likely injection in a screenshot, it steers the model to ask the user for confirmation before the next action. That is great with a human present and actively wrong for an unattended pipeline — which is why there is an opt-out, gated behind contacting support.
OpenAI surfaces pending_safety_checks that your code must explicitly acknowledge (acknowledged_safety_checks) before the loop proceeds. The gate is in your hands, and skipping it is a decision you make in code.
Google returns a safety_decision — allowed, require_confirmation, or blocked — driven by policy categories including FINANCIAL_TRANSACTIONS, COMMUNICATION_TOOL, ACCOUNT_CREATION, SENSITIVE_DATA_MODIFICATION and LEGAL_TERMS_AND_AGREEMENTS. Prompt-injection screening of screenshots is available as an opt-in.
The threat is real and specific: a screenshot is untrusted input. Text rendered on a webpage, in an image, in a PDF the agent opened — all of it reaches the model through the same channel as your instructions. Anthropic's own docs note that Claude will, in some circumstances, follow instructions found in content even when they conflict with yours. Every provider's mitigation converges on the same three rules: isolate the environment, keep a human on high-impact actions, and treat everything on screen as hostile.
If the agent must log in, that risk goes up sharply — credentials plus injectable content is the worst combination in this space. See Securing Local and Hybrid Agents and Prompt Injection for the defensive playbook.
Reading OSWorld honestly
OSWorld is the shared yardstick, and it's a good one because it's hard to fake: it drops an agent into a real OS with real applications, and grades with execution-based verification — a script checks whether the file actually got saved, not whether the agent claimed it did. The benchmark spans 369 tasks (361 in the standard evaluation set, since a handful of Google Drive tasks need manual setup) and ships 134 execution-based evaluation functions. OSWorld-Verified is the cleaned-up revision, with community-reported broken examples fixed and evaluation time cut to about an hour on AWS.
Two numbers worth holding together:
- The human baseline is ~72.4%. Not 100%. These tasks are genuinely fiddly, and humans fumble them too.
- When OSWorld launched, the best model scored 12.24%.
Frontier agents now report scores in the 70s and up — which means the headline "agents have reached human level on computer use" is arithmetically defensible and practically misleading. A benchmark score is a model + harness + scaffold result, exactly like a coding benchmark (see The Capability–Reliability Gap). Your harness is not their harness. And a 75% success rate on single tasks compounds brutally: an eight-step workflow where each step is 95% reliable succeeds about 66% of the time.
Treat OSWorld as evidence the capability exists, and your own eval set as the only evidence that your product works.
The reasoning-effort result nobody expects
For Claude computer use specifically, Anthropic's internal benchmarking gives guidance that inverts the usual intuition:
- Opus 4.7:
higheffort as the default; drop tolowfor high-throughput or cost-sensitive workloads. - Sonnet 4.6 and Opus 4.6:
mediumis the best accuracy-to-cost ratio. Avoidmax— on UI tasks it adds token cost without improving accuracy. - The counterintuitive one: on those models,
loweffort uses fewer output tokens than disabling thinking entirely. A little thinking prevents mistakes, and mistakes cost you retries — and retries cost far more tokens than the thinking did.
So "turn thinking off to save money" is, for computer use, often wrong. Cheapest is a little thinking.
One more model-selection wrinkle: mechanical click precision is not the same axis as intelligence. Sonnet 4.6 is more mechanically precise at clicking than Opus 4.6, and more robust when screenshots have been heavily downscaled. Opus 4.7 narrows that gap and raises the pixel limit, so it needs less downscaling in the first place.
A harness that survives contact
- A container or VM with nothing valuable in it. Every provider says this and they all mean it: the agent will eventually click something you did not intend.
- Capture, resize client-side to your model's limit, declare the exact dimensions you sent, and keep the scale factor. Write this once and never think about it again.
- Termination is 'no further action returned' — which never fires if the agent is stuck in a modal it can't dismiss. Add a max-step budget and a stall detector (identical screenshots N times running).
- Human-in-the-loop, or unattended with an opt-out and a narrower blast radius. Do not discover this by accident when a confirmation prompt deadlocks your cron job.
- Execution-verified, like OSWorld does it: check the world changed, not that the agent said it did. This is the only number about your system that means anything.
- The most reliable computer-use agent is the one that doesn't have to use the computer. If the target has an API, an MCP server, or a CLI, that path is faster, cheaper and orders of magnitude more reliable. Screens are the fallback, not the goal.
That last step is the strategic one. Computer use is the universal adapter for software with no API — legacy desktop apps, vendor portals, anything behind a login with no integration story. It is a magnificent hack, and it is still a hack. Reach for MCP and real tools first.
Check yourself
0/5Sources & further reading
- Anthropic — Computer use tool — beta headers, action set, image limits, coordinate scaling, effort guidance, injection classifiers.
- OpenAI — Computer use guide — the
computer_call/computer_call_outputloop, safety checks, recommended resolutions. - Google — Gemini computer use — the semantic action space across browser, mobile and desktop;
safety_decisionpolicy categories. - OSWorld and xlang-ai/OSWorld — the benchmark, its execution-based verification, and the human baseline.
- anthropics/anthropic-quickstarts — the
computer-use-demoreference implementation (container, virtual display, agent loop). - openai/openai-cua-sample-app — OpenAI's reference computer-use harness.
- browser-use/browser-use — a widely used model-agnostic browser-agent harness, if you want the browser case without building the loop yourself.