TL;DR: Computer use lets an AI agent operate software the way a person does: it looks at a screenshot, decides where to click or what to type, issues the action, and looks again. No API required - if a human can drive the interface, the agent can attempt it. That universality is the appeal, and the fragility.
How it works
Ordinary tool use requires someone to have wrapped each capability in a function. Computer use replaces that with one universal tool: the screen. Each cycle, the harness captures a screenshot and sends it to a multimodal model, which reads the pixels - buttons, form fields, error dialogs - and responds with a concrete action: click at these coordinates, type this string, press this key, scroll. The harness executes the action against a real desktop or browser, takes a fresh screenshot, and the loop repeats. The model is doing genuine visual grounding: mapping "the Submit button" to a pixel location it can only infer from the image.
A concrete run: "download last month's invoice from the vendor portal." The agent opens the browser, screenshots the login page, types credentials from its configuration, finds Billing in the navigation, clicks through to invoice history, locates the row for last month, and clicks the download icon - eight or ten observe-act cycles for a task with no public API. Legacy line-of-business apps, admin consoles, and third-party portals are the natural habitat: software that will never expose a programmatic interface.
The trade-offs are structural. Every step costs a full image round trip through the model, so screen-driven work runs far slower and pricier than an equivalent API call. Interfaces shift - a moved button, an unexpected popup, a slow-loading page - and a misread screen sends a click into the wrong element. Accuracy on long interface tasks has improved steadily but still trails a careful human, so production deployments treat computer use as the fallback when no API exists, not the default path. A hybrid pattern dominates: API tools where they exist, screen control for the gaps, often mediated by browser automation rather than raw pixels.
Because the agent wields a real cursor with real credentials, containment does the heavy lifting. Sessions run in sandboxed virtual machines or dedicated browser profiles; sensitive actions - payments, deletions, sending mail - route through guardrails or a human approval gate. Screen content is also an injection surface: a malicious page can display instructions hoping the model treats them as commands, so harnesses distinguish what the user asked from what the screen says.
Where it sits in the AI stack
Computer use is a perception-action loop between the model and an ordinary screen:
Think of it as the last-mile layer of the agent stack: everything an agent cannot reach through a clean interface eventually funnels through the screen.
Key tools and implementations
-
Provider computer-use APIs
Model capabilities that accept screenshots and return click, type, and scroll actions.
-
Browser automation bridges
Playwright-style drivers that give agents structured page access instead of raw pixels.
-
Sandboxed desktops
Disposable VMs and containers that contain the blast radius of a misdirected click.
-
Browser-native agents
Products that embed the observe-act loop directly into a managed browser session.
Related entries
- AI agent A system where a language model plans, calls tools, and loops on results to finish a task with minimal supervision.
- Tool use (function calling) Letting a language model request actions by emitting structured calls that your code executes and whose results feed back in.
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.
- Human in the loop A design pattern where a person reviews, approves, or corrects an AI system's actions at defined checkpoints.