Term
Computer Use
Computer Use is a capability of modern AI models that lets them operate a computer like a human — see the screen, move the mouse, use the keyboard — to perform tasks across arbitrary applications without an API.
Computer Use — explained in more detail
Anthropic introduced Computer Use as a public beta in October 2024 with Claude 3.5 Sonnet; OpenAI followed with the “Computer-Using Agent” in its Operator product, and Google ships comparable mechanics in Gemini. The model receives screenshots of the screen as input and returns actions like “click at (412, 287)”, “type ‘hello’”, “press Cmd+T”. A surrounding layer (browser automation, container, desktop VM) executes those actions and feeds back the next screenshot. The crucial point: no special API endpoints or plugins are required — the model operates the UI that already exists.
Example / Practical context
Typical use cases: web research behind logins (classic scraper APIs fail there), data entry into legacy software without an API, end-to-end tests of web apps, extracting data from PDFs into local tools. In practice Computer Use is still noticeably slower and more error-prone than API-based tasks — every step costs a full image inference call, and one wrong click can derail the sequence. Pure API paths remain preferable wherever they are available.
How it differs from related terms
Classic tool use calls structured functions — precise, fast, but limited to existing APIs. Browser automation with Playwright or Selenium relies on deterministic selectors instead of visual understanding. Computer Use combines the best of both worlds: visual understanding like a human plus the ability to drive a full operating system.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarEnsemble / Multi-Model Orchestration
Ensemble means combining several deliberately varied LLM runs or models whose findings complement each other. Multi-model orchestration drives these runs via orchestrators with sub-agents, so the union of results is larger than any single run.
LexikonFunction Calling / Tool Use
How an LLM uses tools: define a tool as a schema, the model picks the function and arguments, the result returns to the chat — the basis of every agent.