cua_agent (Computer Use Agent)

This repo is organized as a small multi-package workspace:

• cua_agent/: OS-agnostic core (planner, orchestrator loop, memory, policies, prompts/tool mapping).
• macos_cua_agent/: macOS adapter implementing the "computer" capabilities (screen capture, HID, Accessibility, AppleScript browser, permission checks).
• windows_cua_agent/: Windows adapter (screen capture, SendInput HID, PowerShell shell sandbox, CDP browser; UIA semantic grounding + Phantom Mode via comtypes, with OCR/blob fallback when UIA is unavailable).

Architecture boundary: core business logic lives in cua_agent/; adapter packages keep OS-specific drivers/integration code only.

What's inside (core + adapter)

• Orchestrator loop with planner + structured reflection, stagnation detection, and auto-replanning; episodes are summarized and logged.
• Grounding from an accessibility/UI tree with numbered Set-of-Mark overlays; visual fallback uses optional detector backend (ultralytics), OCR, and blob proposals when semantic trees are unavailable; pHash/SSIM change detection on logical-resolution captures.
• Desktop console UI (python -m cua_agent.ui): live session view (frame + crosshair + plan + cognitive trace + event feed), task launcher with skill fast-path match, HITL approval/question dialogs, memory curation (episodes + skills), turn-by-turn session replay, benchmark runner, health checks with fix hints, and a settings screen that writes .env. A compact always-on-top pill HUD is available via --hud.
• Optional live debug dashboard (FastAPI) with real-time screenshot preview, next-click crosshair, cognitive trace, and plan status stream.
• Action execution via an adapter-provided computer implementation; adapters may offer semantic (Accessibility/UIA) paths, HID paths, browser ops, and sandboxed shell ops.
• Memory layer storing episodes/logs/semantic notes and procedural skills with semantic hints; skills are retrieved via embeddings/keywords, with optional local vector indexing via Chroma.
• Safety rules from cua_agent/policies/safety_rules.yaml.
• HITL confirmations for policy-marked high-risk actions: terminal [y/N] when running in an interactive shell, or an approval card (approve / deny / deny-with-guidance, 120s auto-deny) when the console UI is running.
• Sensitive-screen redaction pass (OCR + blur) before screenshot frames are sent to model APIs.
• Execution profiles to constrain tooling by context: local_gui, remote_cli, or hybrid.

Requirements

• Python 3.11+; install base deps with pip install -r requirements.txt.
• Optional feature deps live in requirements-optional.txt (pip install -r requirements-optional.txt); each is gated by a feature flag and the core degrades gracefully without it:
  • chromadb — local skill vector indexing (ENABLE_CHROMA_SKILLS=true).
  • fastapi + uvicorn — desktop console UI (python -m cua_agent.ui) and live debug dashboard (ENABLE_DEBUG_DASHBOARD=true).
  • pywebview — hosts the compact HUD (python -m cua_agent.ui --hud) in a frameless always-on-top window; without it the HUD opens as a browser tab.
  • ultralytics — detector-assisted visual grounding (ENABLE_VISUAL_DETECTOR=true).
• Optional: brew install tesseract to improve OCR for the visual fallback path.
• OpenRouter account/key to drive the planner, cognitive core, and reflector models; without a key the agent runs in noop/stub mode.

Run

• Desktop console UI: python -m cua_agent.ui — opens the full console (session, launcher, memory, replay, bench, health, settings) in your browser. Flags: --hud (compact always-on-top pill HUD; auto-dodges the agent's next click and is excluded from screen capture on macOS), --adapter, --host/--port, --auth-token, --no-browser. The /api/* endpoints are always token-gated (every bind, including loopback, to resist other local processes and DNS-rebinding): a token is generated at startup — or set your own with --auth-token — and the console is opened via the tokenized URL printed at startup, which drops an auth cookie. Opening a plain http://127.0.0.1:8788 by hand still serves the page, but its API calls return 401 until you load the tokenized URL once. Session replay recording (screenshots to disk) is opt-in via ENABLE_TRAJECTORY_RECORDING. The console boots even with missing OS permissions — the Health screen doubles as first-run onboarding.
• Terminal (classic): Windows python -m windows_cua_agent.main (if ENABLE_HID=true and WINDOWS_AUTO_ELEVATE=true, the agent may request elevation via UAC; WINDOWS_CYBORG_MODE=true avoids CDP and drives Chrome via UIA/HID; for full browser tool support, launch Chrome/Edge with --remote-debugging-port=9222 on a non-default profile); macOS python -m macos_cua_agent.main.
• Core entrypoint (auto-selects adapter by OS, or override): python -m cua_agent (or python -m cua_agent --adapter windows_cua_agent)
• Benchmark suites for the console's Bench screen are JSON files in benchmark_suites/ (see custom-smoke.json; a task passes when the agent declares done).

Setup

• Create a .env with your keys and toggles (see .env.example).
• Install deps with pip install -r requirements.txt.
• Choose an execution profile:
  • EXECUTION_PROFILE=local_gui: computer/browser enabled, shell blocked.
  • EXECUTION_PROFILE=remote_cli: shell enabled, GUI/browser blocked.
  • EXECUTION_PROFILE=hybrid: both enabled.
• Optional debugging UX:
  • ENABLE_DEBUG_DASHBOARD=true
  • DEBUG_DASHBOARD_HOST=127.0.0.1
  • DEBUG_DASHBOARD_PORT=8765
• Strict post-action validation (recommended):
  • STRICT_POST_ACTION_STATE_CHANGE=true
• Procedural fast-path (the keyword path needs no extra dependency):
  • ENABLE_FAST_PATH_SKILLS=true
  • Optional vector retrieval on top (heavy dep, from requirements-optional.txt): ENABLE_EMBEDDINGS=true, ENABLE_CHROMA_SKILLS=true, CHROMA_PERSIST_DIR=.agent_memory/chroma

Testing

• pytest