The computer-use layer for AI agents, on Windows and Linux. Ghost lets an agent operate any desktop app — including the ones with no API — in the background without taking your screen or cursor, and it proves every action actually happened.
Like Playwright, but for native desktop apps, and built for agents: an MCP server any model can mount to see and drive the desktop.
One MCP surface, two engines: Win32 UI Automation on Windows, AT-SPI2 over D-Bus on Linux. The verbs, the locator tiers and the act-then-verify loop are written once and behave the same on both. Platform support · Linux setup
- Runs in the background. An agent can click, type, and use shortcuts inside an app while you keep working in another window — no focus steal, no cursor jump. It posts window messages to real controls; most tools can only drive whatever is in the foreground. (how)
- Every action is verified. Ghost re-checks the screen (or reads the control's
value back) after acting and returns
verified/focus_confirmed— never a blindok:true. Agents fail by acting and not knowing if it worked; Ghost closes that loop. - Drives apps with no API. Legacy Win32, WPF, Electron, UWP, vendor portals — the software that has no integration and most needs automating. No CDP, no browser, no app cooperation required.
- Model-agnostic. Vision grounding works with any OpenAI-compatible model (NVIDIA, OpenAI, Gemini, Groq, local vLLM/Ollama) or Anthropic. No vendor lock-in.
- Accessibility-native and deep. Real element discovery through the OS's own accessibility API — UI Automation on Windows, AT-SPI2 on Linux — not pixel-guessing. Elements come back with real names, roles and bounds.
See it in one script: examples/background_agent_demo.py
drives an app in the background while the foreground stays yours.
Honest comparison vs Playwright-MCP / cua-driver / Computer Use:
docs/comparison.md.
Ghost gives you programmatic control over any desktop application — native Win32, Electron, WPF, UWP, GTK, Qt, or otherwise.
On Windows it uses UI Automation for element discovery, SendInput for keyboard/mouse injection, and DXGI/GDI for screen capture. On Linux it uses AT-SPI2 over D-Bus for discovery and actions, XTEST (X11) or the RemoteDesktop portal / uinput (Wayland) for input, and X11 GetImage or the Screenshot portal for capture. The Linux engine is pure Rust — no -devel packages to install.
Ship it three ways:
ghostCLI — one-shot commands, great for scripts and CI (ghost click --name "Submit")ghost-httpserver — local REST API, call it from Python, Node, curl, anything (curl http://127.0.0.1:7878/list-windows)ghost-mcpserver — Model Context Protocol server for Claude, Cursor, and any MCP client (20 verbs)
No Claude required. No browser required. No CDP. It drives apps through the OS's own automation and input APIs, so it works with native apps that have no API and no automation hooks of their own — the same reliability whether or not an app was built to be automated.
| Platform | Status | Engine |
|---|---|---|
| Windows | ✅ full and verified | ghost-core — Win32 UI Automation, SendInput, posted window messages, DXGI/GDI capture |
| Linux | ✅ functional — X11 + AT-SPI2 verified by live CI tests | ghost-linux — AT-SPI2 over D-Bus, XTEST / RemoteDesktop portal / uinput, X11 GetImage / Screenshot portal |
| macOS | 🚧 scaffold | Accessibility + CGEvent + ScreenCaptureKit — to be built on a Mac |
ghost-session and ghost-mcp are shared: the locator tiers, grounding cascade,
act-then-verify loop and all 20 MCP verbs are written once and run on both
platforms. Only the engine underneath changes, behind a one-line cfg alias.
The wedge survives the port. On Windows, driving an app without stealing focus is built on posted window messages. Linux has a cleaner analogue in AT-SPI2 actions: the application performs the operation through its own toolkit, so there is no pointer to move and no window to raise — and it behaves the same under X11 and Wayland. Synthetic input is only the fallback there.
This is tested, not asserted: CI stands up a real desktop (Xvfb + D-Bus + at-spi-bus-launcher), drives a real GTK application, and requires that text written through AT-SPI reads back from the app and that invoking a button actually dismisses the dialog. Wayland portal input and capture are implemented but not yet verified on hardware.
Linux setup, verification checklist and honest limitations:
docs/linux-fedora.md. Capability matrix across all
three: docs/cross-platform.md.
Ghost is a general-purpose automation tool. Use it on systems you own or are authorized to automate, and in line with the terms of the software you drive.
Option A — Prebuilt binaries (free). Every release ships signed-by-checksum archives for both platforms on the Releases page:
# Linux x86_64
curl -LO https://github.com/NORTHTEKDevs/ghost/releases/latest/download/ghost-linux-x86_64.tar.gz
curl -LO https://github.com/NORTHTEKDevs/ghost/releases/latest/download/ghost-linux-x86_64.tar.gz.sha256
sha256sum -c ghost-linux-x86_64.tar.gz.sha256
tar -xzf ghost-linux-x86_64.tar.gz && ./install.shWindows: download ghost-windows-x64.zip from the same page. Verify the
checksum, unzip, and add the folder to your PATH. Then run ghost doctor.
Option B — Ready-to-run Windows kit ($20, one-time). Prebuilt Windows binaries (ghost.exe,
ghost-http.exe, ghost-mcp.exe) plus a quick-start, MCP config, and examples — no Rust toolchain, runs in
two minutes. Every kit is built by scripts/package-kit.ps1, which refuses to package unless the full live
desktop suite passes. Get it at northtek.io/ghost.
The binaries are not code-signed, so Windows SmartScreen will warn you on first run (click More info → Run anyway). The kit buys convenience, not capability — everything Ghost can do is in the free source below, and building it yourself takes one command.
Option C — Build from source (free, MIT). Ghost is open source. Compile it yourself:
git clone https://github.com/NORTHTEKDevs/ghost
cd ghost
cargo build --release --bin ghost --bin ghost-http --bin ghost-mcp
# binaries in target/release/Requirements: Windows 10 build 19041+, or Linux with at-spi2-core (and Rust
stable only if building from source).
On Linux:
sudo dnf install at-spi2-core xdg-desktop-portal xdg-desktop-portal-gnome
gsettings set org.gnome.desktop.interface toolkit-accessibility true
./scripts/install.sh # build, install, register the MCP server, run doctorNo -devel packages are needed — the Linux engine is pure Rust. Full setup and
troubleshooting: docs/linux-fedora.md.
Check your machine first:
ghost doctorReports PASS/WARN/FAIL and exits 1 if anything is FAIL. Run it before opening an issue — it usually names the problem outright.
- Windows: build version, interactive desktop, UI Automation, DPI awareness, monitor layout, screen capture, optional vision credentials.
- Linux: session type (X11/Wayland), AT-SPI bus reachability, whether applications are actually exposing accessible trees, the selected input backend, and screen capture.
# Launch Notepad and type into it
ghost launch notepad.exe
ghost focus-window "Notepad"
ghost type --role edit --text "hello from ghost"
# Keys and hotkeys
ghost press Enter
ghost hotkey --mods Ctrl --key s
# Screenshot
ghost screenshot --out shot.png
# Enumerate windows or UI
ghost list-windows
ghost describe --window "Notepad"
# Click at coords or by name
ghost click-at 500 300
ghost click --name "Save"
# Run a JSON intent (finite-state machine with retries, timeouts, conditions)
ghost run my-flow.json
echo '{"ops":[{"op":"launch","exe":"notepad.exe"}]}' | ghost run -Everything outputs JSON for easy piping into jq or scripts.
Start the server:
ghost-http --addr 127.0.0.1:7878Then from any language:
# Bash / curl
curl http://127.0.0.1:7878/list-windows
curl -X POST http://127.0.0.1:7878/click \
-H 'content-type: application/json' \
-d '{"name":"Submit"}'
curl http://127.0.0.1:7878/screenshot -o shot.png# Python
import requests
requests.post("http://127.0.0.1:7878/launch", json={"exe": "notepad.exe"})
requests.post("http://127.0.0.1:7878/type",
json={"role": "edit", "text": "hello from python"})// Node
await fetch("http://127.0.0.1:7878/hotkey", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ mods: ["Ctrl"], key: "s" }),
});Endpoints: /health, /tools, /click, /click-at, /type, /press, /hotkey, /screenshot, /launch, /list-windows, /focus-window, /window-state, /describe, /clipboard (GET/POST), /run.
[dependencies]
ghost-session = { git = "https://github.com/NORTHTEKDevs/ghost" }use ghost_session::{GhostSession, By, session::Region};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let session = GhostSession::new()?;
session.launch("notepad.exe").await?;
let edit = session.find(By::role("edit")).await?;
edit.type_text("hello world")?;
let png = session.screenshot(Region::full()).await?;
std::fs::write("screen.png", png)?;
Ok(())
}cargo build -p ghost-mcp --releaseAdd to Claude Desktop config (%APPDATA%\Claude\claude_desktop_config.json):
{
"mcpServers": {
"ghost": { "command": "C:/path/to/ghost-mcp.exe" }
}
}Works with any MCP client (Claude, Cursor, etc.) — 20 lean verbs advertised (legacy names stay dispatchable) covering see/snapshot/find/act/keys/scroll/drag/clipboard/screenshot/windows/shell/waits/query/run.
Ghost drives GUIs and the command line. ghost_shell runs terminal commands and
persistent PowerShell sessions — builds, git, CLIs, file edits on hosts without file
tools, or launching apps. op=run is a one-shot (powershell/pwsh/cmd); op=open
starts a persistent PowerShell whose variables and cwd survive across op=send calls.
Output is merged stdout+stderr, tail-capped for the agent's context window; a timed-out
command keeps running and is drained with op=read; ghost_stop kills a runaway.
Spawn a fresh Claude Code session from the agent:
ghost_shell op=run cmd='Start-Process wt -ArgumentList "pwsh","-NoExit","-Command","claude"',
then drive the new terminal window with ghost_see / ghost_act / ghost_key.
Security: shell access is powerful. Set GHOST_SHELL=off in the server's env to
disable the verb entirely — every op then returns a clear refusal, leaving the GUI
automation verbs fully usable.
Desktop automation driven from an MCP client has a hostile focus environment: between tool calls, the client's own terminal usually retakes OS focus. Ghost is built for that:
ghost_actis atomic — find → bring the target's window to the foreground (AttachThreadInput, confirmed) → act → verify via screen delta. One call, no cross-call race.- Every action response is honest:
verified(did the screen actually change),focus_confirmed(was the right window foreground), and awarningwhen either is off — never a blindok:true. Checkverifiedbefore re-issuing an action. - Anchor to a window —
ghost_key,ghost_find, andghost_actall takewindow; with it, input/resolution is guaranteed to target that window or the call errors. Use it for any multi-window flow. - Disambiguate duplicates —
indexselects the nth match when several elements share a name/role (multiple "Close Tab" buttons); responses carry amatchescount. - Read, don't screenshot —
ghost_see mode=textextracts a window/page's readable text straight from the accessibility tree: faster and ~10x cheaper in tokens than images. - Latency is visible: every response carries
ms, andescalated: trueflags when a find had to pay a network VLM round trip (local tiers: cache → UIA → OCR are all on-device). - Windows never disappear: minimized windows stay in
ghost_window list(withstate) andop=focusauto-restores them. - Stop always works:
ghost_stoppreempts the in-flight call the moment it arrives (dedicated stdin reader), and Ctrl+Alt+G remains the OS-level kill switch.
Agent harnesses (OpenClaw, Hermes/cua-driver, and any MCP client) mount a computer-use tool to let an LLM operate the desktop. Ghost is that tool for Windows — and it can act without stealing your focus or moving your cursor, so an agent drives an app while you keep working in another window.
- True background via posted window messages. Real Win32 controls are driven
with
BM_CLICK/WM_LBUTTONDOWN·UP(click) andWM_SETTEXT(type). These do not activate the window — unlike UIAInvoke/SetValue, whose providers pull the window to the foreground. - Verified even while occluded.
typeis confirmed by reading the control's value back;clickby aPrintWindowbefore/after delta that renders a window that isn't visible. Every response carriesverified,focus_preserved,cursor_preserved— Ghost never claims a background action it can't confirm. - Honest about the edge. Windowless controls (UWP/WinUI/Chromium — no window handle) can't be message-posted by any tool; there Ghost falls back to UIA dispatch (which activates the window) and says so in the response. Classic Win32 line-of-business apps — the software that has no API and most needs automating — drive cleanly in the background.
Supports click, type, double_click, right_click, and hover (posted mouse
messages), plus ghost_key background=true for single keys (Enter/Tab/F-keys/char
via WM_KEYDOWN/WM_CHAR). Modifier combos (Ctrl+C) are rejected in background —
posting can't set the modifier state apps read, so a combo would silently break;
use foreground for those.
Description-based grounding works with any tool-capable vision model behind an
OpenAI-compatible endpoint — NVIDIA (free default), OpenAI, Gemini, Groq, or a
local vLLM / Ollama / LM Studio server — or Anthropic. Point GHOST_VISION_BASE_URL
GHOST_VISION_MODELat your endpoint and setGHOST_VISION_API_KEY(a keyless local server needs only the base URL). No vendor lock-in.
Press Ctrl+Alt+G at any time to immediately halt all automation.
- All queued actions are cancelled
- Any held modifier keys (Shift, Ctrl, Alt) are released immediately
- No stuck keys, no stuck modifier states
session.find(By::name("Save")).await? // by accessible name (substring)
session.find(By::role("edit")).await? // by UIA control type
session.find(By::role("button")).await?From the CLI: ghost click --name "Save" or ghost click --role button.
Write reproducible multi-step flows as JSON. The FSM executor supports retries, timeouts, and JSONLogic conditions for abort_if / retry_if.
{
"ops": [
{ "op": "launch", "exe": "notepad.exe" },
{ "op": "focus_window", "name": "Notepad" },
{ "op": "type", "role": "edit", "text": "hello" },
{ "op": "hotkey", "mods": ["Ctrl"], "key": "s" }
]
}Run with ghost run flow.json, POST /run, or ghost_execute_intent over MCP.
ghost-cli ghost-http ghost-mcp Rust SDK
\ | / |
\ | / |
+-----> ghost-session <----------------+ ← safe Rust API
|
ghost-core ← Win32 FFI: UIA, SendInput, DXGI
|
Windows OS
Supporting crates: ghost-cache (UIA snapshot + delta), ghost-intent (FSM + JSONLogic executor).
When you locate an element by natural-language description (ghost_find description="the blue submit button", or when a name/text lookup misses and
escalates to the VLM), Ghost does not ask the model to guess pixel
coordinates — models are unreliable at that (in testing, a plain "give me the
coordinates of the equals button" landed ~250px off the target). Instead it uses
Set-of-Marks: it overlays numbered badges on the window's detected elements,
sends that marked screenshot plus each badge's accessible-name label, and asks
the model which number matches. The number maps back to that element's exact
rect, so the result is a real on-element coordinate, not a regression guess.
In a live check on Calculator, four descriptions ("the equals button", "the plus button", "the number seven key", "the multiply button") each landed exactly on the correct button — versus ~250px off with coordinate regression.
Honest scope: when detected elements carry accessible names (most apps), the labels do much of the disambiguation; for unlabeled icons the model leans on the badge's visual position/appearance.
Canvas / no-accessibility-tree apps. When the UIA tree is sparse (custom-drawn
UIs, remote-desktop surfaces, game canvases), Ghost augments the Set-of-Marks
candidates with a built-in CPU classical-CV detector (ghost_ground::cv_detect):
edge density → connected components → size/aspect filter, no GPU and no model
download. It gives the VLM real boxes to pick from where the accessibility tree
has nothing. It is coarser than a trained detector — an optional OmniParser ONNX
tier (--features yolo + GHOST_YOLO_MODEL) plugs into the same Set-of-Marks
path when a GPU model is available. The CV-marks → VLM-pick end-to-end needs a
configured vision key.
bench/ holds a reproducible end-to-end benchmark: it drives the real
ghost-mcp binary through 14 Windows desktop tasks and scores each by
re-observing the actual result (does the Calculator display really read 42?
is the typed value really present?), never by trusting a tool call's return.
Latest run (see bench/results/latest.md):
14/14 tasks passed (100%), median ~2.7 s per task (full wall-clock incl. app launch) — perception, click/keyboard action+verify, waits, window management (list/minimize/restore), text extraction, disambiguation, flow chaining, clipboard round-trip, structured errors, element screenshots, and value assertions.
And it proves it can fail: --self-test runs deliberately-wrong negative
controls (assert the display reads 99 when it reads 42, etc.) and passes only if
the harness scores every one as FAIL — so the green run above is a real signal,
not a rubber stamp.
Reproduce on any Windows 10/11 machine:
cargo build --release -p ghost-mcp
python bench/run_bench.py # exit 0 iff every task passed
python bench/run_bench.py --self-test # exit 0 iff the harness caught every planted failurebench/soak.py drives many act-then-verify cycles and gates on the signals unit
tests can't see: how often verified comes back null/false, focus-loss rate,
error rate, whether each action's real effect happened (the display is
re-observed, never trusted from the return), and latency percentiles.
Latest (160 acts): PASS — verify-null 0.0, focus-loss 0.0, effect-mismatch 0 (100% correct), p50 85ms / p95 117ms. See
bench/results/soak.md.
python bench/soak.py # exit 0 iff every reliability threshold holds
python bench/soak.py --cycles 250 # ~1000 acts
python bench/soak.py --self-test # exit 0 iff the harness flags a planted-wrong effectWe deliberately publish only Ghost's own measured numbers — never invented
columns for other tools. bench/README.md gives an honest protocol for
comparing against Playwright-MCP / Computer Use / UI-TARS, and explains why a
naive same-suite comparison isn't apples-to-apples (Playwright is browser-only;
vision agents need an API + VM).
| Operation | Measured |
|---|---|
| Region capture, GDI, any size | ~16.5 ms |
| Region capture, DXGI, 1600x900 | ~70-83 ms |
| BGRA→RGBA convert, 400x300 region | ~206 µs |
| JSONLogic eq/var | 32.2 ns |
| Intent compile (3op) | 1.49 µs |
End-to-end capture measurement (release, tests/capture_latency_probe.rs)
corrected the v0.10.0 assumption: the DXGI acquire dominates and hits a cliff on
large windows, so region captures (act-verify, screenshots, Set-of-Marks) route
through flat ~16.5ms GDI BitBlt in v0.11.0; full-screen still uses DXGI. Run the
convert microbench: cargo bench -p ghost-core --bench convert. Older baselines:
docs/benches/v030-baseline.md.
- Windows 10 build 19041 or later
- Linux with
at-spi2-coreand a desktop session (X11 or Wayland);xdg-desktop-portal-gnomeadditionally for Wayland input and capture - Rust stable (only for building from source)
MIT - Copyright 2026 Northtek