Server data from the Official MCP Registry
Safety-gated computer use for AI agents: screen OCR and vision, mouse, keyboard, window control.
About
Safety-gated computer use for AI agents: screen OCR and vision, mouse, keyboard, window control.
Security Report
Valid MCP server (1 strong, 1 medium validity signals). No known CVEs in dependencies. Imported from the Official MCP Registry.
3 files analyzed ยท No issues found
Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.
Permissions Required
This plugin requests these system permissions. Most are normal for its category.
Documentation
View on GitHubFrom the project's GitHub README.
๐ฅ๏ธ Computer use for all agents
A local remote-control system for AI agents: watch your computer's screen live and send mouse/keyboard commands to it. Everything runs on your own machine โ no data ever leaves it, no cloud middleman.
If You Are an AI Agent
Read AGENT_GUIDE.md first โ it is written for you: endpoints, safety gates and working recipes. This README is the human-facing reference.
First ask the server what it is โ GET /api/capabilities โ platform.
Windows: follow the whole guide as written. Linux: check
echo $XDG_SESSION_TYPE โ X11 is supported (guide โ
Linux (X11) Usage); Wayland is
unsupported: the server refuses to start, HTTP 501
UNSUPPORTED_DISPLAY_SERVER (switch to an X11 session). Full routing lives
in AGENT_GUIDE.md โ Find Your Environment First.
Why Screen Control?
AI agents today can write code and call APIs โ but they can't see or touch your desktop. Screen Control gives any agent general-purpose computer use over a clean, safety-gated HTTP/MCP interface:
- Perceive โ OCR for text, single frames or a live MJPEG stream for vision-capable models, and a text-only diff endpoint for models that can't consume images at all.
- Act โ absolute and relative mouse, Unicode-safe keyboard, window management, background (focus-free) control, virtual desktops.
- Stay safe โ token auth, blocked deadly shortcuts, focus guard, a stuck-input watchdog and an emergency failsafe are all enforced server-side, no matter how confused the agent gets.
One process, zero configuration, works with any language that can speak HTTP โ or natively through MCP in Claude Desktop, Cursor, VS Code and cloud agents.
Performance Is Agent-Bound
Screen Control is the perception and actuation layer โ the eyes and hands. The effective speed and capability of any agent using it are bounded by that agent itself and by the environment it runs in:
- Thinking speed โ one action per agent "turn": the perceive โ plan โ act โ verify loop lives in the agent, so model inference latency and reasoning depth directly set the pace. The API itself adds only milliseconds per call.
- Context capacity โ screen readings (OCR text, frames, diffs) consume the agent's context window; a larger window means more situational awareness before verification degrades.
- Runtime environment โ network latency, MCP/HTTP round-trip overhead, tool-call limits and hosting constraints all stack on top of the loop.
In practice this means: the same repo makes a fast reasoning model fast and capable, and makes a slow model slow โ the toolchain is not the bottleneck. Real-time or action-heavy tasks need an agent with fast inference and tight tool-loop latency; slower agents should prefer deliberate, verification-heavy tasks.
Table of Contents
- If You Are an AI Agent
- Why Screen Control?
- Features
- Architecture
- Installation
- Linux (X11)
- Quick Start
- API Reference
- ๐ค For AI Agents
- MCP Support (One-Click Cloud Agents)
- Security Model
- Game Mode Guide
- Vision Access Guide
- Troubleshooting
- Testing
- Contributing
- License
Features
| Feature | Description |
|---|---|
| ๐ผ๏ธ Live screen feed | Continuously refreshing screenshot in the browser |
| ๐ฑ๏ธ Mouse control | Click, right-click, double-click, scroll, drag & drop via live screenshot |
| โจ๏ธ Keyboard control | Text typing (Unicode/Turkish included, layout-independent), keys and shortcuts (Ctrl+C, Alt+Tabโฆ) |
| ๐๏ธ OCR | Converts on-screen text to machine-readable format |
| ๐ท Vision access | Raw-pixel paths for image-capable models: single frames, MJPEG stream, text-based motion detection |
| ๐ช Window management | List, focus, safe close (WM_CLOSE), kill (task-manager style) |
| ๐ฅ๏ธ Focus-free control | Read/write background windows via PostMessage without stealing focus |
| ๐ฎ Game mode | Camera look via relative mouse movement, hold-to-move keys |
| ๐ Token auth | Every request requires X-Auth-Token (CSRF protection) |
| ๐ฆบ Stuck-input watchdog | Auto-releases held keys after 30 s of inactivity |
| ๐ Failsafe | Cursor to top-left corner aborts all commands (disabled in game mode) |
Architecture
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Browser (Web UI) โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Live โ โ Control โ โ Windows / Game Mode โ โ
โ โ View โ โ Panel โ โ Panel โ โ
โ โโโโโโฌโโโโโโ โโโโโโฌโโโโโโ โโโโโโโโโโโโโฌโโโโโโโโโโโโโ โ
โ โ โ โ โ
โโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโ
โ โ โ
โผ โผ โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ HTTP API (Flask) โ
โ 127.0.0.1:8745 โ
โ โ
โ /api/screenshot /api/mouse /api/key โ
โ /api/vision/* /api/ocr /api/window โ
โ /api/game /api/held /api/release_all โ
โ /api/windows /api/desktops /api/desktop โ
โ โ
โ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ
โ โ Auth Layer โ โ Watchdog โ โ OCR Engine โ โ
โ โ (token) โ โ (30s auto) โ โ (RapidOCR) โ โ
โ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ โ
โผ โผ โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ control.py (Core) โ
โ โ
โ Screen: mss (fast capture), PIL (processing) โ
โ Mouse: pyautogui (absolute), SendInput (relative) โ
โ Keyboard: pyautogui + SendInput+KEYEVENTF_UNICODE โ
โ Windows: Win32 API (EnumWindows, SetForegroundWindow) โ
โ Background: PrintWindow (capture), PostMessage (input) โ
โ Virtual Desktops: pyvda โ
โ Game Mode: ClipCursor + MOUSE_MOVE_RELATIVE โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ backends/ (pluggable) โ
โ WindowsBackend โ LinuxBackend โ MacOSBackendโ
โ (full) โ (X11) โ (stub) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Coordinates & Concurrency
Per-Monitor DPI awareness. control.py calls
SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2) at import time โ
before the pyautogui import, because pyautogui touches coordinate APIs
during import and would otherwise lock the process to the interpreter
manifest's default (system-aware). With PMv2 active, every coordinate in the
system is a physical pixel end to end: mss capture, OCR bounding boxes,
pyautogui/SendInput clicks, ClipCursor. On High-DPI displays (125%/150%
scaling) nothing drifts between what OCR reports and where the mouse clicks.
Lock architecture. The server uses two independent locks instead of one global lock:
| Lock | Protects | Endpoints |
|---|---|---|
_input_lock | mouse, keyboard, game mode, window ops | /api/mouse, /api/key, /api/game, /api/window/post, ... |
_read_lock | capture, OCR, vision, enumeration | /api/screenshot, /api/ocr, /api/vision/*, /api/windows, ... |
A slow OCR (3โ5 s on a busy screen) no longer freezes concurrent screenshot or vision reads โ reads queue behind reads, inputs behind inputs.
Live-Loop Working Principle
This system is designed for a live perceive-act loop, not pre-written command chains:
- READ โ OCR or vision reads the screen before and after every action
- ONE ACTION โ each round sends a single command
- VERIFY โ acceptance is "it appeared on screen", not "I sent it"
- ADAPT โ if verification fails, the next step changes based on what is actually seen
This is enforced by the expect_hwnd guard: typing is refused (409) if
the foreground window doesn't match the target.
Installation
Pick the version that fits how your agent works:
| You are... | Version | Install |
|---|---|---|
| An AI agent user (Claude Code, Cursor, skills-capable agents) | Skill | Skill version โ copy skills/computer-use/ into the agent's skills folder |
| An MCP-capable app (Claude Desktop, VS Code, cloud agents) | MCP | MCP Support โ mcp_server.py over stdio or streamable-HTTP |
| A developer / API client | REST | this page โ pip install -r requirements.txt + python server.py |
Skill version
The portable agent skill lives in skills/computer-use/
(SKILL.md plus a full API reference). It teaches any skills-capable agent
the perceive-act-verify loop, the safety rules and the endpoint catalogue โ
the agent then drives a running server.
# Claude Code โ user-level (available in every project)
cp -r skills/computer-use ~/.claude/skills/
# Claude Code โ this project only
mkdir -p .claude/skills && cp -r skills/computer-use .claude/skills/
For other skill-capable agents, copy the folder into that agent's own skills
directory. Either way the server must be running (python server.py) โ
the skill provides instructions, not a backend.
REST server
cd screen-control
pip install -r requirements.txt
Requirements
| Package | Purpose | Required? |
|---|---|---|
mss | Fast screen capture | โ Yes |
pyautogui | Mouse/keyboard control | โ Yes |
pyvda | Virtual desktop management | โ Yes |
flask | HTTP server | โ Yes |
Pillow | Image processing | โ Yes |
rapidocr-onnxruntime | OCR (screen text reading) | โ ๏ธ Optional |
Note: The OCR package is large and may take a while to install. If it fails, everything else still works โ only the OCR feature is unavailable.
System Requirements
- OS: Windows 10/11 (x64), or Linux (Ubuntu/Mint family, X11 session โ see Linux (X11))
- Python: 3.10+
- Display: Any resolution; the system adapts automatically
Linux (X11)
Linux is supported with limits (ROADMAP Phase 5) โ tested on Linux Mint with Cinnamon in an X11 session; other X11 desktop environments are expected to work. Windows remains the reference backend.
Requirements
- OS: Ubuntu / Linux Mint family (apt-based)
- Session: X11 โ check this first:
echo $XDG_SESSION_TYPE # must print: x11
- Wayland? Not supported โ the server fails fast instead of half-working (see Known limitations).
- System packages (installed by the setup script below):
sudo apt install -y xdotool wmctrl imagemagick python3-venv python3-tk python3-dev
xdotool drives input/focus, wmctrl window management, ImageMagick's
import captures window pixels (optional โ an mss region fallback exists),
python3-venv creates the virtualenv and python3-dev lets evdev
(game mode) compile from source.
One-command setup
cd screen-control
./install-linux.sh
The script is idempotent and checks, in order: Wayland (exits with a clear
message), DISPLAY, the apt packages, the .venv virtualenv,
pip install -r requirements.txt, and finally reports whether game mode can
work.
uinput permissions (game mode)
Game mode writes to /dev/uinput. Verify access with:
[ -w /dev/uinput ] && echo "game mode: OK" || echo "game mode: unavailable"
If it is unavailable, either remediation works (spec ยง5.4):
sudo usermod -aG input $USER # then log out and back in
# โ or a udev rule:
echo 'KERNEL=="uinput", MODE="0660", GROUP="input"' | sudo tee /etc/udev/rules.d/99-uinput.rules
sudo udevadm control --reload && sudo udevadm trigger
Everything except game mode works without this access.
Run
./start-server.sh
# โ same API as Windows: http://127.0.0.1:8745 (X-Auth-Token from .token)
Known limitations (Linux X11)
| Limitation | Detail |
|---|---|
| X11 only | On Wayland the backend fails fast: ApiError("UNSUPPORTED_DISPLAY_SERVER") (HTTP 501), and server.py prints that message and exits instead of serving. Fix: log out โ session chooser at the login screen โ "Cinnamon on Xorg". Wayland support is ROADMAP Phase 6. |
| Background input = focus-first | Every /api/window/post action focuses the target window first, then sends input globally (X11 has no reliable posted-message path) โ i.e. focus stealing. capabilities.background_input is false. |
| No cursor clip in game mode | X11 cannot clip the cursor โ the game must capture the pointer itself. Emergency stop: physical Esc/Alt+Tab or POST /api/release_all. |
| No virtual desktops | capabilities.virtual_desktops is false; GET /api/desktops and POST /api/desktop return 503. |
| Fractional scaling untested | The verification display runs at scaling 1.0; coordinates under fractional scaling are unverified. |
capture_window (window capture) | Client area only, for both client=0 and client=1 โ decorations are not captured (on Windows client=0 includes the frame). Uses ImageMagick import when installed (usually the window's own pixels even when occluded); otherwise falls back to an mss region grab of the window's rectangle, so an occluded window shows whatever is on top. Failures map to HTTP 409. |
| Window geometry source | rect comes from xwininfo -root -tree absolute bounds โ outer/frame bounds (Windows GetWindowRect parity); client_to_screen uses xwininfo "Absolute upper-left". |
window_hotkey response | Returns {"ok","vk","vks"}; for the hotkey action vk (X11 keysym) exists only on Linux (Windows returns {"ok","vks"}) โ cross-platform code must read vks. |
Agent note: OCR is the slowest read (โ5โ6 s full-screen measured). Prefer
/api/vision/frameor/api/streamfor repeated reads, use region OCR, and batch reads between actions โ see AGENT_GUIDE.md โ Linux (X11) Usage.
Quick Start
# 1. Start the server
cd screen-control
python server.py
# 2. Open in browser
# http://127.0.0.1:8745
# 3. Or control via API
TOKEN=$(cat .token)
curl -H "X-Auth-Token: $TOKEN" http://127.0.0.1:8745/api/screenshot -o screen.jpg
API Reference
Capabilities
Returns the active backend name and what it can do. Agents should call this first (see ROADMAP.md for the multi-platform plan).
GET /api/capabilities
โ {"ok": true, "backend": "windows",
"capabilities": {"screen_capture": true, "game_mode": true, ...}}
Feature values: true (supported), false (absent), null (unknown โ
stub backend), "optional" (depends on an optional dependency).
Platform Support Matrix
| Capability | Windows | Linux X11 | Linux Waylandโ | macOS |
|---|---|---|---|---|
| Screen capture | Full | Full | Portal-dependent | Permission required |
| OCR | Full/optional | Full/optional | Full/optional | Full/optional |
| Mouse control | Full | Full | Restricted | Accessibility permission |
| Keyboard control | Full | Full | Restricted | Accessibility permission |
| Window enumeration | Full | WM-dependent | Limited | Accessibility/API-dependent |
| Background input | Strong | Focus-first (stealing) | Usually unavailable | Limited |
| Virtual desktops | Supported | No (503) | DE/WM-dependent | Spaces-specific |
| Game mode | Supported | Experimental | Limited | Experimental |
Linux X11 is implemented and supported with limits โ setup, requirements and the full limitations list are in Linux (X11). Wayland fails fast with
UNSUPPORTED_DISPLAY_SERVER(501), and macOS remains a fail-closed stub: every operation returnsBACKEND_UNAVAILABLE(501) until ROADMAP Phase 7. Windows is the reference backend.โ Wayland column is hypothetical capability planning only โ the server never starts on Wayland (see the note above), so no Wayland cell is currently reachable.
Authentication
Every request must include the X-Auth-Token header. The token is
generated on each server start and written to .token.
TOKEN=$(cat .token)
| Code | Meaning |
|---|---|
| 401 | Missing or invalid token |
| 415 | POST without Content-Type: application/json |
Token bootstrap (for the bundled web UI):
GET /token
โ {"ok": true, "token": "abc123..."}
The
/tokenendpoint answers only loopback callers on a loopback bind: the check uses the accepted socket's peer address (a LAN client cannot spoof it), theHostheader guards against DNS rebinding, and the response is never cached. When the server is bound to0.0.0.0or is reached from a non-loopback peer,/tokenreturns 403.
Screen Capture
GET /api/screenshot
Returns a JPEG screenshot.
| Parameter | Type | Default | Description |
|---|---|---|---|
monitor | int | 1 | Monitor index |
region | string | โ | x,y,w,h sub-region |
curl -H "X-Auth-Token: $TOKEN" -o screen.jpg http://127.0.0.1:8745/api/screenshot
curl -H "X-Auth-Token: $TOKEN" "http://127.0.0.1:8745/api/screenshot?region=0,0,800,600"
GET /api/info
Returns screen dimensions and system state.
{"ok": true, "width": 1920, "height": 1080, "ocr_available": true,
"failsafe": true, "game_mode": false}
Mouse Control
POST /api/mouse
| action | Required params | Optional params | Description |
|---|---|---|---|
move | x, y | duration (default 0.15) | Move cursor to absolute position |
click | x, y | button (left/right), clicks (default 1) | Click at position |
scroll | clicks | x, y | Scroll wheel (positive=up) |
drag | x1, y1, x, y | duration, button | Drag between two points |
down | button (default "left") | โ | Press and hold mouse button |
up | button (default "left") | โ | Release held mouse button |
# Click at center of screen
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"click","x":960,"y":540,"button":"left"}'
# Right-click
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"click","button":"right","x":960,"y":540}'
# Scroll down
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"scroll","clicks":-3}'
Keyboard Control
POST /api/key
| action | Required params | Description |
|---|---|---|
press | key | Press and release a key |
down | key | Hold a key down (tracked for watchdog) |
up | key | Release a held key |
hotkey | keys (array) | Key combination (e.g. ["ctrl","c"]) |
type | text | Type text (Unicode, layout-independent) |
| Optional param | Default | Description |
|---|---|---|
expect_hwnd | โ | Window handle to verify focus (409 if mismatch) |
interval | 0.03 | Delay between characters for type |
# Press Enter
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"press","key":"enter"}'
# Ctrl+C
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"hotkey","keys":["ctrl","c"]}'
# Type text (Turkish characters supported)
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"type","text":"Merhaba dรผnya"}'
# Hold W key down (for walking in games)
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","key":"w"}'
OCR (Screen Reading)
POST /api/ocr
Converts on-screen text to machine-readable format.
| Param | Type | Default | Description |
|---|---|---|---|
region | array | โ | [x, y, w, h] sub-region (faster) |
{
"ok": true,
"text": "Hello World\nFile Edit View",
"lines": ["Hello World", "File Edit View"],
"items": [
{"text": "Hello World", "x": 960, "y": 40},
{"text": "File Edit View", "x": 100, "y": 15}
]
}
# Full screen OCR
curl -X POST http://127.0.0.1:8745/api/ocr -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{}'
# Region-only (faster, ~10x for small regions)
curl -X POST http://127.0.0.1:8745/api/ocr -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"region":[0,0,800,100]}'
Vision Access (Image Models)
Three endpoints for models that can consume images:
| Endpoint | Description |
|---|---|
GET /api/vision/frame | Single JPEG frame (raw or base64) |
GET /api/stream | MJPEG live stream |
POST /api/vision/diff | Text-based motion detection (no vision needed) |
GET /api/vision/frame
| Param | Default | Description |
|---|---|---|
scale | 1.0 | Downscale factor (0.5 = half size) |
gray | 0 | 1 for greyscale |
quality | 80 | JPEG quality (20-95) |
format | โ | base64 for JSON response |
region | โ | x,y,w,h sub-region |
# Half-size greyscale frame as base64 (for text-only models)
curl "http://127.0.0.1:8745/api/vision/frame?scale=0.5&gray=1&format=base64" \
-H "X-Auth-Token: $TOKEN"
GET /api/stream
MJPEG live stream. Drop into <img src> or consume frame-by-frame.
| Param | Default | Description |
|---|---|---|
fps | 10 | Frames per second (1-30) |
quality | 70 | JPEG quality |
scale | 1.0 | Downscale factor |
region | โ | x,y,w,h sub-region |
POST /api/vision/diff
Text-based motion detection โ no vision model required.
| Body | Description |
|---|---|
{} | Compare against last stored frame |
{"grab":"gray"} | Store current frame for next comparison |
{"b64_prev":"..."} | Compare against provided previous frame |
{
"ok": true,
"changed": true,
"changed_pct": 12.5,
"bbox": [100, 200, 400, 350],
"tiles": [
{"row": 2, "col": 4, "pct": 35.2, "center": [1000, 390]}
]
}
Window Management
GET /api/windows
List all visible windows.
{
"ok": true,
"windows": [
{
"hwnd": 123456,
"title": "My Application",
"process": "app.exe",
"pid": 7890,
"focused": true,
"rect": [0, 0, 1920, 1080],
"desktop": 1
}
]
}
POST /api/window
| action | Required | Optional | Description |
|---|---|---|---|
focus | hwnd | โ | Bring window to foreground |
close | hwnd | expect_title, expect_process | Safe close via WM_CLOSE |
kill | hwnd, pid | โ | Force kill (task-manager style) |
topmost | hwnd | โ | Set always-on-top |
untopmost | hwnd | โ | Remove always-on-top |
maximize | hwnd | โ | Maximise window |
# Focus a window
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"hwnd":12345,"action":"focus"}'
# Safe close (with title verification)
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"close","hwnd":12345,"expect_title":"Notepad"}'
# Kill process
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"kill","hwnd":12345,"pid":7890}'
POST /api/launch
Spawn an application, open a URL, or open a local file. Exactly one of
app | url | file. Never runs through a shell: app is a plain command
name resolved on PATH ([A-Za-z0-9._+-]+), url/file go through
xdg-open.
Launch policy: interpreter and executor names (
python*,node,sh/bash/cmd/powershell,env,xargs,sudo,curl, โฆ) are refused with 400 โ/api/launchis for GUI applications, not a remote code-execution endpoint.
| Field | Type | Description |
|---|---|---|
app | string | Command name; must exist on PATH (else 400) |
url | string | scheme://... (e.g. https://, steam://) |
file | string | Absolute path to an existing file |
args | list | Arguments for app only |
expect_title | string | Also wait for a window whose title contains it |
timeout | number | Wait for expect_title: default 10, max 30 s |
{"ok": true, "pid": 4321, "window": {"hwnd": ..., "title": "..."}}
With expect_title, window is the match or null (spawn succeeded, no
window yet) โ the HTTP status stays 200 either way.
curl -X POST http://127.0.0.1:8745/api/launch -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"app":"firefox","expect_title":"Mozilla Firefox","timeout":15}'
GET /api/window/wait
Block until a window whose title contains title (case-insensitive
substring) appears, then return it. timeout default 10 s, max 30 s;
200 with the window, or 408 OPERATION_TIMEOUT.
curl -H "X-Auth-Token: $TOKEN" \
"http://127.0.0.1:8745/api/window/wait?title=Notepad&timeout=10"
Focus-Free (Background) Control
Read and control windows without stealing focus โ the user keeps working on their main desktop.
Linux (X11) note: background control there is focus-first โ the target window is focused, then input is sent globally (focus stealing). Only background capture is truly focus-free. See Known limitations.
GET /api/window/capture
Capture a window via PrintWindow (works even on another virtual desktop).
| Param | Description |
|---|---|
hwnd (required) | Window handle |
client | 1 = client area only |
ocr | 1 = return OCR text instead of image |
# Capture window as PNG
curl "http://127.0.0.1:8745/api/window/capture?hwnd=12345" \
-H "X-Auth-Token: $TOKEN" -o window.png
# Capture + OCR in one call
curl "http://127.0.0.1:8745/api/window/capture?hwnd=12345&ocr=1" \
-H "X-Auth-Token: $TOKEN"
POST /api/window/post
Send input to a window, choosing the delivery path automatically.
| action | Description |
|---|---|
type | Type text (Unicode-safe) |
key | Send a key press |
hotkey | Send a key combination |
click | Click at client coordinates |
scroll | Scroll the window |
drag | Drag inside the window |
Optional mode parameter controls routing:
| mode | Behavior |
|---|---|
auto (default) | Decided by input-mode probe (see below) |
background | Force PostMessage path (window keeps focus/z-order) |
focused | Force focus + SendInput path |
# Type into a background Notepad
curl -X POST http://127.0.0.1:8745/api/window/post -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"hwnd":12345,"action":"type","text":"Hello from background!"}'
Routing rules (mode=auto):
postmessageโ classic Win32 app: background PostMessage, no focus change.uiaโ WinUI/UWP/XAML surface (single DirectX canvas, no Win32 child controls): posted messages are silently swallowed, so the window is focused and the action is replayed through SendInput (client coords converted to screen). This is the documented fallback for modern apps.focusedโ window is already foreground: focused SendInput path.invalidโ HTTP 409; not a reachable top-level window.
GET /api/window/input-mode
Classify how a window receives input before posting to it. Returns one of
focused | postmessage | uia | invalid.
curl "http://127.0.0.1:8745/api/window/input-mode?hwnd=12345" \
-H "X-Auth-Token: $TOKEN"
WinUI note: New Notepad (and other XAML-hosted apps) has no classic child Edit control to post to โ the whole UI is one DirectX surface.
input-modereportsuiafor these;/api/window/postthen automatically uses the focused SendInput path./api/window/childrenremains useful for classic apps with real child controls.
GET /api/window/children
List child controls of a window (class name + title + hwnd).
curl "http://127.0.0.1:8745/api/window/children?hwnd=12345" -H "X-Auth-Token: $TOKEN"
Virtual Desktops
GET /api/desktops
List all virtual desktops.
POST /api/desktop
| action | Params | Description |
|---|---|---|
switch | number | Switch to desktop N |
create | โ | Create a new desktop |
curl http://127.0.0.1:8745/api/desktops -H "X-Auth-Token: $TOKEN"
curl -X POST http://127.0.0.1:8745/api/desktop -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"switch","number":2}'
Game Mode
| action | Params | Description |
|---|---|---|
start | sensitivity (default 12) | Lock cursor to center, enable game input |
move | dx, dy, sensitivity | Rotate camera (relative mouse) |
stop | โ | Release cursor + all held input |
heartbeat | โ | Keep-alive for long holds |
# Start game mode
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"start","sensitivity":12}'
# Look right
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"move","dx":50,"dy":0}'
# Hold W to walk forward
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","key":"w"}'
# ... later ...
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"up","key":"w"}'
# Stop game mode
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"stop"}'
Safety Endpoints
GET /api/held
Returns currently held keys/buttons and watchdog status.
{
"ok": true,
"keys": ["w", "shift"],
"buttons": ["left"],
"game_mode": true,
"idle_seconds": 5.2,
"watchdog_count": 0,
"last_watchdog": null
}
POST /api/release_all
Emergency: release everything (held keys, mouse buttons, game-mode cursor lock).
curl -X POST http://127.0.0.1:8745/api/release_all -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{}'
๐ค For AI Agents
A dedicated, comprehensive guide for AI agents (LLMs, vision models, automation frameworks) is available in AGENT_GUIDE.md.
Prefer loading this as a skill instead of browsing docs? Install the Skill version.
It covers:
- Environment routing โ find Windows vs Linux (X11/Wayland) first
- Perceive-act loop (read โ plan โ act โ verify)
- Focus guard (
expect_hwnd) to prevent wrong-window accidents - App automation and game control workflows
- Linux (X11) usage: slow-OCR batching, capture semantics, game-mode caveats
- Vision access for image-capable models
- Text-based motion detection
- Bandwidth optimization
- Complete curl examples
MCP Support (One-Click Cloud Agents)
Model Context Protocol (MCP) turns this project into a plug-and-play toolbox for any MCP-capable agent: Claude Desktop, Claude Code, Cursor, VS Code Copilot Agent mode, custom cloud agents โ no custom glue code, no curl scripts. The agent discovers and calls the tools natively.
How it works
MCP agent (cloud or desktop)
โ MCP protocol (stdio or streamable-HTTP)
โผ
mcp_server.py โ thin wrapper: tools โ HTTP calls, token auto-read
โ REST + X-Auth-Token (localhost only)
โผ
server.py โ the single source of truth:
auth, locks, watchdog, focus guard, all safety rules
mcp_server.py adds no new powers โ every safety mechanism
(auth token, input/read locks, watchdog, Alt+F4 block, focus guard,
failsafe) stays enforced by server.py.
Setup
pip install mcp # optional dependency (see requirements.txt)
python server.py # start the REST server first (it writes .token)
The MCP server auto-reads the token from .token (or the
SCREEN_CONTROL_TOKEN env var) โ zero configuration.
Desktop agents (stdio transport)
Claude Desktop โ claude_desktop_config.json:
{
"mcpServers": {
"screen-control": {
"command": "python",
"args": ["C:/path/to/screen-control/mcp_server.py"]
}
}
}
Claude Code: claude mcp add screen-control -- python C:/path/to/screen-control/mcp_server.py
Cursor / VS Code: add the same entry to their MCP config files.
Remote / cloud agents (streamable-HTTP transport)
python mcp_server.py --http --port 8751
# MCP endpoint: http://127.0.0.1:8751/mcp
The HTTP transport is token-protected: every request must carry the
X-Auth-Token header (same token as the REST server). Query-string tokens
(?token=...) are rejected by design โ URLs leak into proxy/tunnel
logs, browser history and shared links, and this token grants full desktop
control. Clients that cannot send custom headers should run a local stdio
mcp_server.py instead. Only GET /health is open, for liveness probes.
DNS-rebinding protection is disabled on this transport deliberately โ
tunneled requests arrive with a foreign Host header, and the rebinding
threat is already covered by the token guard.
For a cloud agent, expose it through a tunnel:
cloudflared tunnel --url http://127.0.0.1:8751
# โ prints a https://<random>.trycloudflare.com URL
Then configure the agent's MCP connection with <tunnel-url>/mcp plus the
token from .token as a header (X-Auth-Token).
Headerless connectors (scoped keys)
For clients that cannot send custom headers (e.g. web connectors that only take an endpoint URL), create a scoped API key โ a persistent, optionally time-limited credential โ and embed it in the URL path:
# Create a 24-hour scoped key (requires the server to be running)
curl -X POST http://127.0.0.1:8745/api/keys -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"create","name":"spark","expires_in_hours":24,"scopes":["mcp"]}'
# Connector endpoint becomes:
# https://<tunnel-url>/mcp/<scoped-key>
Design guarantees (SC-06):
- The master session token is refused in URLs (403) โ only scoped keys may travel there
- Keys carry scopes (
rest,mcp): a key minted with{"scopes":["mcp"]}cannot call the REST API at all, and any scoped key is refused on admin endpoints (/api/keys,/api/session*) โ those accept the master token only - Scoped keys expire automatically; expired keys authenticate nothing
- Scoped keys are revocable by name at any moment via
POST /api/keys({"action":"revoke","name":"spark"}) โ revocation takes effect immediately on every endpoint - The
.apikeysfile stores only SHA-256 hashes, never raw keys, and is written with0600permissions
โ ๏ธ A tunnel exposes PC control to the internet. Keep the token secret, prefer short-lived tunnels and scoped keys for headerless connectors, and stop the server when not in use.
One-Command Startup (launcher, tunnel opt-in)
start-server.bat automates the whole setup and prints a paste-ready
scoped key for your cloud agent:
- Downloads
cloudflared.exefrom a pinned release and verifies its SHA-256 against Cloudflare's published checksum (no mutablelatestURL, no unverified binary) - Stops leftover instances from a previous run
- Starts the REST server (port 8745) and the MCP HTTP server (port 8751)
- Waits until both are healthy (
/tokenand/healthprobes) - Starts the quick tunnel only when you opt in with
start-server.bat --tunnel, extracts its public URL fromtunnel.log, and prints the summary:
============================================================
ALL SYSTEMS RUNNING
============================================================
Local REST API : http://127.0.0.1:8745
Local MCP : http://127.0.0.1:8751/mcp
Public MCP URL : https://<random>.trycloudflare.com/mcp
------------------------------------------------------------
PASTE INTO YOUR CLOUD AGENT (MCP connector settings)
------------------------------------------------------------
Endpoint : https://<random>.trycloudflare.com/mcp
Header : X-Auth-Token: <24-hour scoped key "launcher">
(the master token is NOT shown; revoke anytime)
------------------------------------------------------------
Without --tunnel nothing is exposed to the internet โ the launcher is a
local-only startup. The pasted credential is a 24-hour mcp-scoped key
(minted locally, revocable via POST /api/keys {"action":"revoke",...}).
stop-server.bat stops all three (REST, MCP, tunnel) in one go.
Available tools (16)
Documentation truncated โ see the full README on GitHub.
Reviews
No reviews yet
Be the first to review this server!
More Developer Tools MCP Servers
Rea
Freeby Morluto ยท Developer Tools
Reverse engineer anything from your terminal or agent with one CLI and MCP server.
Paperclip
Freeby Paperclipai ยท Developer Tools
Trending hip-hop artist momentum scores across four cultural dimensions.
Fetch
Freeby Modelcontextprotocol ยท Developer Tools
Web content fetching and conversion for efficient LLM usage
Git
Freeby Modelcontextprotocol ยท Developer Tools
Read, search, and manipulate Git repositories programmatically
Toleno
Freeby Toleno ยท Developer Tools
Toleno Network MCP Server โ Manage your Toleno mining account with Claude AI using natural language.
MCP Marketplace
Freeby mcp-marketplace ยท Developer Tools
Search and install MCP servers from inside your AI client.
