Perseus Pisces icon

Desktop automation
that acts like a human

Perseus Pisces captures the screen, asks an LLM what to do, and drives the real mouse and keyboard. Then it records the task as a semantic action graph and replays it deterministically — with drift detection when the UI shifts underneath it. Every action is gated by an auditable whitelist.

Download Get started View on GitHub
0 external dependencies Go stdlib only Windows · macOS · Linux empty go.sum
Record & replay

Record once. Replay deterministically.
Notice when it drifts.

The core of Perseus Pisces is a semantic action graph — a recording that matches what the screen means, not where its pixels were. It survives moved windows, restyled themes, and different resolutions.

  • Each step stores a semantic target (role, label, context) plus a perceptual hash of the screen — not a brittle (x, y).
  • Matching live state replays deterministically; known detours take a recorded branch.
  • The LLM is consulted only at genuine branch points, constrained to the recording's scope.
  • Drift detection surfaces when the UI has moved on — instead of silently clicking the wrong thing.
Read the full spec ↗ 
// One recorded step — matched by meaning, not pixels
{
  "action": { "type": "mouse_click", "reason": "Export report" },
  "target":  { "role": "button", "label": "Export", "context": "top-right" },
  "screen_state_before": { "perceptual_hash": "c3e1a0…" }
}

# Replay anywhere — parameters fill the blanks
$ perseus replay f3a1 --param filename=june-2026.csv
Replay finished: 3 steps, drift=false, 0 warnings
Capabilities

Everything an operator needs

Built from twelve focused subsystems — each using only the Go standard library and OS primitives.

Human-like input

Mouse moves along randomized Bézier curves with jitter and overshoot; keystrokes carry natural 40–220 ms delays.

Whitelist & audit

Every action clears a glob-rule whitelist and is written to an append-only NDJSON log before it ever executes.

Record & replay

Capture tasks as a semantic action graph and replay them with perceptual-hash state matching and drift detection.

Multi-provider LLM

Anthropic, OpenAI, and GitHub Copilot — all with vision, reasoning controls, and structured action output.

MCP server

Expose the desktop as Model Context Protocol tools over stdio, so any MCP client can drive it — pure stdlib JSON-RPC.

Local web UI

A dark, precision-tooling control room embedded in the binary via go:embed. Vanilla JS, no CDN, no frameworks.

Zero dependencies

No go get, no npm, no Docker. GUI primitives are OS syscalls; go.sum stays empty and builds are reproducible.

Cross-platform

One codebase compiles to Windows (syscall), macOS (CoreGraphics), and Linux (X11/XTest, Wayland fallback).

The loop

Screenshot → decision → action

Each iteration mirrors how a person works at a computer — observe, decide, act, verify.

Capture

Take a screenshot of the live screen.

Reason

The LLM reads the screen and returns a structured action.

Clear

The whitelist approves, denies, or pauses for the user.

Act

Human-like mouse and keyboard input executes it.

Audit

Screenshot, decision, and result are logged.

Security

Designed to resist prompt injection

The agent reads untrusted pixels and can drive real input and shell commands — so containment is built into every layer, not bolted on.

Default-deny whitelist

Every action — replayed, agent-driven, or via MCP — clears an allow / deny / ask glob-rule engine before it executes.

Injection-contained replay

During replay the model only sees the screen at genuine branch points, scoped to the recording — on-screen text can't hijack the run.

No shell interpolation

Shell actions run as explicit argv, never re-parsed by a shell — removing the classic command-injection surface.

Loopback · audited · encrypted

The server binds 127.0.0.1; actions stream to an append-only redacted NDJSON audit; secrets are AES-256-GCM encrypted at rest.

Model Context Protocol

Drive your desktop from any MCP client

  • Exposes screenshot, mouse_*, type_string, key_press, shell_command and more as MCP tools.
  • Pure standard-library JSON-RPC 2.0 over stdio — no SDK, no dependencies.
  • Every tool call still passes the same whitelist and audit engine.
  • Works with Claude Desktop, IDEs, and custom agents.
// claude_desktop_config.json
{
  "mcpServers": {
    "perseus": {
      "command": "perseus",
      "args": ["mcp"]
    }
  }
}
Quickstart

Build and run in seconds

Requires Go 1.22+. No other toolchain, runtime, or package manager. 

# Download a prebuilt binary from the releases page,
# or build from source (Go 1.22+, zero dependencies):
git clone https://github.com/DebajyotiSaikia/computer-use
cd computer-use
go build -trimpath -ldflags="-s -w" -o perseus .

# Start the agent + local web UI
./perseus start

# Authenticate a provider
./perseus auth anthropic

# Or expose your desktop to any MCP client
./perseus mcp
View on GitHub