Drawgent: Coding agent on a live Excalidraw canvas

drawgent connects your own Claude Code, Codex or opencode (your install, login, config and repo) to an Excalidraw whiteboard. Ask for a diagram in the chat panel, or write AGENT: … next to the part of a drawing you want changed. The agent looks at the canvas (screenshot + scene), edits it live, checks the result, and marks the note DONE.

Prerequisite: one of claude, codex or opencode installed and logged in. The Claude and Codex bridges also need Node.js ≥ 18 (npm).

drawgent setup claude # once per agent: claude | codex | opencode cd ~/my-repo drawgent up # new agent session in this repo + canvas in your browser drawgent up --attach # or: pick one of your running sessions and connect the canvas to it

drawgent setup <agent>#

Checks everything once, fails with the exact fix when something is missing, and writes ~/.config/drawgent/config.toml:

The agent CLI. Your claude / codex / opencode on PATH.

Login. claude auth status, codex login status or opencode auth list.

Canvas tools for attached sessions. Only Codex needs a change: codex mcp add drawgent -- drawgent mcp. Claude and opencode get the tools at attach time.

Headless Chrome for the renderer. Setup uses your Chrome/Chromium if you have one. Otherwise it proposes:

Non-interactive: --chrome download | system | /path/to/chrome.

drawgent up refuses to start until setup succeeded for that agent, or if something setup recorded disappeared.

Runs in the current directory (the workspace):

The scene is kept in .drawgent/scene.json, which is git-ignored automatically.

Connects the canvas to a session you already run. Without an id it lists the sessions it finds (current directory first) and lets you pick one:

drawgent up --room 'https://excalidraw.com/#room=<id>,<key>'

drawgent joins the room as a collaborator (🤖 Agent, with a cursor that follows its edits). Humans can stay on excalidraw.com: their AGENT: notes reach your agent, and its edits appear there live.

Traffic is end-to-end encrypted with the room key. Empty rooms are loaded from and saved to excalidraw's Firestore storage.

The local editor mirrors the room and still has the chat panel.

Other commands#

drawgent mcp: stdio MCP server with the canvas tools. Agents launch it; it finds the running drawgent up by itself.

drawgent serve: low-level server with explicit agents (--agent claude,codex,opencode as set up, or name=command for any ACP agent); for scripts and containers.

Options for up / serve: --port, --room, --token (API bearer + ?token= in the URL), --permissions canvas|ask|all (default canvas: drawing tools auto-approved, anything else asked in the chat panel), --data, --settle-ms.

Optional: Docker#

docker compose up --build runs a canvas server (drawgent + Chromium, no agents), e.g. to host a shared canvas or a room bridge on a server. Agents are never bundled: they always run with your own setup.

npm ci && npm run build # editor + renderer pages -> dist/ (embedded into the binary) cargo install --path . # or: cargo build --release

Agent tools (MCP)#

get_scene, get_screenshot (vision; zoom with element_ids), add_elements (Excalidraw skeletons; arrows bind by id and are routed edge-to-edge), add_mermaid (auto-layout), update_elements (labels follow shapes, bound arrows re-route), delete_elements, clear_canvas, list_instructions, resolve_instruction, set_status.

GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= · POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear · GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status · POST /api/chat {agent?, text} · WS /ws (browser sync + chat events)

cargo test # fractional indices, room crypto/framing node scripts/smoke.mjs [url] # chat turn + AGENT: note against a running drawgent node scripts/e2e-browser.mjs [url] # real browser: chat panel + note typed on the canvas node scripts/room-e2e.mjs # fresh excalidraw.com room ↔ drawgent, both directions

Layout#

src/ (Rust):

web/: editor (main.jsx, chat.jsx) and renderer page (render.jsx).