Prbot – an agent that fixes PR bugs and proves the fix in a Docker sandbox

A graph-based multi-agent system that reviews a pull request, writes a patch for the security and performance bugs it finds, proves the patch doesn't break the build by running the test suite in a network-isolated Docker container, and iterates on failures before anything is published to GitHub.

flowchart TD S([START]) --> BL["Baseline gate<br/>tests on unpatched PR (and base branch)"] BL -->|suite already failing| F BL -->|green| A["Agent A · Reviewer<br/>structured Findings (Pydantic)"] A -->|no actionable findings| F["Finalize<br/>report · metrics · review comments · fix PR"] A -->|findings at or above min severity| B["Agent B · Patcher<br/>ReAct: read_file · search_code · write_patch"] B --> C["Agent C · Tester<br/>apply patch in temp checkout →<br/>pytest in Docker (--network none, timeout)"] C -->|tests pass| F C -->|"tests fail: all failed diffs + logs fed back, must pivot (attempts 1-2)"| B C -->|attempts exhausted| F F --> E([END])

Loading

Stack: Python · LangGraph · LangChain tool calling · OpenAI (gpt-4.1-mini default) · Pydantic structured outputs · PyGithub (GitHub App auth) · Docker · Pytest

Live demo target: ayanag5076-commits/prbot-demo-shop PR #1

Add an OPENAI_API_KEY secret (Settings -> Secrets and variables -> Actions).

Copy examples/prbot-review.yml to .github/workflows/:

on: { pull_request: { types: [opened, synchronize, reopened] } } permissions: { contents: read, pull-requests: write } jobs: prbot: runs-on: ubuntu-latest steps:

• uses: ayanag5076-commits/prbot@main with: openai-api-key: ${{ secrets.OPENAI_API_KEY }}

Every PR gets inline review comments, a verified fix.patch artifact, and the report in the job summary. Set open-fix-pr: "true" (plus contents: write) to have prbot open a follow-up PR with the fix. Secrets aren't exposed to workflows triggered from forks, so on public repos prbot reviews PRs from branches in the repo.

The platform tracks and records execution safety metrics via a local JSONL telemetry engine to audit agent efficiency and resource overhead.

python -m prbot review --dry-run --path demo_repo --diff demo_repo/pr.diff --repeat 10 # 10 live runs (Docker + OpenAI) python -m prbot bench-sandbox --path demo_repo --runs 10 # Docker vs host overhead python -m prbot metrics --update-readme # fills the table above

Reproduce the demo (no API key, no GitHub token)

git clone https://github.com/ayanag5076-commits/prbot.git && cd prbot python -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" python -m prbot review --repo ayanag5076-commits/prbot-demo-shop --pr 1 --dry-run --offline --no-docker

What that does: reads the public PR anonymously, runs the baseline tests, finds 3 seeded bugs, and runs the real ReAct loop, patch validation and test runner. The first patch deliberately uses MySQL's %s placeholder in SQLite, the tests fail, every failed diff and its log go back to the patcher, and attempt 2 passes. --offline swaps the LLM for a heuristic reviewer and a scripted model. --dry-run guarantees nothing is written to GitHub. Drop --no-docker once Docker is installed (docker build -t prbot-tester:py3.12 -f docker/tester.Dockerfile .).

status: patched | findings: 3 | attempts: 2/3 [high ] performance app/reports.py:10 N+1 query inside a loop [critical] security app/users.py:7 SQL injection via f-string query [medium ] performance app/users.py:14 Quadratic nested loop over the same sequence attempt 1: tests FAIL (exit 1, local) attempt 2: tests PASS (exit 0, local)

The full report (findings, self-correction table, verified diff, test logs) lands in prbot-out/report.md.

The diagram at the top is the exact StateGraph in prbot/graph.py.

GitHub App, not PATs. Installed per repository. Each run mints a token scoped to that repo with contents:read + pull_requests:write; contents:write is requested only for --open-fix-pr. Missing permissions fail fast. See docs/github-app.md.

Secrets stay local. OPENAI_API_KEY and the App private key load from .env / a .pem outside the repo. .env, .env.* and *.pem are git-ignored, and tests/test_session2.py::test_no_secrets_committed fails the build if a key-shaped string or key file ever appears in the tree.

Untrusted test code is sandboxed: no network, read-only root FS, dropped capabilities, resource limits, timeout.

Writes are opt-in: --post-review / --open-fix-pr plus a confirmation prompt; --dry-run can never write; unverified patches are never pushed (saved as unverified.patch).

Run against a real PR

cp .env.example .env # OPENAI_API_KEY, PRBOT_GITHUB_AUTH=app, GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_PATH python -m prbot review --repo owner/name --pr 12 # read-only, report to prbot-out/ python -m prbot review --repo owner/name --pr 12 --post-review # + inline review comments python -m prbot review --repo owner/name --pr 12 --post-review --open-fix-pr # + verified fix PR python -m prbot review --dry-run --path ../some-repo --model gpt-4.1 # any local folder

Exit codes: 0 clean/patched, 1 patch failed verification, 2 usage error, 3 aborted by the baseline gate.

Env var Default Meaning

OPENAI_API_KEY

OpenAI key (local .env only)

OPENAI_MODEL gpt-4.1-mini any tool-calling chat model

PRBOT_GITHUB_AUTH token app (recommended), token, or anonymous

GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY_PATH

GitHub App credentials

GITHUB_APP_INSTALLATION_ID auto otherwise discovered from --repo

GITHUB_TOKEN

PAT fallback for local development

PRBOT_MAX_ATTEMPTS 3 patch → test iterations

PRBOT_MIN_SEVERITY medium findings below this are reported, not patched

PRBOT_TEST_IMAGE prbot-tester:py3.12 tester image (bake test deps in; network is off)

PRBOT_TEST_TIMEOUT 120 seconds before the container is killed

PRBOT_PRICE_INPUT / _CACHED / _OUTPUT

USD per 1M tokens for models not in the built-in price table

Tests

pytest # offline suite (the live Docker egress test runs in CI)

Covers every conditional edge, the baseline abort and its PR comment, the attempt cap, failed-patch history and repeat rejection, in-attempt recovery from bad diffs, GitHub App least-privilege token requests, comment placement, metrics math, and the secret-leak guard.

Why a hand-rolled ReAct loop: the stop condition is "a validated patch exists", and tool errors must come back as observations. ~40 lines, fully traceable (tool_trace), and token usage is counted per call.

Why validate inside write_patch: most LLM patch failures are malformed diffs, not wrong logic. Catching them in-loop saves a full Docker run per mistake.

Why keep every failed diff: models fall into correction loops, resubmitting variations of the same broken patch. The full history plus a pivot instruction and exact-repeat rejection break the loop.

Why a baseline gate: if the suite is already red, no patch can turn it green; the agent would burn every attempt and tokens on code it didn't break.

Layout

prbot/ agents/ (reviewer, patcher, tester) · github/ (auth, client) · graph · models · tools · patching context · metrics · offline · cli demo_repo/ seeded bugs (SQLi, N+1, O(n²)) + pytest suite + pr.diff (mirrors prbot-demo-shop PR #1) docker/ tester image .github/workflows ci.yml: tests, secret guard, live sandbox egress check, Docker demo run docs/ github-app.md tests/ offline unit + integration tests

Contributing

Contributions are welcome. See CONTRIBUTING.md for setup and guidelines.