AetherGrid, an open-source distributed AI workload orchestrator

A distributed AI workload orchestrator built around the problems that make scheduling hard at scale: exclusive execution ownership under failure, reconciliation after partial failures, and enforced resource limits. Not a CRUD tutorial with a scheduler theme.

AetherGrid takes workloads, matches them against available compute nodes based on resource requirements and constraints, and manages the full lifecycle: queued, scheduled, running, completed, failed, retried, cancelled. Jobs run through workers registered against nodes, and job execution ownership is enforced through time-bound leases rather than a simple assignment flag. Every route requires API key authentication, including the endpoint that issues keys.

Try it live: the full console is deployed and reachable at with real Postgres, real auth, and real API-key-gated endpoints.

Most scheduler side-projects are a single main.py script wrapped in a while True loop polling an in-memory dictionary. They work fine, right up until you need to swap the persistence engine, add a new constraint type, or figure out why a job silently disappeared, or why two workers picked up the same job at once.

AetherGrid was built around one rule: the domain logic doesn't know or care where the data lives. Jobs, nodes, workers, and the allocation algorithm are pure Python with zero infrastructure dependencies. The database is a detail, not the foundation. This project is a concrete demonstration that these architectural patterns aren't just conference-talk vocabulary; they're guardrails that keep a codebase understandable as it grows, and as its correctness requirements get harder.

Every non-obvious decision in this codebase, why a domain rule lives where it does, why an obvious-looking shortcut was rejected, what broke and how it got fixed, is written down at the moment it was made, not reconstructed afterward for a portfolio. The ADRs live in . A few worth reading directly if you want to see the reasoning, not just the conclusion:

If you're evaluating whether someone can operate at a systems level rather than a feature level, this is the fastest way to check.

API key authentication on every route except three probes: apart from three exempt probes, no endpoint in the system, including the one that issues keys, is reachable without a valid credential. The exempt probes are GET /, GET /health and GET /livez, and the interactive API docs are off unless NEUROMESH_ENABLE_API_DOCS is set (ADR 0077). The only way to mint the first key is a script run locally with direct database access, never over HTTP, closing the exact self-service-credential hole that pattern would otherwise leave open

Job lifecycle management: explicit state transitions (Queued → Scheduled → Running → Completed/Failed/Cancelled), including cancellation of a job already RUNNING via an explicit CANCELLING state, with configurable retry policies and priority-aware scheduling, plus cancel and retry actions reachable from the dashboard

Per-job lifecycle history: every job has a dedicated detail page (/jobs/{id}) showing its full real event timeline, JobCreated through completion, not just its current status

Constraint-aware, multi-dimensional best-fit allocator: matches workloads to nodes based on resource requirements and labels, skipping nodes that are draining or offline, then breaks ties among sufficient candidates by normalized tightness across every resource dimension a job actually requests, CPU, memory, and VRAM together, not CPU alone (see ADR 0043)

Node draining: a healthy node can be taken out of scheduling rotation for maintenance without killing it outright; the scheduler stops assigning it new work while anything already running on it continues to completion

Worker registration and heartbeats: registering a node automatically registers a worker against it, so it's immediately capable of claiming and executing work, not just existing as unused capacity

Standalone worker agent with exclusive job ownership: scripts/run_agent.py runs as a real, separate process, polling the API over HTTP for assigned work, executing it as a real local subprocess, and heartbeating on its own background thread for the agent's entire lifetime, independent of whatever job it's currently executing (see ADR 0019). This replaces the dashboard's client-side heartbeat as the liveness mechanism for any worker running it; a worker with no agent process attached still falls back to node liveness alone. Every worker is tagged with an explicit managed_by field set at registration (DASHBOARD or AGENT); the in-process scheduler loop skips any worker marked AGENT entirely, so a standalone agent's jobs are executed exactly once, by the agent, never raced against the in-process path.

Lease-based execution ownership: when a worker accepts a job, it holds a renewable, expiring lease on that job, continuously renewed for the job's entire execution, so retries, reconnects, network failures, and jobs that simply run long can't result in two workers executing the same job

Explicit execution-start confirmation: POST /workers/{worker_id}/jobs/{job_id}/start lets whatever is actually executing a job, the in-process scheduler loop for dashboard-managed workers, a standalone agent for agent-managed ones (ADR 0019), confirm execution has genuinely begun. This is the one call that transitions a job from Scheduled to Running; assignment alone no longer does (see ADR 0019)

Real subprocess execution with enforced timeouts: jobs with a command run as real subprocesses, with a two-stage shutdown (graceful SIGTERM, then SIGKILL after a grace period) if a job overruns its execution timeout

Node liveness tracking: heartbeat-based health checks, automatic detection of offline nodes, and resource reclamation when work fails or nodes disappear

Reconciliation with bounded retries: jobs abandoned by a dead worker or an offline node are reclaimed back to the queue within their retry budget, and fail outright once that budget is exhausted, so a single unhealthy node can't cause a job to be reassigned and abandoned indefinitely, with the reclaim ordered to close a real race where a dying worker's renewal could land after reconciliation had already started reassigning its lease

Domain event recording: every lifecycle transition a job goes through, JobCreated, JobScheduled, WorkerAssigned, LeaseAcquired, LeaseReleased, JobCompleted/JobFailed, and JobReclaimed, is persisted as an immutable event at the exact point it happens

Live event feed: GET /events and a real-time Activity Feed on the dashboard, polling every 3 seconds, so the story an individual job tells on its own detail page is also visible as it happens for every job in the caller's tenant

Worker visibility: a dedicated Workers table showing every registered worker, its status, the node it belongs to, what it's running, and when it was last seen

Idempotent worker re-registration per node: an agent restarting after a crash reclaims its existing worker identity instead of leaving a dead row behind, using the same reclaim sequence reconciliation already trusts, lease deletion, resource release, job reclaim, so an abandoned job is fully cleaned up before the worker is recovered rather than silently discarded (see ADR 0030, ADR 0039)

Multi-page dashboard: real client-side routing (/, /nodes, /jobs, /jobs/{id}, /workers, /workers/{id}) instead of a single page, with active-route highlighting in the sidebar and a global command palette (Cmd+K / Ctrl+K) for jumping between top-level views without touching the mouse

Architecture

The system is split into four layers, with dependencies pointing inward:

Domain: Job, Node, Worker, Lease, Event, and ApiKey aggregates enforce their own invariants. The scheduling algorithm and job lifecycle state machine live here as plain Python, with no imports from FastAPI or psycopg. Delete the infrastructure layer entirely and the domain tests still pass.

Application: Services such as ScheduleJobService/SchedulerService, AssignWorkerService, AcquireLeaseService, StartJobService, DrainNodeService, ClusterHealthService, and AuthenticateApiKeyService coordinate domain objects and repositories without embedding business rules that belong one layer down. A WorkerExecutionLoop drives a worker through executing its assigned job as a real subprocess, continuously renewing its lease on a background thread for the job's entire runtime, recording the real outcome, and releasing the lease regardless of that outcome. A renewal that fails means the lease has already been reclaimed elsewhere, and the loop discards its result rather than risk persisting it against another worker's in-progress or completed work. A ReconciliationLoop catches the failure modes the happy path can't: crashed workers, expired leases, state left inconsistent by infrastructure failures.

Infrastructure: PostgreSQL implementations exist for every repository (Node, Job, Worker, Lease, Event, ApiKey, Tenant), written with raw psycopg instead of an ORM, a deliberate choice to keep query behavior and transaction boundaries visible rather than abstracted away. Node, Job, and Event additionally have SQLite implementations for local development; ApiKey deliberately does not, since local development already runs against the same PostgreSQL backend production uses, and a SQLite path would reintroduce the environment drift that consolidation was built to remove. Every repository is validated against a shared contract test suite run against each backend it supports, so switching between implementations, or trusting that they behave identically, is a tested guarantee rather than an assumption.

Presentation: FastAPI endpoints for jobs, nodes, workers, events, cluster health, and API keys that validate input, call an application service, and return a response. Every route requires a valid API key except GET /, GET /health and GET /livez, and the interactive API docs are off unless NEUROMESH_ENABLE_API_DOCS is set (ADR 0077). No business logic lives in this layer. The frontend mirrors the same discipline: api/.ts typed HTTP calls, hooks/.ts data-fetching hooks, and page/component composition, no business logic embedded in components either.

Every non-obvious decision, why domain owns scheduling instead of application, why raw psycopg over an ORM, how job lifecycle transitions are enforced, why leases exist instead of a simple assignment field, why renewal is a strict update rather than an upsert, why opaque tokens were chosen over JWTs, why job commands were deliberately kept unreachable from the public API until authentication existed, then reopened narrowly, first for workers reading only their own assigned job's command (ADR 0020), later for the public CreateJobRequest API itself (ADR 0028), is documented as an ADR in /docs/adr.

The execution engine runs real, arbitrary commands as subprocesses with real timeout enforcement. Job.command and Job.exit_code are fully tested at the service layer. Until recently, neither was exposed through the public CreateJobRequest API, a boundary enforced by a test asserting the field's absence, not left as a comment.

Shipping the capability before exposing it was the deliberate call: build it correctly, prove it works, defer the exposure decision until it can be made deliberately rather than by default (ADR 0012).

Authentication closed the first gap. Every route except GET /, GET /health and GET /livez now requires a valid API key, the interactive docs are off by default (ADR 0077), and the only way to mint one without already holding one is a script run locally with direct database access, never over HTTP (ADR 0015). That left a sharper question open: what should any authenticated caller be trusted to do, while the system still had a single key tier with no role distinction.

That question has been answered twice, narrowly each time. ADR 0020 let an assigned worker read the one command already set for its own job, nothing broader. ADR 002