A migration job failed against a "healthy" Postgres

Back to the JournalJournal

/

Guides

Live dispatch/Oct 16, 2026/10 min read

Gate your app on a Postgres migration job with depends_on: service_completed_successfully

Gate your API on a one-shot Postgres migration with depends_on: service_completed_successfully. Tested on Hubfly: a failed deploy, the fix, real logs.

TL;DR - What you get: a 3-service Compose file (Postgres, a one-shot migration job, an API) where the API cannot start until the migration exits 0. - Who it's for: anyone whose app has ever booted against a database with a missing table. - Time: about 20 minutes. Our second rollout took 59 seconds from click to "Stack deployment finished".

Your app started before your schema did

You push a release. The API boots, runs its first query, and gets relation "todos" does not exist. The migration ran too, just 3 seconds later.

Most Compose files only say depends_on: [db]. That waits for the database container to start, not for the schema.

Docker's docs say so plainly: Compose waits until a dependency is running, not until it is ready (Docker docs).

The fix is well known: a healthcheck on Postgres, a one-shot migration service, and condition: service_completed_successfully on the app. What the popular guides skip is what happens when the gate itself misfires. We ran this on real infrastructure and it misfired on the first try. This post shows the file that survived.

db (postgres:16-alpine) stage 1 │ healthcheck: pg_isready over TCP ▼ condition: service_healthy migrate (one-shot job, exits 0) stage 2 │ waits for DNS + port, then runs idempotent SQL ▼ condition: service_completed_successfully api (PostgREST, serves /todos) stage 3

The API is PostgREST, which turns a Postgres schema into a REST API. It's a good test subject because it reads the schema at boot. If the migration hasn't run, there is nothing to serve.

Hubfly's composer turned these 3 services into 3 deploy stages. It marked migrate as a Job with restart policy Never, because of restart: "no".

db: image: postgres:16-alpine environment: POSTGRES_DB: app POSTGRES_USER: app POSTGRES_PASSWORD: lab-only-change-me # replace before you deploy volumes:

• pgdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"] interval: 5s timeout: 3s retries: 10

Note the -h 127.0.0.1. Without it, pg_isready checks the Unix socket.

On first boot, the official image starts a temporary server to create your database. In our logs, that temporary server listened only on the Unix socket. The real server came up afterwards with listening on IPv4 address "0.0.0.0", port 5432.

A socket check could go green during that setup window, so we switched to TCP as a precaution. A TCP check only passes once other containers can actually connect.

migrate: image: postgres:16-alpine restart: "no" environment: PGHOST: "{{service:db.host}}" PGUSER: app PGPASSWORD: lab-only-change-me PGDATABASE: app command:

• sh
• -c
• | timeout 60 sh -c 'until pg_isready -t 2; do echo "waiting for db"; sleep 2; done' || exit 1 psql -v ON_ERROR_STOP=1
  -c "CREATE TABLE IF NOT EXISTS todos (id serial PRIMARY KEY, task text NOT NULL, done boolean NOT NULL DEFAULT false)"
  -c "DO 'BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = ''web_anon'') THEN CREATE ROLE web_anon NOLOGIN; END IF; END'"
  -c "GRANT USAGE ON SCHEMA public TO web_anon"
  -c "GRANT SELECT ON todos TO web_anon"
  -c "INSERT INTO todos (task) SELECT 'ship the migration gate' WHERE NOT EXISTS (SELECT 1 FROM todos)" depends_on: db: condition: service_healthy

Three details matter here.

{{service:db.host}} is Hubfly's service reference. At deploy time it became the database's private hostname, db-1-7befe5b4.eucentral.hubfly.internal.

On plain Docker Compose (we didn't test this part), replace {{service:db.host}} with the service name db in both places: PGHOST: db and PGRST_DB_URI: postgres://app:…@db:5432/app.

The DO '...' block uses ordinary quotes instead of $$. That's deliberate: Compose treats $ as variable interpolation, and quoted SQL dodges the whole escaping question.

api: image: postgrest/postgrest:v12.2.3 environment: PGRST_DB_URI: "postgres://app:lab-only-change-me@{{service:db.host}}:5432/app" PGRST_DB_SCHEMAS: public PGRST_DB_ANON_ROLE: web_anon PGRST_SERVER_PORT: "3000" ports:

• "3000:3000" depends_on: migrate: condition: service_completed_successfully

volumes: pgdata:

service_completed_successfully means exit code 0, nothing else. If the job exits 1, the API is never created. That's the whole point: a bad migration stops the release instead of shipping it.

The anonymous role gets SELECT only. We checked: a POST /todos from the browser returned 401 with permission denied for table todos.

For production, have PostgREST log in as a dedicated low-privilege authenticator role instead of the app owner. We kept one user here to keep the file short.

Here's the whole file, exactly as we deployed it:

services: db: image: postgres:16-alpine environment: POSTGRES_DB: app POSTGRES_USER: app POSTGRES_PASSWORD: lab-only-change-me volumes:

• pgdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"] interval: 5s timeout: 3s retries: 10

migrate: image: postgres:16-alpine restart: "no" environment: PGHOST: "{{service:db.host}}" PGUSER: app PGPASSWORD: lab-only-change-me PGDATABASE: app command:

• sh
• -c
• | timeout 60 sh -c 'until pg_isready -t 2; do echo "waiting for db"; sleep 2; done' || exit 1 psql -v ON_ERROR_STOP=1
  -c "CREATE TABLE IF NOT EXISTS todos (id serial PRIMARY KEY, task text NOT NULL, done boolean NOT NULL DEFAULT false)"
  -c "DO 'BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = ''web_anon'') THEN CREATE ROLE web_anon NOLOGIN; END IF; END'"
  -c "GRANT USAGE ON SCHEMA public TO web_anon"
  -c "GRANT SELECT ON todos TO web_anon"
  -c "INSERT INTO todos (task) SELECT 'ship the migration gate' WHERE NOT EXISTS (SELECT 1 FROM todos)" depends_on: db: condition: service_healthy

api: image: postgrest/postgrest:v12.2.3 environment: PGRST_DB_URI: "postgres://app:lab-only-change-me@{{service:db.host}}:5432/app" PGRST_DB_SCHEMAS: public PGRST_DB_ANON_ROLE: web_anon PGRST_SERVER_PORT: "3000" ports:

• "3000:3000" depends_on: migrate: condition: service_completed_successfully

volumes: pgdata:

Deploy it on Hubfly Space

Create a project, open Create → Compose Stack, choose the YAML tab, paste the complete file from the end of Step 3 and click Import YAML.

Check the canvas: 3 services, the pgdata volume, and 2 DEPENDS arrows. The composer's monthly estimate for this stack on the shared tier was $0.01.

Click Deploy Stack and watch the activity log.

On the api container, open Network → Add Domain, pick port 3000, and create the subdomain.

Open https://<your-subdomain>.eu1.hubfly.app/todos.

We got this back:

[{"id":1,"task":"ship the migration gate","done":false}]

A second GET /todos from the browser took 271 ms end to end.

One surprise: ports: "3000:3000" also created a public TCP endpoint on tcp.eu1.hubfly.app by itself. That's fine for an API. Never add ports: to db.

Hubfly marked db healthy about 2 seconds after it deployed, in both runs. So don't rely on the healthcheck alone. The wait loop is what made the second run work. Here is the second run's activity log, stage by stage:

Our first file didn't have the wait loop or the TCP healthcheck. Here's what happened.

db went healthy at 06:28:24. migrate deployed at 06:28:36 and failed 2 seconds later. The API was never created, so the gate did its job. But why did a migration fail against a healthy database?

The job's container was rolled back, so its logs were gone. We can't prove the cause, but 2 clues from the second run point to it.

The migration job's own first line, with the wait loop in place:

db-1-7befe5b4.eucentral.hubfly.internal:5432 - no response waiting for db db-1-7befe5b4.eucentral.hubfly.internal:5432 - accepting connections CREATE TABLE ...

And PostgREST's first 2 seconds of life (container logs are in UTC; the deploy table is CAT, UTC+2):

04:33:10 Failed to query the PostgreSQL version ... could not translate host name "db-1-7befe5b4.eucentral.hubfly.internal" to address: Name does not resolve 04:33:12 Successfully connected to PostgreSQL 16.15 04:33:14 Schema cache loaded 1 Relations

In the second run, a freshly started container could not resolve the database's private hostname for about 2 seconds. PostgREST shrugs that off because it retries. A one-shot psql doesn't retry. It fails, exits non-zero, and takes your deploy down with it.

That's not a Hubfly quirk you can ignore elsewhere. On any orchestrator, "the dependency is healthy" and "my container can reach it right now" are two different facts. A job that runs once must check the second one itself.

1. The job fails in 2 seconds and you can't see why.

A failed job may be rolled back with its logs. Add the wait loop from Step 2, and make the job print what it's doing (echo "waiting for db"). If it still fails, the next run's logs will show whether it's DNS, the port or the SQL.

pg_isready without -h checks the Unix socket, and the first-boot setup server listens only there. Use pg_isready -h 127.0.0.1 -U app -d app, and still keep the wait loop in the job: in our runs, db was marked healthy about 2 seconds after deploy either way.

3. A failed rollout leaves an old database behind.

After our failed first deploy, the rollback of db itself failed (Rollback failed for 'db'). The old db was left behind, asleep and still attached to the volume. Our second deploy then created db-1 next to it, on the same pgdata volume.

Two Postgres servers on one data directory is how you lose data. Before you redeploy after a failure, open the project, and stop or remove any leftover database container.

The job runs on every deploy. CREATE TABLE without IF NOT EXISTS will fail on deploy number 2 and block every release after it. Guard every statement, or use a migration tool that records what it applied (Prisma's migrate deploy, Flyway, golang-migrate, Alembic).

service_completed_successfully is a good gate. It only works if the job behind it is honest: it waits for the database itself, stops on the first error, and is safe to run again. Put those 3 properties in your migration job and a broken schema stops the release instead of reaching users.

Paste the Compose file into a new Hubfly project (Create → Compose Stack → YAML) and click Deploy Stack. The shared tier needs no card. Change lab-only-change-me first.

The composer estimated $0.01/month for this stack on the shared tier. That matches a 1 GB volume: 1 GB × $0.00001536 per GB-hour × 730 h = $0.011 (the free tier includes 1 GB of storage).

Shared containers auto-sleep, and their compute is free within the free tier's 360 hours a month. Two containers kept awake all month would use 1,460 hours.

If you keep db and api awake using a dedicated container (0.5 vCPU, 256 MB):

The migration job runs for a couple of seconds per deploy, which rounds to $0.00.