Fly.io Sprites: Storage not ready blocked execution and checkpoint access

2026 年 10 月 29 日下午，我的 Fly.io Sprites 实例 dev2 再次离线。直接执行 /bin/echo 返回 503 Storage not ready。健康检查曾报告修复完成，重启请求也曾被接受，但随后命令和 checkpoint 接口仍然失败。

截至这轮记录结束，实例没有恢复。连包含自动 checkpoint 的列表、恢复点详情和创建保护快照都报存储未就绪，因而没有找到可确认的恢复点，也没有执行 restore。

本文发表于 2026 年 10 月 29 日，覆盖当天约 07:29–08:00 UTC，即北京时间 15:29–16:00 的记录。最后一次直接执行响应为 07:47:36 UTC，之后的 checkpoint 检查继续失败。本文不把记录结束后的状态或数据永久丢失当作已证实事实。表内时间均为 UTC。

实例：dev2，Sprite ID sprite-615374a5-571c-41d7-bb2b-80d52d561f1e。

重启接口返回的 Machine ID：85e714a4902138。

Sprites 环境版本：0.0.4；之前已确认使用 SBD。这里是平台环境版本，不是其中运行的应用版本。

Morty 页面显示实例离线、重连失败，消息输入框不可用。直接访问 Sprites 执行接口同样失败，因此不只是网页显示了一个离线状态。

需要交代的前情

当天中午，dev2 曾在严重资源压力下失去响应：多组并行测试进程运行时，内存占用接近配置的 8 GiB 上限。之后 boot ID 改变，表明机器重启；约 05:07 UTC / 北京时间 13:07，应用连接恢复。那一轮记录没有确定是谁触发了重启，也没有确认临时清理脚本成功终止过进程。

本文记录的是之后再次出现的下午故障。 下午未能取得机器内的内存指标或内核日志，不能把这次 Storage not ready 直接归因于中午的测试负载，也不能声称已证实是同一个底层故障。

10 月 29 日 UTC 操作或观察 结果

约 07:29–07:34 页面离线，读取平台状态并测试执行接口 状态曾显示 cold，短暂出现 running 也未对应到可用的命令执行。

07:34:33 直接 HTTP 执行 /bin/echo HTTP 503，Storage not ready，Sprites 版本 0.0.4。

07:37:09 健康检查尝试自动修复 repaired / restarted stopped machine。

07:38:31–07:39:03 请求整机重启 HTTP 504，restart outcome unknown、outcome_unknown: true；没有立刻重复请求。

07:39:51 重启请求后的直接执行验证 HTTP 503，Storage not ready。

07:40:03 再次健康检查 healthy / machine is running；随后的执行探测仍超时。

07:42:00 请求默认环境更新 already_current / skipped，仍为 0.0.4；没有发生升级，也未启用存储迁移。

07:43:47 核对状态后再尝试一次重启 HTTP 202 Accepted，返回 Machine ID。

07:44:41–07:45:47 验证执行和服务列表 执行 55 秒超时，服务列表约 10 秒超时。

07:47:36 再次直接执行 /bin/echo 仍为 HTTP 503，Storage not ready。

07:50:08–07:54:57 普通 checkpoint 列表和 includeAuto=true 列表 均为 HTTP 503，Storage not ready。

07:58:48–07:58:49 查询 Current 和已知旧 checkpoint v1 的详情 两项均为 HTTP 503，Storage not ready。

07:59:11 发起 尝试在恢复前创建保护快照 CLI 报 HTTP 503，Storage not ready；没有成功创建。

可核对的响应

以下摘录保留原始状态、版本、错误正文和 Fly 请求 ID；省略其他响应头。

2026-10-29 07:34:33 UTC — direct /bin/echo HTTP/1.1 503 Service Unavailable sprite-version: 0.0.4 fly-request-id: 01M6398ZHNEXCC0H6EXJK0RGJJ-nrt Storage not ready

2026-10-29 07:47:36 UTC — exec after accepted restart HTTP/1.1 503 Service Unavailable sprite-version: 0.0.4 fly-request-id: 01M63A0WR8CZYQRXE45P0C6VMN-nrt Storage not ready

2026-10-29 07:51:12 UTC — checkpoint list including automatic checkpoints HTTP/1.1 503 Service Unavailable sprite-version: 0.0.4 fly-request-id: 01M63A7FDBPEH8SCERMN8MREZR-nrt Storage not ready

2026-10-29 07:59:11 UTC — checkpoint creation attempt started Error: failed to create checkpoint: API returned status 503: Storage not ready fly-request-id: 01M63AQ35S6R7NBGY1YQA978B0-nrt

为什么没有执行 checkpoint 恢复

此前保存的普通列表里曾出现 10 月 23 日的初始 checkpoint v1，但它不是本轮确认过的最新可恢复点。官方 checkpoint 文档说明，自动 checkpoint 默认隐藏，需要显式包含；restore 会覆盖当前文件系统，并不会自动备份被覆盖的状态。

这轮排查已尝试包含自动 checkpoint 的列表、v1 与 Current 详情，以及恢复前保护快照，全部失败。没有向 restore 接口发起恢复请求，也没有覆盖当前磁盘。 因此，准确表述是恢复准备被存储错误阻断，不是“restore 命令执行后失败”，也不是“已经证明没有快照”。

恢复尝试没有带来可用的环境。healthy、repaired 和 HTTP 202 Accepted 都没有证明命令、文件或 checkpoint 可用。可以确认的阻断层级是 Sprites 返回的存储未就绪；底层原因及数据完整性仍需要 Fly 的存储和宿主机日志确认。

原实例未在这轮排查中被删除或重建。这份记录请求 Fly 保留当前卷，检查存储初始化，并确认是否存在最近健康的自动 checkpoint 或其他平台侧恢复路径。

我公开这段经历，是因为 Sprites 产品页将持久化和 checkpoint 恢复列为核心能力，而这次故障同时阻断了使用环境与查看恢复点。我的实际损失在这份记录中表现为环境不可用、工作中断、无法通过已尝试的自助操作恢复；数据是否永久丢失尚未确定。

On the afternoon of 29 October 2026, my Fly.io Sprite dev2 went offline again. Direct execution of /bin/echo returned 503 Storage not ready. A health check reported a repair, and a restart request was later accepted, but execution and checkpoint endpoints continued to fail.

The environment had not recovered by the end of this recorded investigation. Listing checkpoints, including automatic ones, reading checkpoint details and creating a protective checkpoint all failed. No recovery point could be verified, and no restore request was issued.

Published on 29 October 2026, this account covers approximately 07:29–08:00 UTC that day, or 15:29–16:00 in Asia/Shanghai. The last direct execution response was at 07:47:36 UTC; subsequent checkpoint checks also failed. This account does not establish the state after that window or permanent data loss. All timeline times below are UTC.

Sprite: dev2, ID sprite-615374a5-571c-41d7-bb2b-80d52d561f1e.

Machine ID returned by the restart endpoint: 85e714a4902138.

Sprites environment version: 0.0.4; prior inspection had established SBD storage. This is the platform environment version, not an application version.

Morty showed the instance offline and unable to reconnect, with its message input disabled. Direct Sprites API execution also failed, so this was more than an offline indicator in the application UI.

Relevant earlier context

Around midday, dev2 had become unresponsive under severe resource pressure. Multiple concurrent test processes were running, and memory consumption approached its configured 8 GiB limit. The boot ID subsequently changed, establishing that the machine rebooted; application connections recovered at approximately 05:07 UTC / 13:07 Asia/Shanghai. That investigation did not establish what triggered the reboot or confirm that a temporary cleanup script successfully terminated any process.

This account concerns the later afternoon recurrence. Guest memory metrics and kernel logs could not be obtained during the afternoon incident. I cannot attribute its Storage not ready responses to the earlier workload or establish that both events had the same underlying cause.

29 October UTC Action or observation Result

Approximately 07:29–07:34 Application offline; metadata and execution probes Metadata showed cold; a brief running state did not establish working execution.

07:34:33 Direct HTTP execution of /bin/echo HTTP 503, Storage not ready, Sprites version 0.0.4.

07:37:09 Health check with automatic repair repaired / restarted stopped machine.

07:38:31–07:39:03 Request machine restart HTTP 504, restart outcome unknown, outcome_unknown: true. It was not immediately retried.

07:39:51 Direct execution after restart request HTTP 503, Storage not ready.

07:40:03 Another health check healthy / machine is running; a subsequent execution probe still timed out.

07:42:00 Request default environment update already_current / skipped, remaining on 0.0.4; no upgrade or storage migration occurred.

07:43:47 One further restart after checking status HTTP 202 Accepted and a Machine ID.

07:44:41–07:45:47 Check execution and service listing Execution timed out after 55 seconds; service listing after about 10 seconds.

07:47:36 Another direct /bin/echo Still HTTP 503, Storage not ready.

07:50:08–07:54:57 List checkpoints normally and with includeAuto=true All returned HTTP 503, Storage not ready.

07:58:48–07:58:49 Read Current and previously known checkpoint v1 details Both returned HTTP 503, Storage not ready.

Started 07:59:11 Attempt a protective checkpoint before recovery CLI reported HTTP 503, Storage not ready; no checkpoint was created successfully.

The original response excerpts appear above. The attached evidence.json and evidence.txt contain selected responses and request IDs.

A previously saved ordinary list contained an initial checkpoint v1 from 23 October. That did not establish the latest usable recovery point. The checkpoint documentation says automatic checkpoints are hidden unless explicitly included, and restoring overwrites the current filesystem without backing up the replaced state.

This investigation attempted the list including automatic checkpoints, details for v1 and Current, and a protective checkpoint. All failed. No restore request was issued, and the current disk was not overwritten. Recovery preparation was blocked by storage errors; I am not claiming that a restore operation ran and failed, or that no checkpoints existed.

The attempted recovery operations did not yield a usable environment. healthy, repaired and HTTP 202 Accepted did not establish working execution, filesystem or checkpoint access. The confirmed visible failure was the platform's storage readiness error; its underlying cause and the integrity of the data require Fly's storage and host logs.

The original Sprite was not deleted or rebuilt during this investigation. I am asking Fly to preserve the current volume, inspect storage initialization, and determine whether a recent healthy automatic checkpoint or another platform-side recovery route exists.

I am publishing this because persistence and checkpoint recovery are central capabilities on the Sprites product page. In this incident, both use of the environment and access to recovery points were blocked. The documented impact is an unavailable working environment, interrupted work and unsuccessful self-service recovery attempts. Permanent data loss has not been established.

附件保留的是我在排查时采集的记录摘选。原始响应与本地诊断摘要分别标注；账号信息、本地路径和无关内容已省略，保留的错误正文、请求 ID 和维护事件未改写。这些是用户采集的证据，不是 Fly 官方事故报告，可供 Fly 按请求 ID 核查后台日志。

evidence.json contains selected fields and responses from my locally captured records; evidence.txt presents the same material for reading. Account context, local paths and unrelated logs are omitted. Retained errors, request IDs and maintenance events are not rewritten. Earlier workload context and UI observations are explicitly labeled as excerpts from local diagnostic notes, distinct from raw HTTP responses.

HTTP Date headers, service event times and client timestamps are distinguished in the attachments. A CLI/curl exit code of zero in a source record does not establish HTTP success. These are user-collected records, not a Fly-authored postmortem; request IDs allow Fly to correlate the observations with server logs.