返回资讯中心
外部精选
软件工程
#开发者实时流

IETF 草案提出 KVCache 高效存储索引机制

来源标题:Draft-li-kvcache-efficient-storage-00

一份 IETF 草案定义了 KVCache 的高效存储与索引方案:holder 通过 PUT 向索引服务发布副本记录,记录在 ttl_ms 后过期,需在淘汰副本前用 DELETE 撤回。索引仅作参考,客户端将 holder 返回的 404 视为缓存未命中并继续查找下一副本。

Hacker News 最新simonpure2026/11/4 · 13 天前26 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

{holder-id} and {instance-id} are the base64url encoding without padding [RFC4648] of the SHA-256 of the holder's or instance's base URI, which gives each a fixed-length path-safe identifier.¶

A holder publishes each replica it is willing to serve with PUT, carrying a Replica Record. The Index Service MUST verify that the authenticated client is the holder named in the record or is authorized to publish on its behalf, and that the client is authorized for the record's scope. A replica record expires after ttl_ms; the IS caps ttl_ms at a configured maximum. Holders refresh records before expiry with the same PUT, which is idempotent. A holder MUST withdraw a record with DELETE before or immediately after evicting the replica, and MUST NOT evict a replica covered by an unexpired lease (Section 6.6).¶

The index is advisory: a holder can evict an unleased replica before the withdrawal reaches the IS. Clients therefore treat a 404 from a holder as a cache miss and continue with the next replica or with recomputation.¶

A client (normally the PC) computes the Block Keys of the prompt and sends them, in prefix order, to the IS:¶

POST {is-base}lookup { "scope_id": "<hex16>", "compat_id": "<hex32>", "block_keys": ["<hex32>", "<hex32>", "<hex32>"], "targets": ["https://ii7.example/kvsp/", "https://ii9.example/kvsp/"] }

The IS returns "matched", the number of leading keys for which at least one unexpired replica exists, and for each matched key the replica records. Because Block Keys are chained, the IS MUST stop at the first key without a replica. When "targets" is present and the IS has the needed status and link information, it also returns, for each target instance, "fetch_us" (the estimated time to make the matched blocks resident at that instance, Section 6.4) and "recompute_us" (the estimated time to prefill the same tokens on that instance). The IS MUST NOT return replicas of a scope for which the client is not authorized.¶

Each inference instance sends its Instance Status Record to the PC at an interval no longer than one third of its ttl_ms. The PC considers an instance failed when its record expires. The "epoch" of an instance MUST increase every time it restarts; the start time of the instance in milliseconds since the Unix epoch satisfies this without persistent state, provided that the instance's clock does not move backward across restarts; otherwise the instance persists a counter. The PC MUST discard records with an epoch lower than the highest it has seen for that instance.¶

Instances move between the states active, draining, sleeping, and failed. An instance whose load has stayed below a configured threshold for a configured time MAY ask to sleep by setting "requested_state" to "sleeping". The PC answers the status PUT with 200 and a body {"state": "draining"} to accept, or {"state": "active"} to refuse. The PC can also change the state of an instance with PUT state on the instance.¶

The PC chooses an instance for each request. The policy is local, but it is expected to compare, for each candidate, the cost of loading cached blocks with the cost of recomputing them. The following estimates SHOULD be used, by the IS when it answers a lookup with targets and by the PC otherwise:¶

fetch_us(i) = max over sources s of ( rtt_us(i,s)

  • 8 * bytes_from(s) / mbit_per_s(i,s) ) for blocks not already resident at i recompute_us(i) = 1e6 * tokens / prefill_tokens_per_s(i) wait_us(i) = queued(i) * mean service time at i cost(i) = wait_us(i)
  • min( fetch_us(i) + recompute_us_of_unmatched(i), recompute_us_of_all(i) )

where bytes_from(s) is the volume the instance would load from source s, and mbit_per_s(i,s) is the link rate reported by instance i for source s, reduced, for layers that the source holds in a storage tier, to the "read_mbit_per_s" of that tier in the Replica Record. The PC SHOULD assign the request to an active instance with the lowest cost. When no active instance has acceptable cost or capacity, the PC MAY start a new instance and SHOULD send it a prefetch directive at once, so that cache loading overlaps the instance start.¶

To load caches into an instance before a request reaches it, the PC sends a prefetch directive:¶

POST {ii-base}prefetch { "directive": "a2b7c9e0-1f3d-4c1a-8f5b-6d2e9a7c3b10", "request_ref": "opaque-value-from-PC", "scope_id": "<hex16>", "compat_id": "<hex32>", "blocks": [ { "block_key": "<hex32>", "sources": ["https://h1.example/kvsp/"] } ], "layer_order": "ascending", "deadline_ms": 50, "lease_ms": 2000 }

The instance answers 202 (Accepted) with a Location field naming the directive's status resource, and processes the directive while it continues to execute current work; prefetch MUST NOT block execution of requests already admitted. For each block it does not already hold, the instance acquires a lease of lease_ms at one of the listed sources (Section 6.6), so that the source cannot evict the block while it is being loaded, retrieves the block (Section 6.7), verifies the digest, places it in its tiers, and publishes its new replica (Section 6.1). The state of the directive ("pending", "partial", "complete", or "failed", with the list of loaded keys) can be read with GET. If the deadline expires, the instance stops loading and reports "partial"; the request is then served with the loaded prefix and the remaining tokens are recomputed.¶

"request_ref" is an opaque value that lets the PC associate the directive with the inference request it will route; how the request carries the value is outside the scope of this document.¶

POST leases on a holder, with a body listing block keys and "ttl_ms", creates a lease and returns its identifier and the granted TTL. PUT on the lease renews it; DELETE releases it. While a lease is valid the holder MUST NOT evict the leased replicas. A holder MUST enforce a per-client quota on leased octets and lease TTL and returns the problem type "kvsp-lease-quota" when a request exceeds it. A holder that must evict leased data due to a fault MUST first withdraw the affected replica records.¶

Block resources are named blocks/{compat}/{scope}/{block_key}, where {compat} is the Compatibility ID (hex32), {scope} the Scope ID (hex16), and {block_key} the Block Key (hex32), so that retrieval and storage address the same resource. GET on it returns the container (Section 4.3) with media type "application/kvsp-block". The query parameter "layers=a-b" requests the inclusive layer range a..b; a holder that does not hold those layers returns 404 with problem type "kvsp-layers-unavailable". The query parameter "codec=n" requests a codec; a holder MAY transcode. Holders MUST include a "Repr-Digest" field [RFC9530] with the SHA-256 of the returned container. Block representations are immutable but confidential; holders SHOULD send "Cache-Control: private, immutable" with a max-age no longer than the remaining TTL of the replica.¶

Clients MUST verify Repr-Digest. When the returned codec equals the replica's codec, clients MUST also verify the layer digests of the returned layers against the Replica Record, and, when all layers were retrieved, the replica digest. Transcoded representations are protected only by Repr-Digest, so clients that need producer-level integrity request the replica's codec.¶

PUT on the block resource with "If-None-Match: *" stores a block at a holder. The holder MUST reject the request if the client is not authorized for the scope. The key identifies the token prefix, not the stored octets, so a holder that already has a replica for the key returns 412 and keeps its copy.¶

An instance that serves a session SHOULD publish a Session Checkpoint to the PC with PUT sessions/{session}/checkpoint whenever a block of generated tokens completes, and at least every 64 output tokens. It also publishes replicas for the generated blocks, so that their KV cache survives the instance. The PC records the checkpoint with the highest output_offset for the current session epoch and MUST reject a checkpoint with a lower epoch with problem type "kvsp-stale-epoch".¶

Output continuity depends on the sampler. If the checkpoint states that the sampler is deterministic, a resumed instance regenerates exactly the tokens the failed instance produced after the last checkpoint, and the front end discards the duplicates by index. Otherwise the instance MUST NOT release an output token to the front end before the PC has acknowledged a checkpoint that covers it; it can checkpoint several tokens at a time to bound the added latency. Every output token carries its index and the session epoch, and front ends MUST discard tokens that carry a lower epoch than one they have already seen for the session.¶

When the instance fails, the PC selects a new instance as in Section 6.4, increments the session epoch, and sends POST sessions/{session}/resume to it with the latest checkpoint and the new epoch. The new instance:¶

Any message from the failed instance carrying the old epoch MUST be rejected. Under these rules the output already delivered is never repeated or altered, and the client receives a single consistent sample.¶

For long contexts, a parallel group can split one context by token range across several instances. The PC describes the split in a Shard Map:¶

{ "map": "c3f8...e1", "version": 4, "shards": [ { "tokens": [0, 32768], "instance": "https://ii1.example/kvsp/" }, { "tokens": [32768, 65536], "instance": "https://ii2.example/kvsp/" }, { "tokens": [65536, 98304], "instance": "https://ii3.example/kvsp/" } ] }

Token ranges are half-open, contiguous, and non-overlapping. The computing demand of a context changes between phases: prefill is compute-bound and benefits from many instances, while early decoding steps need fewer instances and later ones need more memory. The PC can therefore change the group between iterations. To move from version v to v+1, it sends each instance in either version a reshard plan with POST reshard:¶

{ "map": "c3f8...e1", "from": 4, "to": 5, "deadline_ms": 200, "retain": [[0, 32768]], "fetch": [{ "tokens": [32768, 49152], "from": "https://ii2.example/kvsp/" }], "release": [] }

The following rules apply:¶

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →