A Go server that is wire-compatible with Hugging Face's real Xet storage protocol - the content-defined-chunking, dedup-first storage layer that replaces Git LFS for large model files on the Hub. Point the real hf CLI (huggingface_hub + hf_xet) at this server and it works: hf upload, hf download, byte-identical round-trips, real BLAKE3 hashing, real xorb/shard binary formats, real LZ4/byte-grouping compression.

This started as a simplified, non-wire-compatible chunking/dedup demo (kept in internal/chunk, internal/manifest, internal/api, cmd/xet for quick manual testing) and grew into a from-scratch, spec-driven reimplementation of the actual protocol - verified end-to-end against a live hf_xet client, with several real wire-format quirks discovered only by capturing and replaying genuine client traffic. See docs/PROTOCOL.md for that story.

Also included: xet-proxyd, a caching pull-through proxy for the real huggingface.co - not just a performance cache, but an offline-resilience layer. Point it at the real Hub, use it like a normal HF_ENDPOINT, and every repo/file it has successfully served once stays servable via hf download/hf upload even after huggingface.co becomes unreachable - including handing its cache directory to a plain xetd as a permanent, disconnected replacement. See Caching pull-through proxy below.

Features

Architecture

Installation

Usage

Run the wire-compatible server

Authentication

Point the real hf CLI at it

Xet Data API + CLI

Caching pull-through proxy (xet-proxyd)

HTTP API

Interactive API docs (Swagger UI)

Storage backends

Testing

Documentation

Where this diverges from real Xet

Troubleshooting

License

Contributing

Acknowledgments & Related Projects

Features

Wire-compatible CAS HTTP API (internal/casserver): xorb upload/fetch, shard upload, file reconstruction with Range-based paging, matching xet-core's own openapi/cas.openapi.yaml - verified byte-identical against a real hf_xet client, for both compressible and incompressible content.

Hub API shim (internal/hubserver): enough of huggingface.co's Hub REST API (repo create, preupload, xet-{read,write}-token, commit, resolve/HEAD) that the real hf upload/hf download shell commands work against this server via HF_ENDPOINT - including real, independent revisions/branches per repo, not just an implicit main.

xet-proxyd: a caching pull-through proxy for the real huggingface.co (cmd/xet-proxyd, internal/proxycas, internal/proxyhub, internal/hfclient) - an offline-resilience layer, not just a performance cache: any repo metadata or xorb bytes it has successfully relayed once stay servable even after the real huggingface.co becomes unreachable, and its cache directory is a real xetd data directory a plain xetd can take over directly, no proxy process required. See Caching pull-through proxy.

Real BLAKE3-keyed Merkle hashing (internal/merklehash): a byte-for-byte port of xet-core's DataHash, verified against xet-core's own published reference vectors - not an approximation.

Real xorb and shard binary formats (internal/xorbformat, internal/shardformat), each verified against real bytes captured from a live hf_xet upload, including the footer-less upload behavior real clients actually use (see docs/PROTOCOL.md).

From-scratch LZ4 decoder and ByteGrouping4 codec (internal/lz4, internal/bg4): written directly from the public LZ4 spec / verified against a third-party reference implementation, so chunk hashes can be independently re-verified regardless of compression scheme.

Pluggable, streaming storage (internal/storage): a Store interface with filesystem (fsstore) and S3-compatible (s3store) backends, built on io.Reader/io.ReadCloser rather than []byte - neither backend ever buffers a full object in memory, so a multi-GB upload/download costs a fixed amount of memory. The S3 backend uses a from-scratch AWS SigV4 signer (internal/sigv4), no AWS SDK dependency. Verified end-to-end with real GGUF model files up to 27.6 GB.

Zero required external dependencies to build the demo path; the protocol path adds exactly one pure-Go module (github.com/zeebo/blake3), pinned in go.sum.

Optional storage auto-pruning and request rate limiting (internal/eviction, internal/ratelimit): bounded, observable defense-in-depth for a server left running against untrusted traffic - neither adds a dependency, and both are off unless explicitly enabled. xetd rate-limits uploads only (the expensive local operation there); xet-proxyd rate-limits every route on both ports, since even a read can trigger a real outbound call to the real huggingface.co.

Optional dedup-hit content verification (internal/storage's VerifyingStore): byte-compares an incoming upload against the stored blob on a dedup hit instead of trusting the content hash alone, bailing at the first mismatch - a hash-collision/corruption safety net, off by default since it roughly doubles I/O on a dedup hit.

A real global chunk-dedup index and V2 (multi-range) reconstruction (internal/casserver): GET /v1/chunks/{prefix}/{hash} returns the actual shard bytes referencing a queried chunk (the real wire contract, confirmed against xet-core's own client source), and GET /v2/reconstructions/{file_id} returns the multi-range-optimized response shape instead of always falling back to V1.

Restart-surviving metadata persistence: casserver/hubserver (and, identically, xet-proxyd's embedded copies of each) periodically (and on graceful shutdown) checkpoint their in-memory reconstruction/repo indices to disk as an atomic JSON snapshot - see -snapshot-interval below.

Real-client regression fixtures: several packages carry testdata/ captured directly from a live hf_xet session, replayed in unit tests - the strongest guard against silently regressing wire compatibility.

Architecture

See docs/ARCHITECTURE.md for the full system diagram (upload/download sequence diagrams, package responsibility table, and the key design decisions). Short version:

Git LFS

With -hub-addr set, xetd is also a Git LFS server: set a repo's lfs.url to http://<hub>/{owner}/{name}.git/info/lfs and the stock git-lfs client plus git-xet push LFS objects into the Xet CAS with chunk-level dedup, pull them back over the basic transfer, and use git lfs lock. With -sync-folder the data directory can be a Dropbox/Syncthing folder shared by one xetd per machine, no server anywhere. See docs/GIT_LFS.md.

Go 1.27 or later

GNU Make

Bash (for integration tests)

(Optional, for the real hf CLI round-trip test) pipenv, to install huggingface_hub + hf_xet

(Optional) merman-cli or mmdc (mermaid-cli), to render Mermaid diagrams as SVG in make docs/make docs-serve - falls back to a source + mermaid.live link if neither is available

If pipenv can't reach pypi.org directly (a corporate proxy, an air-gapped environment), copy .env.example to .env and set PIPENV_PYPI_MIRROR to a reachable index mirror; .env is read by make install and is gitignored.

make pre-check # verify Go/bash are present make build # -> bin/xetd, bin/xet-proxyd, bin/xet

Install

Via go install

Requires Go 1.27 or later. Installs into $(go env GOPATH)/bin (typically ~/go/bin; add that directory to your PATH if it isn't already):

go install github.com/guilt/xet-server/cmd/xetd@latest go install github.com/guilt/xet-server/cmd/xet-proxyd@latest go install github.com/guilt/xet-server/cmd/xet@latest

Prebuilt binaries

Every tagged release publishes per-platform zips (xet-server-<version>-<platform>.zip) from the GitHub Releases page - each zip bundles all three binaries (xetd, xet-proxyd, xet) plus this README, the CHANGELOG, the LICENSE, CONTRIBUTING, the docs/ directory (both the Markdown sources and the browsable HTML generated by make docs, Mermaid diagrams included), and a VERSION file. A checksums.txt (SHA-256 of every zip) is published alongside them.

Run the wire-compatible server

CAS server only, on :8420

./bin/xetd -addr :8420 -data ./xet-data

CAS server + Hub API shim (needed for the real hf CLI), on :8420 / :8421

./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-data

With storage auto-pruning (evict least-recently-used xorbs over 10GB,

checked every 5 minutes) and per-source-IP upload rate limiting (5

requests/sec sustained, burst of 20) - both optional, both off by default:

./bin/xetd -addr :8420 -data ./xet-data
-max-storage-bytes 10737418240 -eviction-interval 5m
-rate-limit-rps 5 -rate-limit-burst 20

With dedup-hit content verification (see "Storage backends" below) and

a faster persistence checkpoint interval than the 1-minute default:

./bin/xetd -addr :8420 -data ./xet-data -verify-dedup -snapshot-interval 15s

Opening either port's root path (http://localhost:8420/ or, if -hub-addr is set, http://localhost:8421/) in a browser shows a landing page listing every endpoint that port serves - useful for orienting yourself without needing to read this README first.

By default xetd enforces no authentication at all - any client can read and write, matching this project's behavior prior to v0.8.0. Passing -auth-token <secret> (or, preferably, setting $XETD_AUTH_TOKEN) requires every request to carry Authorization: Bearer <secret>, split into read/write scope per endpoint (uploads/commits need write; downloads/ reconstructions/resolve need read) the same way real Xet/HF Hub tokens work. -auth-token None (the default) or leaving it unset disables enforcement entirely.

Minted tokens (the short-lived credentials the Hub shim issues to hf_xet and git-xet, and puts on git-lfs download links) are also bound to one repository: the repo type and ID are inside the HMAC-signed payload, so a token for alice/one gets 403 on bob/two's read, write, and token-refresh routes. A repo-scoped write token may create a new repo only inside its own namespace; creating under another owner requires the shared secret. The shared secret itself stays unrestricted (it carries the admin scope).

export XETD_AUTH_TOKEN="a-long-random-secret" ./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-data

./bin/xet push -server http://localhost:8420 -auth-token "a-long-random-secret" ./model.safetensors

or, equivalently, export XET_AUTH_TOKEN (or HF_TOKEN, the same variable

the real hf CLI reads) instead of passing -auth-token:

export XET_AUTH_TOKEN="a-long-random-secret" ./bin/xet push -server http://localhost:8420 ./model.safetensors

Prefer the environment variable over the flag ($XETD_AUTH_TOKEN / $XET_AUTH_TOKEN) whenever the machine is shared or the command runs where its argument list might be logged: a -auth-token flag value is visible to any other local user via ps -ef / /proc/<pid>/cmdline and gets written to shell history, while an environment variable set through a secrets manager, a .env file kept out of history, or read -s is not. An explicit -auth-token flag always takes precedence over the environment variable if both are set, so scripts that need to override a machine-wide/session-wide token can still do so per invocation.

-auth-token/the env vars above configure this project's built-in auth.StaticTokenAuth (server) and auth.BearerCredentialHelper (client)

export HF_ENDPOINT="http://localhost:8421" # the Hub shim's address export HF_TOKEN="anything" # ignored unless xetd was started with -auth-token/$XETD_AUTH_TOKEN - see Authentication above

hf upload myuser/my-model ./model.safetensors model.safetensors hf download myuser/my-model model.safetensors --local-dir ./downloaded cmp ./model.safetensors ./downloaded/model.safetensors # byte-identical

You can also drive hf_xet's low-level XetSession Python API directly against the CAS server's own address (:8420 above) with no Hub API involved at all - useful for isolating whether an issue is in the CAS protocol or the Hub shim.

Want to run this as a real, standing mirror rather than a one-off local test? See docs/MIRRORING.md for a full step-by-step guide (securing it, pointing multiple machines at it, troubleshooting).

A simple, non-wire-compatible gear-hash-CD