
开发者用 Rust 构建 o5s,尝试统一 GitHub、Docker Hub、JFrog 与 Hugging Face
来源标题:Can We Unify GitHub, Docker Hub, JFrog, and Hugging Face?
开发者 Hideaki Takahashi 推出 o5s,一个基于 Rust 和 h5i 框架构建的通用数据仓库,试图用单一基础设施替代 GitHub、Docker Hub、JFrog 和 Hugging Face 等平台。o5s 中的对象不可变,以 BLAKE3 内容哈希标识,名称则是通过 compare-and-swap 更新的可变引用,以防止写入冲突。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Storage
Github
Hugging Face
Docker
Can We Unify GitHub, Docker Hub, JFrog, and Hugging Face?
Hideaki Takahashi
7 min read·
Just now
--
Introducing o5s: One universal repository for any kind of data, built in Rust with h5i framework.
Each comes with its own infrastructure, APIs, authentication, permissions, versioning, and release workflows.
At the end of the day, they are all just sequences of 0s and 1s. While different formats have distinct workflows, those differences don’t justify maintaining separate repository infrastructures. That’s why we’re building o5s, the universal repository for any kind of data.
Let’s forget Git repositories, container registries, package indexes, and model hubs for a moment. What does a repository actually need to do?
I think the answer can be surprisingly simple:
That’s it. In o5s, objects are immutable and identified by their BLAKE3 content hashes. Names are mutable references to those objects, updated using compare-and-swap (CAS) to prevent conflicting writes.
o5s -C /data/repo init
H=$(o5s -C /data/repo put model.safetensors) o5s -C /data/repo replace models/acme/main $H --expect absent o5s -C /data/repo get $H > model.safetensors
The same operations work regardless of data type (source code, container image, Python package, an AI model’s weights, or anything else).
Different data shouldn’t require fundamentally different repository semantics.
Our architecture uses one universal core with domain-specific frontends.
Git Hugging Face Docker/OCI pip/npm/Maven | | | | v v v v +--------------------------------------------------+ | Protocol Fronts | | | | Git HF/Xet OCI Package APIs | | | | Protocols, ID mapping, encoding profiles | +------------------------+-------------------------+ | v +--------------------------------------------------+ | o5s Core | | | | Immutable Objects Names & CAS | | Content Hashing Namespace Policies | | Provenance Authorization | | GC & Retention Signing & Promotion | +------------------------+-------------------------+ | v +--------------------------------------------------+ | Storage Engine | | | | FastCDC / Xet Chunking | | zstd / Byte Grouping / Delta Encoding | | Content-addressed Packs & Indexes | +------------------------+-------------------------+ | +--------+--------+ | | v v Local Disk S3-compatible Storage
Chunking: Why Store a 10 GB Model as One Object?
Suppose you’re uploading a 10 GB AI model. Storing it as a single blob is simple. But what happens when you fine-tune the model and only part of the data changes? Re-uploading and storing the entire 10 GB can be wasteful.
This is where content-defined chunking comes in. Instead of splitting data into fixed-size blocks, o5s identifies chunk boundaries based on the content itself.
model.safetensors | v Content-Defined Chunker | +-------+-------+ | | | v v v Chunk A Chunk B Chunk C | | | v v v Hash A Hash B Hash C \ | / \ | / Manifest Tree
If part of a file changes, unchanged chunks can be reused rather than stored again.
Specifically, o5s supports two chunking approaches:
While the strategy can be selected by namespace, the object’s identity is independent of how it’s chunked. The object ID is always the BLAKE3 hash of its complete original content. This allows us to optimize the storage representation without changing the object’s identity.
Git has spent decades optimizing how related versions of source code are stored. Hugging Face’s Xet tackles deduplication for large files. Compression algorithms like zstd exploit repeated byte patterns.
These techniques were developed or adopted for different workloads, but the underlying ideas are much more general. Why should a good compression technique be restricted to the system that popularized it? Thus, we’re building a shared storage engine that can combine several complementary techniques.
The first trick we can use is similarity-aware compression. Instead of compressing every object independently, o5s can group similar objects into shared compressed frames. The storage engine can use content-similarity features to guide grouping, rather than relying only on filenames or paths.
Second, We also use byte grouping for floating-point data like AI weights, since corresponding bytes in floating-points across adjacent values often have similar distributions. Consider 16-bit values:
Original byte order:
A0 A1 | B0 B1 | C0 C1 | D0 D1 After byte grouping: A0 B0 C0 D0 | A1 B1 C1 D1
Grouping corresponding byte positions into separate planes before applying zstd can improve compression. The transformation is lossless and can be applied to BF16, FP16, FP32, and similar data.
Third method is XOR delta encoding. Suppose you have two versions of the same model. Instead of independently storing both, o5s can represent a new version using a base object and an XOR patch.
Base Model | v +-------------+ | XOR Delta | <--- New Model +-------------+ | v Delta Object
Reconstruction: Base + Delta | v New Model
These techniques are composable: o5s can chunk an object, apply byte grouping or XOR delta encoding, and pack similar chunks into shared zstd frames.
It’s easy to propose a universal storage architecture. The harder question is whether it can compete with systems that have spent years optimizing individual workloads.
So we benchmarked o5s across several protocols using real, unmodified clients (You can find the detailed experiments, methodology, and limitations in our benchmark documentation and compression experiments.).
The following measurements compare o5s with specialized systems on a four-vCPU AMD EPYC machine with 15 GB RAM and NVMe storage.
Ratios are o5s divided by the comparison system. Lower is better. The metrics use different domain-specific workloads and should not be compared across rows.
Some results are encouraging. On the tested Git workloads, push and clone times are competitive with Git daemon. OCI pushes were faster than Distribution, and the package workloads showed substantial storage savings.
While o5s already does not beats every specialized system at everything, the experiment still shows that one shared repository architecture is already competitive across several very different workloads.
We also tested our similarity-aware compression algorithm separately from the full server.
First, we compared it against git gc --aggressive, using four real source repositories. The experimental compressor produced smaller representations on all four: approximately 15.8% less storage on a geometric-mean basis. More interestingly, removing Git-specific path hints changed the results by no more than about 0.4% per repository. In other words, the algorithm wasn’t relying on knowing which files belonged to Git. Then we tested the same similarity-aware approach on other data.
These are separate compression experiments with different baselines, not interchangeable performance comparisons.
Finally, we tested the storage engine independently of its protocol adapters. In a benchmark containing ten million distinct 64-byte objects, o5s achieved:
These are local core measurements, not end-to-end HTTP performance.
There are still performance costs to address. Verified reads require hashing, durable writes are affected by storage throughput, and the index consumes meaningful memory and disk space at scale.
But taken together, the results suggest that sharing storage infrastructure doesn’t inherently mean giving up competitive performance.
We’re not asking developers to abandon Git, Docker, pip, npm, or Hugging Face, and o5s provides compatible interfaces for Git, OCI, Hugging Face (including LFS and Xet), PyPI, npm, and Maven.
Existing clients can communicate with the o5s server using their familiar protocols.
Git
git clone https://your-server/git/team/project.git
Container images
docker push your-server/team/app:1.0
Hugging Face
HF_ENDPOINT=https://your-server hf upload team/model ./model
Python packages
twine upload
--repository-url https://your-server/pypi/team/ dist/*
A Few Obvious Questions
“Why should I use o5s?” → Less infrastructure to manage and no need to change the tools you already use.
“Isn’t this just S3 with adapters?” → S3 stores data. o5s manages how data is versioned, referenced, and released.
“Merge in Git is necessary” → Merge is just a special case of replace.
“Git packs source code better” → Then use it for every domain.
“Hugging Face’s Xet makes large weights fast” → Then use it for every domain.
“Registries need their own rules” → They’re just per-namespace rules (mutable, immutable, head-release).
“One store means one bug breaks everything” → Every decision is formally verified in Lean 4.
“How do I migrate everything?” → Start with one registry and keep your existing tools.
We’re Just Getting Started
o5s is still early, and there’s plenty left to improve. But our goal is simple: one universal repository for any kind of data, without forcing developers to change their existing tools.
Check it out on GitHub. We’d love your feedback!
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。