返回资讯中心
外部精选
软件工程
#开发者实时流

PIZD, a Rust compressor exploring structural deduplication

A compressor built around the repetition in logs, JSON and tabular data. Reuse record structure, pack recurring values, and restore every original byte.

Hacker News 最新torxx6662026/10/26 · 20 天前8 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

A compressor built around the repetition in logs, JSON and tabular data. Reuse record structure, pack recurring values, and restore every original byte.

Native Rust and optional Zstandard-backed engines. Self-contained archives with --standalone.

30 public excerpts · density settings · all sources included

Measured comparison available on the benchmarks page.

Size, encoding time and verified decoding are reported together.

View the full comparison and methodology →

Repeated field names, timestamps and related values are useful signals. PIZD separates reusable structure from changing values before selecting a lossless representation.

Structural projections and column models capture repeated layouts and value coherence in logs and structured records.

Use default, fast or thin profiles. Native encoding keeps a Rust pipeline; Hybrid uses Zstandard as a backend.

Standalone archives carry what they need. Verification checks the restored contents; dictionary sharing is optional.

A compressor should earn its place on your workload. Start with measured examples, then compare size and throughput on your own files.

Examples are labeled by source and preset. They are illustrations, not a substitute for the complete panel.

No dictionary service is required for this workflow. Keep the original file, compress it, then restore and compare.

Compress without a dictionary service

pizd compress access.log --standalone --offline --thin -o access.pizd

Restore and verify the archive checksum

pizd decompress access.pizd --offline -o restored.log

Independently compare the original and restored files

cmp access.log restored.log

Copy commands

Get a build you can trace.

Use the downloads page for release availability, build instructions and provenance verification.

Start with default. Use --fast to favor speed or --thin to spend more effort on size.

pizd bench input.log compares PIZD with other codecs. Check decoding time as well as archive size.

The latest benchmark uses the same public excerpts for every codec. Raw files, source identities, presets and limitations remain available without filling the front page with development logs.

Default, fast and thin. Size, encoding time and verified decoding.

Deposited on Zenodo; not submitted to arXiv.

Source provenance, original CSVs, independent checks and historical investigations.

The interactive public comparison is the frozen 6.0.0-rc1 run; its version is retained. The separate 18-input generated panel measures 6.0.0-rc9. Neither is a fresh comparison of the isolated rc9 candidate.

The rc9 density candidate reduces public30 Native thin/template archives from 687,537 to 657,356 bytes (4.39%), but verified in-process decoding takes 5.778 times longer in its paired audit. This is a separate candidate pending integration review, not a universal speed gain. Familiar inputs and same-family later windows do not establish independent generalization.

The engineering contribution combines exact-layout models, exception-aware field dependencies and complete-cost representation choices. These build on established compression methods. No matched comparison with specialized log compressors or new entropy-coding theorem is claimed.

Publication claim audit → · Candidate sizes, CPU costs and source identities →

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →