Show HN: Pure-Rust port of bzip3 compressor. Format-compatible with C

Each block passes through four stages: run-length coding, LZP, a Burrows-Wheeler transform (suffix array), and an order-0 context-mixing coder.

cargo add bzip3-rs

The library name in Rust code is bzip3_rs.

cargo install bzip3-rs --features cli

Debian and Ubuntu

Download the .deb file for your architecture (amd64 or arm64) from the release page. Then install it:

sudo apt install ./bzip3-rs_*.deb

The package replaces the C bzip3 package. It supports Debian 12 and later, and Ubuntu 22.04 and later.

Each example below is a complete program.

use bzip3_rs::{BlockSize, compress, decompress};

fn main() -> Result<(), Box<dyn std::error::Error>> { let data = b"hello world ".repeat(1000);

let coded = compress(&data, BlockSize::DEFAULT)?; let plain = decompress(&coded)?;

assert_eq!(plain, data); Ok(()) }

Parallel blocks

Jobs::SERIAL runs on the calling thread. Jobs::Global uses the global rayon pool. Jobs::from_count(4) builds a private pool. Output bytes are the same for every job count.

use bzip3_rs::{BlockSize, Jobs, compress_with_jobs, decompress_with_jobs};

fn main() -> Result<(), Box<dyn std::error::Error>> { let data = b"hello world ".repeat(1000); let jobs = Jobs::from_count(4);

let coded = compress_with_jobs(&data, BlockSize::DEFAULT, jobs)?; let plain = decompress_with_jobs(&coded, jobs)?;

assert_eq!(plain, data); Ok(()) }

Streams

Module stream reads from a Read value and writes to a Write value. It uses the 9-byte streaming frame.

use std::io::Cursor; use bzip3_rs::{BlockSize, Jobs, stream};

fn main() -> Result<(), Box<dyn std::error::Error>> { let input = b"hello world ".repeat(1000); let size = BlockSize::DEFAULT; let jobs = Jobs::SERIAL;

let mut coded = Vec::new(); stream::compress(Cursor::new(&input), &mut coded, size, jobs)?;

let mut plain = Vec::new(); stream::decompress(Cursor::new(coded.as_slice()), &mut plain, jobs)?; assert_eq!(plain, input);

// Use compress_slice when the input is in memory. let mut coded_again = Vec::new(); stream::compress_slice(&input, &mut coded_again, size, jobs)?; assert_eq!(coded_again, coded); Ok(()) }

Read and Write adapters

use std::io::{Cursor, Read, Write}; use bzip3_rs::BlockSize; use bzip3_rs::read::{Decoder as ReadDecoder, Encoder as ReadEncoder}; use bzip3_rs::write::{Decoder as WriteDecoder, Encoder as WriteEncoder};

fn main() -> Result<(), Box<dyn std::error::Error>> { let input = b"hello world ".repeat(1000); let size = BlockSize::MIN;

// Read adapters. let mut encoder = ReadEncoder::new(Cursor::new(&input), size)?; let mut coded = Vec::new(); encoder.read_to_end(&mut coded)?;

let mut decoder = ReadDecoder::new(Cursor::new(coded.as_slice()))?; let mut plain = Vec::new(); decoder.read_to_end(&mut plain)?; assert_eq!(plain, input);

// Write adapters. Call finish to close the stream and to see errors. let mut encoder = WriteEncoder::new(Vec::new(), size)?; encoder.write_all(&input)?; let coded = encoder.finish()?;

let mut decoder = WriteDecoder::new(Vec::new()); decoder.write_all(&coded)?; let plain = decoder.finish()?; assert_eq!(plain, input); Ok(()) }

write::Encoder also writes the last block when you drop it. A drop ignores errors, so call finish.

Use BlockEncoder and BlockDecoder to build the streaming frame yourself.

use bzip3_rs::{BlockDecoder, BlockEncoder, BlockHeader, BlockSize, StreamHeader};

fn main() -> Result<(), Box<dyn std::error::Error>> { let input = b"hello world ".repeat(1000); let size = BlockSize::MIN;

// Encode: the stream header, then a block header and a payload for each block. let mut coded = StreamHeader::new(size).to_bytes().to_vec(); let mut encoder = BlockEncoder::new(size)?; for chunk in input.chunks(size.as_usize()) { let payload = encoder.encode_block(chunk)?; coded.extend_from_slice(&BlockHeader::for_block(payload.len(), chunk.len())?.to_bytes()); coded.extend_from_slice(&payload); }

// Decode the same bytes. let (head, mut rest) = coded.split_at(StreamHeader::LEN); let header = StreamHeader::from_bytes(head.try_into()?)?; let mut decoder = BlockDecoder::new(header.block_size)?; let mut plain = Vec::new(); while !rest.is_empty() { let (block, tail) = rest.split_at(BlockHeader::LEN); let block = BlockHeader::from_bytes(block.try_into()?); let (payload, tail) = tail.split_at(block.compressed_size as usize); decoder.decode_block_into(payload, block.original_size, &mut plain)?; rest = tail; } assert_eq!(plain, input); Ok(()) }

BlockDecoder::recover_block_into decodes a damaged block. It always appends the original number of bytes. A checksum fault keeps the decoded bytes. Any other fault appends zero bytes. The call returns a BlockStatus value that tells which case happened.

use bzip3_rs::{BlockSize, MAGIC_NUMBER, compress_bound};

fn main() -> Result<(), Box<dyn std::error::Error>> { assert_eq!(MAGIC_NUMBER, *b"BZ3v1"); assert_eq!(compress_bound(1000), 1052);

let _bytes = BlockSize::new(16 * 1024 * 1024)?; let _kib = BlockSize::kib(65)?; let _mib = BlockSize::mib(16)?; let _level = BlockSize::from_level(9)?; // Level 9 is 16 MiB. Ok(()) }

Block size

Item Value

BlockSize::MIN 65 KiB

BlockSize::DEFAULT 16 MiB

BlockSize::MAX 511 MiB

BlockSize::from_level maps levels 1 to 14 to these sizes:

Full API documentation: https://docs.rs/bzip3-rs.

Build the tool with the cli feature. The cargo install command above does this.

bzip3 -e input.bin # writes input.bin.bz3, removes input.bin bzip3 -d input.bin.bz3 # writes input.bin, removes input.bin.bz3 bzip3 -t input.bin.bz3 # checks the file, writes nothing bzip3 -r input.bin.bz3 # recovers a damaged file, keeps input.bin.bz3 bzip3 -c -d input.bin.bz3 # writes decoded data to standard output

To run from a checkout without install:

cargo run --release --features cli --bin bzip3 -- --help

Modes

The last mode flag wins.

Flag Action

-c, --stdout Write to standard output.

-f, --force Overwrite existing output.

-k, --keep Keep input files.

-v, --verbose Show byte counts.

-q, --quiet Hide non-essential warnings.

-b N, --block=N Set the block size. N is in MiB, or use NK / NKiB for KiB. Range: 65 KiB to 511 MiB. Default: 16 MiB.

-j N, --jobs=N Set the thread count, 0 to 64. Default: 0 (all cores).

-1 to -14 Set the block size by level. See the level table above.

--fast Same as -1.

--best Same as -14.

File rules

Encode adds the suffix .bz3.

Decode removes the suffix .bz3. If the name has no suffix, decode writes <name>.out.

Encode refuses an input name that already ends with .bz3.

After success, the tool removes the input file. It keeps the file with -k, -c, test mode, or standard input.

Recover mode never removes the input file. It keeps its output after a failure.

If argv[0] is bunzip3 or bz3cat, the tool decodes. bz3cat also writes to standard output.

Recover mode

Recover mode decodes every block that it can find.

The repository holds the scripts bz3diff, bz3grep, and bz3more. These scripts need the bzip3 tool in PATH. The directory man/man1 holds the man pages.

This crate reads files from C bzip3. C bzip3 reads files from this crate.

One-shot output uses a 13-byte frame header.

Streaming output uses a 9-byte frame header.

One-shot decode ignores bytes after the last block.

Streaming decode rejects trailing bytes that do not form a full block.

Streaming encode records the requested block size as given.

One-shot encode clamps small inputs to 65 KiB.

Differences from C bzip3

The default job count is 0 (all cores). C bzip3 uses 1 thread.

Output bytes are the same at every job count.

Recover mode writes zero bytes for a block with a fault other than the checksum. C bzip3 writes the decode buffer, which holds partial data.

Recover mode exits with code 2 when it finds damage. C bzip3 exits with code 0.

Recover mode never removes the input file. C bzip3 removes it unless -k is set.

Recover mode takes the block size from the block headers when the frame header has a bad block size. The reference uses 511 MiB.

Batch remove paths are absent.

Cargo features

Feature Default Purpose

cli no Builds the bzip3 tool. Adds the clap dependency.

bench-internals no Exposes stage functions for benchmarks. Not public API.

Minimum Rust version

Rust 1.88 or later. The crate uses edition 2024.

cargo test cargo test --features cli cargo clippy --all-targets --features cli cargo bench --bench end_to_end cargo bench --bench stages

The workflow perf-gate compares the speed of the bzip3 tool with the C bzip3 tool. The file tests/corpus/README.md describes the test corpus.

The crate has the license LGPL-3.0-only AND Apache-2.0.

This crate is an independent rewrite in Rust of bzip3 by Kamila Szewczyk.