BINPACK is a high-performance polymorphic binary compression engine designed to resolve structural entropy bottlenecks in standard LZ77 and Finite State Entropy (FSE) compressors (such as ZIP/Deflate, BZIP2, and raw Meta Zstandard) when operating on non-text payloads.

It profiles input buffers in sub-microsecond latency (< 1 µs) to detect underlying data layout, applies zero-loss reversible structural pre-filters at memory-bus speeds (> 2 GB/s), and feeds normalized streams into high-density entropy and deduplication backends.

Standard general-purpose compressors view incoming data as flat streams of opaque bytes. On structured binary data, this approach incurs substantial compression penalties:

Input Buffer │ ▼ [Polymorphic Profiler] (C99, < 1 µs inspection) │ ├─► PE / ELF / Mach-O (x86/x64) ──────► [BCJ x86 Filter] (Relative offset normalization) ├─► PE / ELF / Mach-O (ARM/ARM64) ────► [BCJ ARM Filter] (BL instruction normalization) ├─► Continuous 16-bit Waveforms ──────► [Delta-2 Filter] (Finite difference transformation) ├─► Structured 16-bit Records ────────► [Stride-2 Transpose] (Channel separation) ├─► 32-bit Floats / 3D Coordinates ───► [Stride-4 Transpose] (Field separation) ├─► 64-bit Doubles / Timestamps ──────► [Stride-8 Transpose] (Field separation) └─► Unstructured Binary ──────────────► [Passthrough] │ ▼ [High-Density Backend] (Zstandard / LZMA2) │ ▼ .bpk Container (Magic + Mode + SHA-256 Checksum)

Benchmark Results

All benchmarks executed under identical hardware conditions on official dataset corpora and standard operating system binaries.

Comparisons against Meta's Zstandard Level 22 (Ultra) and standard Deflate (Level 9):

All outputs verified 100% lossless bit-for-bit with SHA-256 checksum validation.

Measured on x86-64 hardware using the native C99 implementation (binpack.c):

Include binpack.h in your C or C++ application:

#include "binpack.h" #include <stdio.h> #include <stdlib.h>

void process_buffer(const uint8_t data, size_t size) { / 1. Sub-microsecond format detection */ binpack_mode_t mode = binpack_detect_mode(data, size); printf("Detected layout: %s\n", binpack_mode_name(mode));

/* 2. Apply appropriate domain-specific transformation / uint8_t filtered = (uint8_t)malloc(size + 16); if (mode == BINPACK_MODE_BCJ_X86) { memcpy(filtered, data, size); binpack_filter_bcj_x86(filtered, size, 1 / encode /); } else if (mode == BINPACK_MODE_DELTA_2) { binpack_filter_delta_encode(data, filtered, size, 2 / distance /); } else if (mode == BINPACK_MODE_STRIDE_4) { binpack_transpose(data, filtered, size, 4 / stride */); }

/* 3. Feed filtered stream into Zstandard or storage engine */ // ...

free(filtered); }

CLI Reference Implementation

The repository includes a Python CLI utility (binpack.py):

Python 3.10+

pip install zstandard numpy

Usage

Run benchmark against ZIP, BZIP2, and Meta Zstandard Ultra

python binpack.py bench payload.exe

Compress a file

python binpack.py compress input.dll -o input.dll.bpk

Decompress a file (with automatic SHA-256 integrity check)

python binpack.py decompress input.dll.bpk -o input_restored.dll

On Windows, use binpack.cmd:

.\binpack.cmd bench ooffice

Building the Native Engine

Windows (MSVC or GCC)

Run the provided build script:

cl /O2 /W3 binpack.c test_native.c /Fe:test_native.exe

Linux / macOS

make ./test_native

Licensing

This project is licensed under the MIT License.

It is 100% free and open-source software. You are free to study, modify, distribute, and integrate this engine into both open-source and proprietary commercial software without fees or restrictions.

For questions, technical feedback, or collaboration:

santiagorobledo198@gmail.com

Or open an inquiry via GitHub Repository Issues.

BINPACK is an independent, 100% free, and open-source project under the MIT License. If this engine saves you bandwidth, accelerates your storage pipelines, or assists your research, consider supporting continued algorithmic optimizations:

This architectural proposal and empirical findings have been formally submitted upstream to the Meta Zstandard engineering team for technical discussion:

To reference this project in academic research or technical benchmarks: