返回资讯中心
外部精选
软件工程
#开发者实时流

Pcodec is 15% smaller and 2x faster than ClickHouse's DoubleDelta in compression

I showed how ClickHouse outcompressed our tuned Parquet setup on our IoT dataset.

Hacker News 最新theanonymousone2026/10/19 · 27 天前19 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

In my previous post, I showed how ClickHouse outcompressed our tuned Parquet setup on our IoT dataset. It did this by having better compression codecs than Parquet, and the star of the show was the DoubleDelta codec. In this post, I will show the key ideas used by pcodec, a more modern compression codec added just days ago to ClickHouse, to significantly beat DoubleDelta on both size and decompression speed, at the cost of compression speed.

A quick recap of the previous post. We ingest over a trillion GPS pings from vehicles per year, and sort them in a columnar format such that consecutive pings from the same vehicle are next to each other. The tuned Parquet schema looks like this:

int64 s2_cell_id INTEGER(64, false); -- PLAIN binary session_id STRING; -- PLAIN int32 latitude_e6 INTEGER(32, true); -- DELTA_BINARY_PACKED int32 longitude_e6 INTEGER(32, true); -- DELTA_BINARY_PACKED int64 time_sent TIMESTAMP(MILLIS, true); -- DELTA_BINARY_PACKED int64 time_received TIMESTAMP(MILLIS, true); -- DELTA_BINARY_PACKED int32 speed INTEGER(8, false); -- PLAIN int32 heading INTEGER(16, false); -- PLAIN int32 hdop INTEGER(16, false); -- PLAIN

And the ClickHouse schema looks like this:

CREATE TABLE gps ( s2_cell_id UInt64 CODEC(ZSTD(3)), session_id String CODEC(ZSTD(3)), latitude_e6 Int32 CODEC(DoubleDelta), longitude_e6 Int32 CODEC(DoubleDelta), time_sent DateTime('UTC') CODEC(DoubleDelta, ZSTD(3)), received_latency Int32 CODEC(T64, ZSTD(3)), time_received DateTime('UTC') ALIAS time_sent + received_latency, speed UInt8 CODEC(Delta, ZSTD(3)), heading UInt16 CODEC(ZSTD(3)), hdop UInt16 CODEC(T64, ZSTD(3)) ) ENGINE = MergeTree ORDER BY (s2_cell_id, session_id, time_sent)

The Parquet schema gets about 8.2 compressed bytes/row, while the ClickHouse schema reaches about 5.9 bytes/row. A large part of the difference comes from the latitude and longitude columns. They are a combined 3.92 compressed bytes/row in Parquet vs 3.08 bytes/row in ClickHouse, and the difference can be explained by the fact that DoubleDelta is a better "predictor" than DELTA_BINARY_PACKED's single delta:

Imagine a vehicle is driving along a straight road. Delta stores the difference between two consecutive points (blue vectors). DoubleDelta stores the difference between two consecutive deltas (red vectors). Another way to think about it is that DoubleDelta "predicts" the next point will be at the same delta offset as the previous point's delta, and then "corrects" itself with a second offset. In this case, it is clear that the red vectors are smaller than the blue vectors, which can intuitively explain why it compresses better.

The 5.9 compressed bytes/row result was sufficiently good for my employer, but on a personal level, I was interested in seeing if it could be improved further. That's when I found out about pcodec. It seemed ideal for my use case, and initial tests were very promising.

When I started doing research into pcodec last week, it was a rather niche codec, only implemented in experimental file formats like Vortex. I was quite surprised by this, as the relative obscurity of the codec contrasted heavily with the remarkable results I was getting. While there had been some attempts to add pcodec to ClickHouse since at least 2024, these had not been successful. I was initially planning to just write a more theoretical blog post about this codec, noting that it was not really accessible right now.

That is, until just this week, when Alexey Milovidov himself finally added a beta implementation of the codec to ClickHouse! This means that, when ClickHouse 26.10 drops at the end of this month, everyone running on the latest version of ClickHouse will have access to this new codec. Suddenly, this blog post feels a lot more relevant!

Let's show the results first. When we add pcodec to our toolbox of codecs, we drop down to an incredible 4.88 compressed bytes/row! Here's the updated schema:

SET enable_pco_codec = 1; -- Needed as PCO is in beta

CREATE TABLE gps ( s2_cell_id UInt64 CODEC(ZSTD(3)), session_id String CODEC(ZSTD(3)), latitude_e6 Int32 CODEC(PCO), longitude_e6 Int32 CODEC(PCO), time_sent DateTime('UTC') CODEC(PCO), received_latency Int32 CODEC(PCO), time_received DateTime('UTC') ALIAS time_sent + received_latency, speed UInt8 CODEC(Delta, ZSTD(3)), heading UInt16 CODEC(PCO), hdop UInt16 CODEC(PCO) ) ENGINE = MergeTree ORDER BY (s2_cell_id, session_id, time_sent);

And here are the per-column results for the columns we switched to PCO:

PCO achieves this while beating DoubleDelta on decompression speed. Compression speed is about 2x worse, but this is generally an acceptable trade-off. All values in the table below are "X Million values per second (de)compressed":

Compared to other codecs, PCO is surprisingly strong across the board. It matched or beat our previous result on all columns I tried it on1.

That's exactly what I wanted to know as well after Claude spit out the initial 4.88 bytes/row result. So I went to the pcodec GitHub repo where I found this intimidating explanation:

Clearly, this codec is quite a bit more complex than most. Luckily, after diving through the source code, it can be broken down into three understandable steps. The first two steps are data preparation steps that don't compress the data, while the final step does compress the data, and is the most interesting part of the codec.

GCD is a niche but useful data preparation codec in ClickHouse. It calculates the greatest common divisor of all values in each block of data (1 MB uncompressed by default), and divides each value by the GCD of its block. In cases where your data has a GCD > 1 (e.g., you use second precision timestamps but your timestamps are always on an exact minute: GCD of 60), it's a good idea to use this codec, as it reduces absolute values of the data, making it easier to compress.

Unfortunately, GCD has a weakness: It can be "poisoned". If all but one values in a block are divisible by 60, that one value will ruin the GCD for the entire block, and you won't get the compression benefits. PCO solves this by using heuristics to determine whether it's worth doing the "GCD" step anyway, even if not all values are divisible by it. For values that are divisible by the "GCD", it will just store the result of that division. For values that aren't, it will store both the result of the division and its remainder.

Most of the time, this step doesn't help, as the data doesn't have a "GCD" > 1. Therefore, I won't go into detail on the exact heuristics used.

Step 2 is similar to ClickHouse's Delta and DoubleDelta codecs. PCO takes a small sample of values, tries a few delta encoding setups, and runs it through the final compression step to see if delta encoding can improve the result. It tries:

It then picks the setup on which the compressor for step 3 performed best.

For our latitude/longitude columns, PCO finds a "GCD" of 1, and finds that "delta of delta" gets the best results. This means that up to this point, it exactly matches what DoubleDelta does.

Both PCO and DoubleDelta take the residuals from the previous step and bit pack them. The difference lies in how that bit packing is done.

DoubleDelta uses a static table to convert the residuals to bits in a single bitstream. The logic is as follows:

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →