Building DuckDB's Community Extensions, Faster and More Consistently

Every DuckDB engine release needs its community extensions rebuilt before they install. With data from 1,400+ PRs and a full-catalogue rebuild, we make the case for treating the catalogue as a distribution — and show Haybarn rebuilds it ~2.4× faster.

Haybarn CI (GitHub organization webhooks, snapshotted to DuckDB) and the GitHub Actions REST API for duckdb/community-extensions. Build time excludes queue wait. See Methodology.

Summary. DuckDB is superb at being a database engine. But upgrading it can still break things: install a new DuckDB, run INSTALL airport; LOAD airport;, and you may be told the extension is not available for your version — because each community extension is a separate binary that must be rebuilt for every engine release. Building the engine and distributing the software around it are different problems, and the distribution side has two: at an engine release the extensions aren’t built yet, and between releases it is unpredictable how long an extension update takes to ship. We argue — with data — that DuckDB’s extension catalogue is best treated as a distribution, the way Ubuntu is a distribution of Linux. Haybarn is that distribution for DuckDB — installable today with npx haybarn@rc or uvx haybarn-cli — and the findings below measure what it delivers.

Haybarn is an independent, derived distribution of DuckDB published by Query Farm LLC. It is not affiliated with or endorsed by the DuckDB Foundation; DuckDB is a trademark of the DuckDB Foundation.

DuckDB ships two things that travel together but are built and released separately. The engine — the duckdb binary and its embedded libraries — is the equivalent of an operating-system kernel: a fast-moving core that defines an ABI. The community extensions — over 250 of them — are the packages: independently-authored capabilities (geospatial, formats, connectors, ML) that must be compiled against a specific engine version and distributed as per-platform binaries.

A new engine release does not carry its extensions with it. Every extension binary is keyed to the engine version it was built against; when the engine moves, the entire constellation must be re-compiled and re-published before users on the new version can install any of it. This is the same split Linux has between a kernel and a distribution. DuckDB Labs has solved the hard part — the engine — brilliantly; what is not yet solved is the binary distribution and devops around it: compiling, caching, testing, and publishing the entire extension catalogue for every release, on every platform. That is distribution work, and it is exactly what Ubuntu does for Linux — take a fast-moving core and ship a coherent, tested, fully-packaged system on top of it. Few organisations run a raw kernel in production; they run Ubuntu.

This is not a knock on the DuckDB team so much as a matter of scope and capacity. A focused core team has its hands full advancing the engine, and the long tail of commercial devops and packaging concerns is a different, open-ended job — one the engine roadmap should not have to absorb. Several of those concerns are already concrete:

These are the everyday expectations of a distribution — multiple channels, version pinning, reproducible rebuilds — and they are exactly what commercial users need but do not yet get from the engine project alone.

Haybarn is a distribution of DuckDB, in the sense Ubuntu is a distribution of Linux. It tracks each engine release, rebuilds and tests the whole community-extension estate against it across the full platform matrix (Linux, macOS, Windows, WebAssembly; x64 and arm64), and publishes the binaries so that “upgrade the engine” and “all my extensions still work” become a single, consistent event rather than two events separated by an unpredictable gap of days. Crucially, on the Linux legs each build runs inside a fixed, pre-published container image rather than one rebuilt from scratch per job, so the toolchain is identical across the estate — reproducibility a distribution depends on, and, as we show below, a large source of its speed.

When DuckDB announces a core release, the community-extension binaries for that version do not yet exist — they are produced afterwards, extension by extension, as each is rebuilt and deployed. That window can stretch from hours to days, and it applies to more than major versions: a point release requires rebuilding every extension too, so each patch must wait for the catalogue to be rebuilt before it is whole. During that window a user who upgrades on the strength of the announcement and then runs INSTALL airport; LOAD airport; can be told the extension is unavailable for their version. The engine works; the distribution is not yet ready. And because that window varies — sometimes hours, sometimes days — it is increasingly hard to predict when it will close, leaving both customers and the developers who serve them wondering when it is actually safe to upgrade.

This is a packaging-readiness problem, and other ecosystems solve it by convention. A Debian release is not declared stable until its package set is built and frozen; the kernel advancing upstream does not, by itself, push a release to users. DuckDB core advances on its own cadence — on the order of 10 releases a year (12 in the trailing twelve months), per the official release calendar — and community extensions are version-locked: each must be compiled against, and published for, the engine versions it targets. A release therefore is not fully usable until its extensions have been rebuilt and published. For commercial users, predictability — knowing that upgrading the engine will not silently break the extensions they depend on — is not a nicety but a precondition for adoption.

We are not arguing that core releases should block on the entire catalogue; doing so perfectly is genuinely hard. We are observing that the absence of any consistency window pushes the integration cost onto every user individually, and that a distribution is the natural place to absorb it.

The packaging gap above is most visible at engine-release time, but the underlying pipeline runs year-round: extensions don’t only change when DuckDB does — authors ship fixes and new versions continuously, each as a pull request to duckdb/community-extensions. The predictability of that everyday, inter-engine pipeline is what a commercial user ultimately depends on, and it is degrading. If you build services or products on DuckDB, that makes it harder to tell customers when an extension update will actually be distributed to them. We analysed every PR merged there in the trailing year (1,435 PRs; 1,027 of them extension release/version PRs — “add extension”, version bumps, and updates) and measured end-to-end time-to-merge (merged_at − created_at), bucketed by month.

To be clear, this is not a knock on the maintainers, who are notably responsive: for much of the year a typical PR merged within hours. The point is the trend. Across complete months, median time-to-merge rose from about 3.4 h (2025-07) to about 13.5 h (2026-05), with p90 widening from 40.8 h to 61.6 h, as monthly submission volume grew several-fold. (The partial June window, n = 79 and overlapping a release crunch, is set aside.) Note this is end-to-end latency — it includes author and CI round-trips, not only maintainer response-time — so it measures how long a version change takes to land overall.

The mechanism matters for consistency. Throughput here is ultimately bounded by maintainer attention, which is finite and batched into working hours and time zones. As submission volume keeps rising, when a version lands becomes harder to predict — and that unpredictability, more than any median, is what makes a consistent release experience hard to guarantee. It is also the front of the same pipeline whose tail end is the INSTALL/LOAD gap above: a version is not usable until it has merged, rebuilt, and published.

Where Haybarn helps — and where it stops today. Haybarn already removes the build-and-distribute half of this lag: once a change is in the catalogue, it is rebuilt and published across every platform in a fast, predictable window. The merge half is still upstream’s — today Haybarn tracks duckdb/community-extensions by syncing its contents, so an extension update reaches Haybarn only once it has merged upstream. Because a distribution is also the natural place to own the merge step, Haybarn is positioned to offer faster, predictable merges directly — decoupling a customer’s update timeline from upstream merge variance — though that is a future capability, not what it does today.

Community extensions are mostly thin wrappers over heavy C/C++ libraries pulled in through vcpkg. The dominant cost of a build is therefore not compiling the extension — it is compiling those third-party dependencies. Whether that cost is paid depends almost entirely on the build cache, and this is where Haybarn and DuckDB differ by design.

DuckDB builds against a read-only cache. duckdb/community-extensions points vcpkg at the shared, read-only cache vcpkg-cache.duckdb.org. This is a sound default for a repository that builds untrusted pull requests: a read-only cache means a contributor’s PR can never poison the shared store, and it serves popular, already-populated dependencies well. Its limitation is that it cannot self-populate — on a miss (a new dependency, compiler, or triple) nothing is written back, so miss-prone targets like Windows-MinGW and the WebAssembly triples recompile the same dependencies from source on every run.

Haybarn builds against a read-write cache it operates. Query Farm rewired its extension-CI toolchain so that vcpkg’s binary cache (compiled dependencies) and asset cache (source tarballs) both point at a read-write store fronted by a Cloudflare Worker backed by R2 object storage. Reads are anonymous; writes are authenticated. A dependency is compiled once, written back on success, and reused by every later build across the whole estate. The asset cache additionally acts as a transparent mirror with origin fallback, so transient source-download outages no longer fail builds — improving build reliability alongside speed. (A ccache hashing fix for the WebAssembly toolchain compounds the effect.)

The result, measured on the same GitHub-hosted runner types via the same build.yml → _extension_distribution.yml matrix, is a per-platform speed-up that is largest exactly where the caching matters most. We compared 14 widely-used extensions built on both CIs, combining the Haybarn webhook-collector snapshot with timings pulled from the DuckDB GitHub Actions API.

A full-catalogue rebuild is about 2.4× faster on Haybarn end-to-end, and the per-platform legs explain why: ratios range from ~1.5× on macOS to 3.1–3.4× on Windows-MinGW and the WebAssembly triples — the dependency-heavy, cache-miss-prone targets. That is the signature of a cache that is populated and reused versus one that cannot self-populate. It is not a hardware advantage — both run on standard GitHub-hosted runners — but a build-pipeline one: Haybarn operates a writable, estate-global dependency cache (and takes on authenticating its writes) where a public PR repository is, reasonably, limited to a read-only one.

Drilling into one extension makes the mechanism concrete. a5 is the floor case: a small Rust extension with almost no native dependencies, so the dependency cache saves little here and what remains is mostly fixed per-build overhead. The chart breaks its linux_amd64 leg into steps on both CIs.

The single largest difference is the build-environment container. On its Linux legs DuckDB rebuilds the Docker image inline (288 s for a5); Haybarn pulls a pre-published image from GHCR (82 s) — saving about 3–4 minutes on this leg alone. Because that is pure fixed overhead, it is paid on every Linux leg of every extension; Haybarn’s CI notes estimate 5–8 minutes per leg for heavier images, on the order of