Brotli on Nginx thread pools: ngx_mbrotli vs. ngx_brotli

On-the-fly Brotli blocks the NGINX event loop. ngx_mbrotli on a thread pool against google/ngx_brotli: p99 latency, requests per second, CPU time and memory.

ngx_mbrotli is an NGINX module for on-the-fly Brotli compression, written in Rust on top of the mbrotli encoder. It is a drop-in replacement for google/ngx_brotli, with the same directives, defaults and $brotli_ratio variable. Unlike google/ngx_brotli, it can compress on NGINX thread pools, so that compressing a large response does not stop the worker from serving anything else.

This post measures what that changes. One NGINX worker served large files compressed on the fly while a second stream of small uncompressed requests went to the same worker. With google/ngx_brotli, the 99th percentile latency of the small requests rose from 1.9 ms to as much as 514 ms. With ngx_mbrotli on a 4-thread pool it stayed at 1.7–3.3 ms.

p99 latency of small requests while large responses are compressed

Large requests arrive 8 at a time. ms, logarithmic scale.

The numbers come from a reduced run under WSL2 on 20 October 2026. They are good for comparing the modules with each other, less so as absolute figures for your hardware. How it was measured has the details and what was not measured.

An NGINX worker process serves thousands of connections from one thread. Its event loop waits for sockets to become ready, runs a short piece of work for each one, and goes back to waiting. That works because each piece of work is short. A filter that runs for 100 ms holds every other connection of that worker for 100 ms.

Compression is such a filter. google/ngx_brotli and the stock gzip module compress in the event loop. When a 1 MB JavaScript bundle is compressed at Brotli level 6, it takes about 16 ms of CPU time, and at level 11 about 1.2 s. A small request that arrives on the same worker meanwhile waits for compression to end. When several large responses arrive at once, the worker compresses them back to back, and the small request waits for all of them.

NGINX has had a way out since version 1.7.11, released on 24 March 2015 with “experimental thread pools support” (CHANGES). NGINX itself uses thread pools for blocking file I/O (aio threads). ngx_mbrotli uses them for compression.

With brotli_thread_pool, the module cuts a response into segments of brotli_parallel_segment_size, 1 MiB by default. Each segment becomes an NGINX thread task (ngx_thread_task_t) and is compressed on one of the pool’s threads. Up to brotli_parallel_tasks segments of one response, 4 by default, are compressed at once. A response that fits in one segment is compressed by one thread, in one piece.

When a task is done, its completion handler runs in the event loop. It passes the compressed bytes down the filter chain in order and wakes the connection, as a write event would. While tasks are in flight, the request stays marked as blocked and buffered, so NGINX neither finalizes it early nor frees memory a thread is still using. Meanwhile the event loop serves other connections.

Segments are compressed independently: Brotli back-references do not cross segment boundaries. That makes segments safe to compress in parallel. It also makes a response longer than one segment compress slightly worse, as the numbers below show.

load_module modules/ngx_mbrotli.so;

thread_pool brotli threads=4;

http { brotli on; brotli_comp_level 6; brotli_types text/plain text/css application/javascript application/json image/svg+xml;

brotli_thread_pool brotli; }

NGINX must be built with --with-threads; nginx -V shows it. nginx.org’s packages are. The pool named default exists without being declared, with 32 threads per worker, and is shared with aio threads.

Small uncompressed requests (small.json, 1.9 KB) were sent at 1,000 per second over 32 connections. At the same time, a second stream requested large compressed files at a constant rate of about half of what google/ngx_brotli can sustain. All variants kept up with both streams.

p99 latency of small requests, large responses one at a time

Large requests arrive on one connection at a constant interval. ms, logarithmic scale.

“8 at once” is the large stream’s 8 connections, which wrk2 starts together and which stay in phase. It is what happens when several clients ask for large files at the same moment. In the event loop, a small request then waits for up to 8 compressions. “One at a time” is a single connection at a constant interval, where a small request waits for at most one. It is the lower bound of the effect: the pool cut p99 by 9–40 times there. On the pool, the maximum latency of a small request one at a time was 3.6 ms.

Without a thread pool, ngx_mbrotli lowers p99 by 14–41% against google/ngx_brotli, only because mbrotli compresses faster. It blocks the event loop all the same. The difference in tail latency comes from moving compression off the event loop, not from a faster encoder.

The same worker under closed-loop load, with wrk and 8 connections:

Requests per second, relative to google/ngx_brotli

8 connections. 1 task per response serves within 3% of 4, except one case. Logarithmic scale.

The pool’s 4 threads use the 4 CPUs that google/ngx_brotli, with one worker, cannot. That explains about 4× at levels 4–6. The rest comes from the encoder. At level 9 the C encoder allocates a 32 MiB hash table for every response over 64 KB and frees it afterwards. That cost does not depend on the response size, so mbrotli’s lead is largest on the 85 KB file. At level 11 a 1 MiB segment compresses faster than a whole file, so the pool gains more than its 4 threads.

With 8 connections, 1 and 4 tasks per response served within 3% of each other, with one exception (+17%). The pool’s threads were already busy with different responses.

More tasks per response help when a response is longer than one segment and nothing else competes for the threads:

libraries.json, three 1 MiB segments, one connection. ms, logarithmic scale.

4 tasks served it 2.0–2.3 times as fast as 1 task, and 2.2–5.4 times as fast as google/ngx_brotli. Responses up to 1 MiB fit in one segment, so the number of tasks does not change how fast they are served. react-dom.development.js is a 1 MiB segment and a 31 KB one, so 4 tasks served it only 4–34% faster than 1.

Level 11 is usually kept for pre-compressed static files, because one response can hold a worker for a second or more. On a thread pool, that time no longer holds up other requests: next to a stream of level-11 responses, small requests kept a p99 of 1.8–2.0 ms. A 2.2 MB response took 371 ms with 4 tasks, against 2,004 ms with google/ngx_brotli.

Level 11 still costs CPU. libraries.json needed 785 ms of CPU time per response on the pool, and the worker served 5.3 such responses per second on 4 CPUs. It fits responses that are large, compressible and requested rarely enough, not a busy API.

Memory. In the event loop, a worker compresses one response at a time and needs one encoder. On a pool, every response being sent holds encoders for its segments until it is sent. Peak memory then grows with the number of responses in flight and with the number of tasks per response.

VmHWM at 8 connections, MiB, logarithmic scale. The labelled value is the pool with 4 tasks.

In the event loop, ngx_mbrotli needs 2–5 MiB more than google/ngx_brotli. On the pool, it needs 11–26 times as much.

CPU time. CPU time per request, including the pool’s threads, was within 0.6–1.4 times that of the event loop. At level 11 the pool needed 37–40% less, because 1 MiB segments compress faster than a whole file. At level 9 it needed 22–40% more.

Worker and pool threads together, 8 connections. ms, logarithmic scale.

Compression ratio. Responses longer than one segment came out 0.9–5.2% larger, because segments are compressed independently. Responses up to one segment are byte-for-byte what the event loop produces.

The defaults suit most sites: a 1 MiB segment and 4 tasks per response. What to change and why:

The ngx_mbrotli README documents every directive.

Hardware. Intel Core i7-13700KF, Windows 10.0.26300.9711, WSL 2.7.14.0 with kernel 6.18.33.2-microsoft-standard-WSL2, Ubuntu 22.04.5. WSL2 shows 12 identical cores with 2 threads each and does not tell which are performance cores, so some of the pinned CPUs may be efficiency cores. Traffic went over loopback without TLS, and files were served from tmpfs (/dev/shm).

Build. NGINX 1.30.6, ./configure --with-compat --with-threads --without-http_rewrite_module, gcc 11.4.0, both modules dynamic. google/ngx_brotli at commit a71f9312 with the Brotli 1.1.0 it pins as a submodule. ngx_mbrotli at commit 8f7508a with mbrotli 0.5.10, built with rustc 1.99.0. This commit predates the 0.4.0 release; the current release is 0.4.5, and the run has not been repeated on it.

Configuration. worker_processes 1, with the worker and its pool pinned to 4 physical cores, and thread_pool cp threads=4. On the pool, brotli_thread_pool cp with brotli_parallel_tasks 1 or the default 4. Everything else was left at its default, including the 1m segment and encoder reuse.

Load. Requests per second and CPU time: wrk 4.1.0 on 4 other physical cores, 2 threads and 8 connections, 4 seconds per run (8 seconds for files over 1 MB, 5 times as long at level 11) after a 1-second warm-up. NGINX was restarted for every run, and each run started once the worker had been idle for 300 ms. Three runs, in an order shuffled between runs; tables give the median. CPU time is the worker’s user and system time from /proc/<pid>/stat, including the pool’s threads. Latency: two wrk2 streams (commit 44a94c1) at once, open loop, with coordinated omission corrected; 8 at once is the median of three 30-second runs, one at a time is a single 30-second run.

Files. jQuery 3.7.1 (jquery.min.js, 85 KB) and React DOM 18.3.1 (react-dom.development.js, 1.03 MB) from cdnjs, the full cdnjs API listing (libraries.json, 2.2 MB) and its first 10 entries (small.json, 1.9 KB), all fetched on 20 October 2026.

Correctness. Before the runs, every response in every combination of module, variant, level and file was decoded with curl’s Brotli decoder and compared with the original by SHA-256. All matched.

Not measured. gzip, more than one worker process, pool sizes other than 4 threads, a response-size threshold for the pool, HTTP/2 and HTTP/3, slow clients, long runs and bare-metal Linux. Runs were 3 per cell and 4–40 seconds long, shorter than a full benchmark.

ngx_mbrotli has packages for Debian, Ubuntu, RHEL and its derivatives, and Amazon Linux: install it from packages, or build it from source. Existing google/ngx_brotli configuration keeps working; add brotli_thread_pool to move compression off the event loop.

The same measurements for Zstandard, with ngx_mzstd against tokers/zstd-nginx-module, are in zstd on NGINX thread pools.

Yes, with google/ngx_brotli and the stock gzip module. The response is compressed by the worker process that serves it, inside its event loop, so the worker’s other connections wait until compression of that buffer ends. With ngx_mbrotli and brotli_thread_pool, compression runs on an NGINX thread pool and the event loop keeps serving other connections.

Install ngx_mbrotli, which accepts google/ngx_brotli’s configuration unchanged, and add brotli_thread_pool default; next to brotli on;. NGINX must be built with --with-threads. To size the pool on its own, declare thread_pool brotli threads=4; in the main context and use brotli_thread_pool brotli;.

With a 4-thread pool, one worker served 3.8–4.5 times as many requests at levels 4–6 and 4.3–15 times as many at levels 9–11, on 85 KB–2.2 MB files. Without the pool, in the event loop like google/ngx_brotli, it served 5–64% more requests, and 4.4 times as many at level 9 on an 85 KB file.

Much more than in the event loop. Every response being sent holds encoders for its segments. At 8 connections on 1–2.2 MB responses, one worker peaked at 138–995 MiB a