Shredding JSON columns gave Logfire up to 1000x faster queries

Querying data in Logfire is about to get dramatically faster. Consider >1,000 times faster in some cases. Queries that previously timed out after 30 seconds now complete in under a second.

"Seeing much faster <customer_attribute> query times! Feels basically instant now", says an early access customer

This technical blog post details how we achieved this through dynamic shredding, an optimization for querying semi-structured attributes data. We'll begin to roll out this feature to all customers in February 2026.

See what they're actually doing with Logfire — traces, evals, cost, and behavior in one view.

OpenTelemetry data (the data that Logfire ingests) is semi-structured data. It contains a mix of:

Semi-structured data is sent via attributes (per-span key-value pairs) or resource attributes (otel_resource_attributes in Logfire; these are per-resource key-value pairs where a resource is a service/k8s pod/etc, essentially an instance of your application or service).

app = FastAPI()

@app.get('/user/{user_id}') async def handle_request(user_id: str) -> dict[str, str]: with logfire.span(f'handle_request for user {user_id}', user_id=user_id): return {"user_id": user_id}

This generates an export along the lines of:

{ "resource": { "attributes": { "service.name": "my-web-service", "region": "us-west-2" } }, "spans": [ // Manual span created by user code { "span_name": "handle_request for user user123", "start_timestamp": "2024-01-01T12:00:00Z", "end_timestamp": "2024-01-01T12:00:01Z", "message": "handle_request for user user123", "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1", "span_id": "abcdef1234567890", "parent_span_id": "1234567890abcdef", "attributes": { "user_id": "user123", "http.status_code": 200, }, }, // Automatic span created by OpenTelemetry FastAPI instrumentation { "span_name": "GET /user/{user_id}", "start_timestamp": "2024-01-01T12:00:00Z", "end_timestamp": "2024-01-01T12:00:01Z", "message": "GET /user/user123", "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1", "span_id": "1234567890abcdef", "parent_span_id": null, "attributes": { "http.method": "GET", "http.response": 200, }, } ] }

I'm using JSON here for illustration, although we support various wire formats (OTLP protobuf, JSON, etc.); most data is sent as protobuf over gRPC. There are also some details like timestamps being represented as integers (epoch nanos) rather than ISO strings and an extra level of nesting from scopes that I've omitted for clarity.

We ultimately store this data as Parquet files in object storage, with a columnar schema roughly like:

start_timestamp: Timestamp(Microsecond, "+00:00") end_timestamp: Timestamp(Microsecond, "+00:00") span_name: Utf8View message: Utf8View trace_id: FixedSizeBinary(16) span_id: FixedSizeBinary(8) parent_span_id: FixedSizeBinary(8) attributes: Utf8View (JSON) otel_resource_attributes: Utf8View (JSON)

Some background on columnar storage, Parquet and zone map pruning is beneficial to understand the rest of this blog post. I recommend reading MotherDuck's excellent overview of the topic.

Pydantic started as a library to validate and parse JSON data efficiently. When we built Logfire, we initially stored all attributes as JSON blobs in a single column and relied on Jiter (one of the fastest JSON parsers out there in any language) to rip through the JSON data as quickly as possible. This is the layout detailed above (attributes: Utf8View (JSON)).

This approach was simple but made a fundamental mistake: we were optimizing to do work faster, but we couldn't really avoid doing the work. Compression is also suboptimal since a single varying attribute makes Parquet level dictionary compression ineffective and even zstd struggles with highly variable data, all the JSON syntax overhead, etc.

The biggest problem really ends up being extra IO: since most of our data is stored on object storage, extra IO from downloading large JSON blobs, and especially the latency of starting these downloads, ends up dominating query runtimes much more so than any CPU work.

To make queries fast, we need to avoid unnecessary work. For normal columns DataFusion / Parquet avoid unnecessary work by using columnar storage (allowing reading only the columns you need) and zone maps / min-max statistics (allowing skipping entire row groups or pages that don't match your filter predicates).

SELECT count(*) FROM records WHERE duration < 0;

The system can efficiently read just the duration column and skip row groups where the max duration is < 0 based on the stored statistics. In this case this is a valid query, duration is a f64 so in theory it could have values <0. In practice though duration is always >= 0 so this query will be very fast: based on statistics we know no rows match and we can return 0 immediately.

So how can we enable this for semi-structured data like attributes stored as JSON?

Our initial solution was static shredding: we manually picked a small set of commonly queried attributes (like http.status_code, http.method, etc.) and extracted these into their own strongly typed columns during ingestion.

Assuming our hardcoded shredding configuration extracts http.response.status_code, url.full, and db.query.text, we would store:

SELECT count(*) FROM records WHERE _lf_attributes_http.response.status_code = 500;

We can use statistics to prune files / row groups / pages that don't match, and then scan just the _lf_attributes_http.response.status_code column which is strongly typed (int64) and compresses well.

Of course you as a user don't have to use the _lf_attributes_http.response.status_code column directly, you can write:

SELECT count(*) FROM records WHERE attributes->'http.response.status_code' = 500;

And we rewrite this under the hood to use the shredded column.

I thought it worth mentioning the _lf_attributes prefix: one of the challenges with the current static shredding approach is that these columns are user-visible, e.g. if you run select * from records limit 10 you'll see these columns. Since these are an internal implementation detail, we'd rather not leak them to you. We want to keep the logical layout of attributes as a single JSON column from your perspective, while optimizing the physical layout under the hood.

There are several issues with this scheme:

Every query touching attributes that isn't filtering on the small hardcoded set of shredded attributes pays these costs.

SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';

This query needs to scan the entire attributes JSON column since my_llm_response is not shredded. Even though my_llm_response is a large attribute it may be a rare attribute that only appears in a small fraction of rows. Think about it this way: you probably have many more spans that don't involve LLM calls than ones that do, even in an agentic application.

┌─────────────────────────────────────────┐ │ Cost 1: Download Entire _lf_attributes │ │ (could be GBs of data) │ └─────────────────────────────────────────┘ ↓ ┌──────────────────────────────────────────┐ │ Cost 2: Poor Compression │ │ Variable structure → zstd 10% efficiency│ └──────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────┐ │ Cost 3: JSON Parsing │ │ Jiter does lazy parsing, worst case │ │ must parse entire row. │ └─────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────┐ │ Cost 4: In-Memory Operations │ │ Create copies of extracted values │ │ Do comparisons in memory │ └─────────────────────────────────────────┘

The Result: this query has to parse and scan potentially GBs of JSON data, much of which is irrelevant to the query, just to get the few rows that contain my_llm_response.

SELECT * FROM events WHERE attributes->>'user_id' = 'user123';

This query now has the opposite problem: even if 1/3 of rows contain a user_id attribute, we have to scan the entire JSON column to find them, which includes those large my_llm_response attributes that bloat the column size.

Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly typed columns. This happens automatically during data ingestion and compaction—no configuration needed.

Let's compare the physical layout of data before and after dynamic shredding.

This layout is generated dynamically for each Parquet file we write out based on the actual data we see. We don't have to include all null columns and we can prioritize shredding the most frequently seen or largest attributes.

SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';

It can efficiently scan just the _lf_attributes_my_llm_response column:

┌─────────────────────────────────────────┐ │ Benefit 1: Index pruning │ │ Zone maps and full text search indexes │ │ let us skip entire files / row groups │ └─────────────────────────────────────────┘ ↓ ┌──────────────────────────────────────────┐ │ Benefit 2: Download Only Relevant Column│ │ (_lf_attributes_my_llm_response) │ └──────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────┐ │ Benefit 3: Better Compression │ │ English text string data │ │ via zstd + dictionary encoding │ │ Maybe FastLanes in future? │ └─────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────┐ │ Benefit 4: Zero JSON Parsing │ │ Data already extracted & typed │ │ No deserialization, zero copy into mem │ └─────────────────────────────────────────┘

This query now benefits from the full efficiency of progressive narrowing / pruning and columnar storage. It is as fast as if my_llm_response were a first-class column in the schema.

We've been working on this feature for nearly 6 months now. It's required a lot of upstream changes in DataFusion to enable expression pushdown and predicate pushdown for JSON accessors. This is the blessing and the curse of working with an upstream open source project: we get to build on a solid foundation, but we also have to wait for upstream changes to land and get released. We've worked on this feature in a way that benefits not only us but anyone using DataFusion with semi-structured or nested data, including Parquet/Arrow struct columns and the newly introduced Json Variant type.

We've had to refactor core parts of the DataFusion execution engine to push down expressions into the Parquet reader so that it can use the physical file layout to optimize expression evaluation (in our case to decide if we need to read from the _lf_attributes remainder column or if there is a shredded column we can use instead). This work is done and has fully landed as of DataFusion 52.1.0 (Jan 2026).

The remaining work involves actually getting these expressions into the scan operators, e.g. for cases like:

SELECT attributes->>'http.response.status_code', count(*) FROM records GROUP BY attributes->>'http.response.status_code';

Pulling out the attributes->>'http.response.status_code' expression from the GROUP BY clause and pushing it down into the Parquet scan so that we can use the shredded column directly is a bit tricky, and involves complex expression rewriting and optimization passes, but we expect to land this work in DataFusion by Feb 2026. The upshot of this is that we've been able to contribute major improvements to DataFusion that benefit a broad community of users.

Our plan to land this while minimizing bugs and breakage for users is to first rewrite our query path (non-destructive, most of this has already been done) to use dynamic shredding but keep the static shredding in the write path. Along with this change we'll eliminate and isolate all notion of hardcoded shredded columns from the codebase into a self-contained section of the write path. Essentially our initial heuristic for dynamic shredding will mimic the current static shredding configuration. This lets us test