The Iceberg Interoperability Myth: Write Once, Read Anywhere?

15 min readOn this page26 sectionsFirst, What Does "Open Table Format" Actually Promise?Next, Meet the Table Before Anything Is DeletedThen a Delete ArrivesWhat an Equality Delete SaysSpark Reads the New SnapshotSnowflake Reaches the Same Table, Then StopsWhy the Catalog Cannot Fix the ProblemThe Other Row-Level Delete Encodings1. Position deletes: name the exact row location2. Deletion vectors: keep a compact bitmap3. Copy-on-write: remove outstanding delete semanticsThe Compatibility Rule: Use the Intersection, Not the UnionHow to Prevent the FailureStep 1: List every required read and write pathStep 2: Build a feature-level matrixStep 3: Make the writer compatibility-awareStep 4: Test an actual changed snapshotStep 5: Recheck compatibility when the stack changesWhat If Equality Deletes Are Already in Table?Rewrite the affected dataConvert to a supported positional representationPut a compatible serving table in front of the readerTemporarily use append-only ingestionA Practical Pre-Production ChecklistOpen Does Not Mean IdenticalFurther Reading

"Apache Iceberg is open, so any Iceberg engine can read an Iceberg table."

It sounds reasonable but it is incomplete.

Iceberg gives engines a shared specification for describing a table. That is a major step toward interoperability, but a specification is a menu of capabilities, not proof that every engine implements every item on that menu.

Row-level deletes make this gap easy to see. A writer can create a valid Iceberg table with equality delete files. Apache Spark can understand those files and return the current rows. Another engine may not support equality deletes. It can reach the catalog, load the metadata, and read the Parquet files, yet still be unable to produce the correct result.

The table is valid. The reader is valid. The combination is incompatible.

This blog follows one small customer table from its first write to the moment two engines disagree about whether they can read it. Along the way, we will separate four layers that are often bundled together under the word open:

By the end, you will have a practical way to evaluate Iceberg compatibility before a production query discovers the mismatch for you.

Imagine object storage as a warehouse full of sealed boxes. Each box is a Parquet data file. Parquet defines how the rows inside a box are encoded, but it does not tell you:

Iceberg adds that table-level instruction set. Its metadata tree connects snapshots, manifest lists, manifests, data files, and delete files. Because the specification is open, different systems can implement the same rules without relying on a proprietary storage layout.

That promise is real. The common mistake is stretching it too far.

"An engine is allowed to implement these structures and semantics."

"Every product, version, connector, and catalog integration has implemented all of them."

Web browsers offer a familiar parallel. HTML is an open standard, but a browser that supports HTML does not automatically support every new web API. In the same way, an "Iceberg-compatible" engine may support basic scans while lacking a newer format version, a data type, a write operation, or one of Iceberg's delete encodings.

Suppose a company keeps this small customers table in Iceberg:

For simplicity, assume all three rows live in one Parquet file:

s3://warehouse/customers/data-0001.parquet

The catalog stores, or helps an engine locate, the current Iceberg metadata file. That metadata eventually points to data-0001.parquet.

At this moment, interoperability looks effortless. Spark and Snowflake can both:

This happy path is where the phrase "write once, read anywhere" feels true. The table contains only data files and broadly supported metadata. No reader has yet been asked to interpret a row-level change.

Now Ben asks for his account to be removed. The source database emits a CDC event equivalent to:

DELETE FROM customers

WHERE customer_id = 102;

Parquet files are immutable. A writer cannot simply open data-0001.parquet, erase its second row, and save the file in place. It must represent the change another way.

At a high level, an Iceberg writer has two strategies, copy-on-write and merge-on-read:

Copy-on-write leaves a simple read path, but rewriting a large file for a small change can be expensive. Merge-on-read makes the write lighter, which is attractive for frequent CDC updates, but it moves work, and a compatibility requirement, to every reader.

Our writer chooses merge-on-read and emits an equality delete.

An equality delete identifies rows by one or more field values. In this example, the delete file effectively says:

For the applicable older data: hide rows where customer_id = 102

The real Iceberg rules also use field IDs, partitions, and sequence numbers to determine which data files and rows the delete applies to. Those details prevent an old delete from incorrectly removing a newer row that happens to reuse the same key. For understanding interoperability, however, the value match is the important part.

data-0001.parquet # still contains 101, 102, and 103

delete-0002.parquet # contains equality value customer_id = 102

The new Iceberg snapshot references both files. Reading the data file alone is no longer enough. A correct reader must plan the data file together with the applicable equality delete, compare the equality fields, and filter Ben's row from the result.

Spark loads the same table again. Its Iceberg integration recognizes the content type of the delete file and supports equality-delete semantics.

Conceptually, Spark performs this operation:

SELECT data.*

FROM data_file AS data

LEFT ANTI JOIN equality_delete AS deleted

ON data.customer_id = deleted.customer_id;

That is not the literal physical plan, but it is a useful mental model. Spark returns:

Nothing surprising happens because Spark and the writer share the required capability: both understand Iceberg v2 equality deletes.

Now consider a Snowflake access path that can discover this externally managed Iceberg table but does not support reading equality delete files.

Snowflake can still reach the catalog, resolve the current metadata file, and understand the table schema. But that does not mean it can execute a query against the current snapshot.

When the scan planner encounters delete-0002.parquet, Snowflake cannot interpret the equality-delete semantics. It therefore throws an unsupported-feature error and the query fails.

So the same snapshot produces two outcomes:

This is the interoperability myth in one line:

Access to the same catalog and files does not guarantee support for the same table semantics.

Product capabilities change over time and often differ between native-managed tables, externally managed tables, and catalog-linked tables.

The word catalog sometimes suggests a smart translation layer. Usually, that is not its job.

An Iceberg catalog primarily helps with operations such as:

The catalog tells a reader where the current table state is. It does not normally open equality delete files, convert them into position deletes, or rewrite data files for each engine.

Think of a library catalog. It can tell two readers where a book is located. It does not translate the book into a language one reader understands. A REST catalog such as Apache Polaris still only locates metadata; it does not apply or rewrite equality deletes for engines that lack that capability.

Catalog choice still matters because products expose different features through different catalog integrations. A query engine may offer full read/write behavior for its native catalog, partial reads through a REST catalog, and a different subset through Glue or another integration. The key is to test the pair:

query engine × catalog × table feature

Checking only "Does this engine support Iceberg?" is too coarse. For OLake-written tables, start with the engine and catalog pairing table.

Equality deletes are only one way to record a row-level change. Understanding the alternatives, position deletes, deletion vectors, and copy-on-write, shows why a writer's decision affects every future reader.

A position delete records a data-file path and row position. In simplified form:

s3://warehouse/customers/data-0001.parquet, row 1

The reader does not have to match customer_id values across the scan. It skips a known row position. The trade-off is on the write side: the writer first needs to find the file and position containing the target record.

Position deletes may improve compatibility when all target readers support them, but converting from equality deletes is not a metadata rename. Someone must resolve keys to physical file positions or rewrite the affected data.

Iceberg v3 adds deletion vectors: compact bitmaps that identify deleted row positions for a data file. They reduce the proliferation and scan overhead of separate positional delete records.

They also raise the compatibility bar. An engine that supports Iceberg v2 but not v3 cannot automatically read a v3 table using deletion vectors. Newer is not universally safer; it is safer only when every required reader supports it.

v3 also adds row lineage: a stable _row_id on each row so engines can track which records changed.

With copy-on-write, the writer creates a replacement data file:

data-0003.parquet # contains only 101 and 103

The new snapshot no longer needs an outstanding row-level delete for this change. Readers only need to scan the rewritten data file. This often gives the broadest read compatibility, at the cost of more write I/O.

The "best" method is therefore not just the fastest one for a single engine. It is the method supported by the full set of readers while meeting the workload's write and read requirements.

If Spark supports equality deletes and position deletes, while the other engines support only position deletes, the shared capability is position deletes.

safe table features =

writer capabilities

∩ Spark read capabilities

∩ Snowflake read capabilities

∩ every other required reader's capabilities

The union answers, "What can at least one engine use?" The intersection answers the operationally useful question: "What can every required engine use?"

This rule applies beyond deletes. Format version, timestamp precision, variant or geospatial types, partition transforms, encryption, and branch/tag behavior can all create similar gaps.

A reliable design starts before the first CDC delete is committed.

Spark 3.x + Iceberg library version + REST catalog

Snowflake external Iceberg table + catalog integration type

Trino version + Glue catalog

A native table and an externally managed table in the same product may have different capabilities.

Record whether each capability is generally available, in preview, read-only, or unsupported. Include the date and versions because this matrix will change.

The writer should choose a strategy from the intersection of downstream capabilities. That may mean:

For CDC, this choice is part of the table contract. It should not be an accidental default hidden in one writer configuration.

An initial SELECT COUNT(*) proves very little. A meaningful interoperability test should:

Adding an engine, changing a catalog integration, upgrading to Iceberg v3, or enabling merge-on-read can change the safe intersection. Treat each as a compatibility review, not a routine connection change.

If one of your required readers cannot process the current snapshots, you have several options.

Apply the equality deletes and create clean replacement data files. This copy-on-write style migration removes the outstanding equality-delete requirement from the resulting snapshot. We documented one such path: a MOR-to-COW rewrite that makes equality-delete tables readable in Databricks and other engines that cannot apply those files.

Resolve each deleted key to its data file and row position, then write a representation supported by the target readers. This can avo