LTAP 是否只是 CDC?业界探讨湖仓事务与分析处理的关系
来源标题:Is Lake Transactional/Analytical Processing (LTAP) Just CDC?
有分析指出,Lake Transactional/Analytical Processing(LTAP)与 CDC 的关系值得澄清。LTAP 强调在数据湖上同时支持事务处理与分析处理,而 CDC 仅负责捕获数据变更。两者定位不同,LTAP 并非 CDC 的简单替代或等同。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Reynold Xin's post "From monolith to Lakebase to LTAP" explained what LTAP is and how the Lakebase architecture makes it possible: one governed copy of operational data, serving both Postgres and analytics, with no CDC pipeline and no second database.
Much of the commentary that followed read that claim through systems people already know. Four readings came up often:
Each of these myths rests on a wrong assumption about how Lakebase stores data. Let's take them one at a time.
The reality: LTAP keeps one durable copy of each table, in the lake. The layers in front of it hold some of the same bytes to make that table durable and fast, but none of them is a second database.
From the outside, LTAP can look like a familiar pipeline. Postgres writes rows, a process tails the log, the changes land in an Iceberg table, and analytics reads Iceberg. That pipeline is CDC. It is also the structure of logical replication, mirroring, and most zero-ETL products. So it is easy to classify LTAP as the same design.
To see why it isn't, it helps to be precise about what counts as a second copy. There are two different ways a system can end up holding the same data in more than one place:
So the test is simple: can this layer be queried, or failed over to, as its own database? If yes, it is a replica. If no, it is a tier.
By that test, Postgres has always been tiered. The WAL, the buffer cache, and the indexes all store bytes that overlap with the heap. Nobody calls an index a replica of the table, because it has no catalog or timeline of its own.
LTAP keeps that same hierarchy. It moves it onto object storage and lets a lakehouse engine read the bottom, durable tier.
Here is what that hierarchy looks like in Lakebase, following a write and then a read:
Object storage itself works the same way. When you PUT an object into S3, it is stored more than once, often across availability zones using erasure coding. The API still names one object, and no sync job runs between the copies. Lakebase stacks the same idea in front of the lake: log, caches, pageserver, object storage. The extra files are how one table is made durable and fast.
CDC does create a second database. It decodes a logical change stream into a foreign schema, usually keyed by primary key, and upserts into a table the source never reads back. That second table can be queried on its own, and it can fall out of step with the source: types can drift, DDL can go missing, a replication slot can drop rows, and tables nobody selected never appear. The result is two catalogs.
LTAP has no second table to keep in step. There is no list of tables to opt into. A table that exists in Postgres exists in the lake, and the pages the primary serves are caches of that store.
The reality: two lossless encodings of the same rows can serve different access paths - heap pages for point lookups, Parquet for scans. Each query reads the layout that's fast for it.
Even if the extra files are a hierarchy, layout remains a separate question. Point lookups are efficient on rows, scans are efficient on columns. If there is only one encoding on disk, one of those access patterns pays in I/O. Calling the table a single logical copy does not change that.
That reasoning is correct, but it rests on one assumption - that one copy implies one encoding. It doesn't, as long as two formats can represent the same rows without loss.
So the question is whether LTAP's two formats are lossless:
Because nothing is lost, the columnar copy can be the durable store in the lake, and the heap pages become a cache of it. Each access path then reads the format that suits it. Lookups hit 8 KB pages on the compute node, on local disk, and on the pageserver. Scans hit Parquet through an OLAP engine. Indexes remain in the hot tier; they are not transcoded into Parquet.
Postgres already works this way. It keeps two physical structures over the same rows for different access paths: a heap and a covering index. Nobody treats the index as a CDC replica of customer. SELECT * FROM t WHERE id = 1 and SELECT count(*) FROM t are two access paths over one table.
A narrower observation is valid. The columnar files from Postgres are heap-ordered: rows sit in page-number order, because the page is the unit rewritten on each compaction. That layout is efficient for reconstructing a page and for rewriting only dirty pages. It is a poor clustering key for WHERE order_date >= …. If a warehouse query needs that clustering, it can be derived the way silver tables are already derived from any operational source. What is no longer required is a CDC pipeline whose purpose was to create the lake copy.
The reality: every analytical query reads a consistent snapshot. Commits that haven't reached the lake yet are read from the pageservers, so nothing committed before the query is missed.
Iceberg and Delta provide snapshot isolation on object storage, and object storage has high latency. So it is natural to conclude that analytics is always behind, that many small tables cannot stay fresh, and that a join across tables cannot be consistent. That description fits CDC ingestion into a lake.
It depends on one assumption: that a query can only see what has already landed in object storage. In LTAP, a query can also see pages that are not in object storage yet. That changes the answer, and it is easiest to see by following a query.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。