Weekly Paper Notes — one of the top picks from the 2026-11-14 CS paper digest. Area: Databases.

Authors: Keonwoo Oh (University of Maryland), Akshay Mehra (Dolby Labs), Joshua Kimball (Dolby Labs), Amol Deshpande (University of Maryland) arXiv: 2611.15975 · PDF · Code · Submitted to ICDE 2027

TL;DR

Lakehouse metadata sits at one of two extremes. Open table formats (Iceberg, Delta Lake, Hudi) write everything as files next to the data. That scales, but each small commit leaves a pile of tiny metadata files behind, commits are slow because they’re a chain of object-store writes plus a compare-and-swap, transactions stop at one table, and garbage collection is retention-based and can delete files that are still in use. DuckLake and TreeCat go the other way and put everything in a transactional database. That fixes the transaction problems but runs into per-file column statistics, which grow with columns × files: DuckLake needs 109 GB of Postgres for a single 10,000-column, 100,000-file table.

Raven splits the metadata by what it’s used for. Facts that need transactional guarantees, such as which files belong to which snapshot and which entities reference which, go into a database (Postgres in the prototype). Analytical metadata, mainly per-file min/max/null-count statistics, stays in the OTF’s own manifests. On top of that split the paper builds three features: extended Merge-on-Read for cheap small commits, multi-table transactions under snapshot isolation, and reachability-based garbage collection over a user-extensible entity–relationship model. Integrated with Iceberg and Delta Lake on S3, Raven raises small-insert throughput 55–61%, keeps its catalog under 140 MB where DuckLake’s grows past 3 GB, scales multi-table commits to 146 txn/s where DuckLake stalls at 25, and deletes about 2,200 obsolete files per second, close to the S3 DELETE rate limit.

Where lakehouse metadata breaks

The paper’s background is a short history of catalogs. Hive Metastore kept a pointer from each table or partition to a directory, and engines had to list() the object store to find files, which is slow with millions of files and offers no transactional guarantees. OTFs fixed that by writing the file list down as metadata files: an Iceberg commit writes a manifest, a manifest list, and a new root metadata JSON, then CASes the catalog’s pointer to it. Modern catalogs (Unity Catalog, Polaris, Nessie, OpenHouse) still mostly store that one pointer per table and leave file-level tracking to the format.

The authors list four problems that follow:

  1. Small files. Every streaming micro-commit produces several metadata files, which makes reads slower and puts load on the object store.
  2. Commit latency. Several object-store writes plus a CAS on every commit.
  3. No multi-table transactions. The atomic step is a CAS on one table’s pointer.
  4. Unsafe deletion. Retention-based GC (expireSnapshots, VACUUM) assumes old snapshots are unused. An ETL job that runs longer than the retention window, or a shallow clone in another table, can lose files out from under it. Delta’s VACUUM doesn’t check for incoming references from shallow clones, and Iceberg only stays safe by restricting clones to a single table directory.

The database-only fix runs into a different wall. Industry has hit it already: BigQuery moved part of its file metadata out of Spanner and into a BigQuery table, and Databricks has reported compressed metadata reaching 1 TB for a single table. The paper’s own measurement makes the scaling explicit:

Columns DuckLake Postgres size (100k files)
10 0.15 GB
100 1.14 GB
1,000 10.97 GB
10,000 109.31 GB

At 10,000 columns, statistics account for over 99.9% of the footprint.

The split

Raven’s central decision is which metadata goes where:

Metadata HMS + Iceberg Raven + Iceberg DuckLake
Table name DB DB DB
OTF root path DB DB —
Table versioning Object store DB DB
Schema Object store DB DB
File visibility Object store DB DB
File statistics Object store Object store DB

Everything needed to decide what a snapshot contains and whether a file is still referenced is transactional and lives in the database. Everything needed to prune files during a scan stays in Iceberg manifests or Delta checkpoints, where engines already know how to read it.

Figure 2 of the paper: an ER schema with Workspace, Database, User, Asset (Table/Model), Asset Snapshot (Table/Model Snapshot), Session and File entities linked by Contains, Owns and Pins relationships. Figure: An example Raven metadata schema. Red edges are referential relationships that the garbage collector follows; dashed boxes are abstract entity types such as Asset. Source: arXiv:2611.15975, Figure 2.

The catalog’s schema is user-defined, written in a small DSL called ERQL that is compiled to SQL. You declare entities and relationships, including abstract ones, so that tables and ML models can both be “Assets” inheriting Contains(Database, Asset):

CREATE ABSTRACT ENTITY Asset ();
CREATE RELATIONSHIP Contains (Database, Asset);
CREATE ENTITY Table (...) IMPLEMENTS Asset;
CREATE ENTITY Model (...) IMPLEMENTS Asset;

File-level tracking goes through a special construct, FILELIST. It is a versioned collection of files, keyed by (object ID, version ID), that exposes a flat relational view (path, format, size, mtime, tag) whatever the internal OTF layout. An Iceberg snapshot becomes one FILELIST version whose rows tag each file as metadata, manifest list, manifest or data. Clients query it directly, and a DELTA form returns added and removed files between two versions, which is useful for CDC and incremental cache refresh:

SELECT action, file_path, tag
FROM FILELIST DELTA TableSnapshot(13, 2, 3);

Physically, each FILELIST history is a Postgres partition in which every file row carries start_vid and end_vid. A snapshot read is a range predicate, and millions of files shared between consecutive snapshots aren’t duplicated.

Three features built on the split

Figure 4 of the paper: client, catalog service, metadata store, garbage collector with graph store and graph engine. Figure: Raven’s components. The catalog service compiles ERQL to SQL against the metadata store; the garbage collector consumes edge logs into its own graph store and in-memory engine. Source: arXiv:2611.15975, Figure 4.

Base protocol. Concurrency control is OCC over MVCC. The client fetches a table snapshot and its version ID, reads files, writes new ones, and submits the added and removed files. Raven executes the whole change as one serializable Postgres transaction that acts as the CAS. For Iceberg this replaces the catalog-pointer CAS entirely. Delta still needs its atomic log rename.

Extended Merge-on-Read. Since Raven already knows which files are visible, a small append doesn’t have to produce OTF metadata immediately. The fast path registers the new Parquet file in Raven tagged new data and commits, so the data is visible at once. Readers reconstruct the OTF snapshot and union in the new data files. Every k transactions, a compaction step writes proper Iceberg or Delta metadata for the accumulated files and flips their tags to data, inside the same OCC protocol. If the compaction loses a conflict, its OTF files were never visible and can be dropped safely. The cost is that new data files have no statistics in the metadata store, so they can’t be pruned and every scan reads them.

Multi-table transactions. The client opens a transaction over a set of tables, reads all their snapshots in one DB transaction, writes data and OTF files for each, and commits everything as one serializable transaction that acts as a multi-row CAS. Raven validates only the tables a transaction writes, so it detects write-write conflicts and allows write skew. That gives snapshot isolation, not serializability. Orphan files from a client that crashes mid-transaction aren’t handled yet; the authors sketch a fix based on pre-allocated, session-referenced file names.

Reachability GC. This is the most interesting part of the design. Relationships can be declared referential and entity types declared roots:

ALTER RELATIONSHIP Contains (Database, Asset) SET Database REFERENCES Asset;
ALTER ENTITY Workspace SET ROOT;
ALTER ENTITY Session SET ROOT;
ALTER RELATIONSHIP Contains (Database, Table) SET RETENTION INTERVAL '7 days';

An entity is alive if a path of references, each either active or deleted-but-still-in-retention, leads to it from a root. Sessions are roots, so a long ETL job pins its snapshot by creating a session that references it, kept alive by a lease and heartbeat. Dropping a table makes its references “ghosts” for the retention window, so an accidental drop can be undone. GC uses reachability and retention; neither one is safe alone.

The collector doesn’t scan Postgres. The metadata store appends reference additions and deletions to edge-log tables. A dedicated machine ingests them each epoch into a RocksDB checkpoint and an in-memory graph engine, which reference-counts and walks outward from newly unreferenced vertices rather than tracing from the roots. Files are tracked per FILELIST history rather than individually: deleting a version decrements a global per-file count for files whose lifetime fell between the neighbouring live versions, and background threads delete files whose count reaches zero. That keeps the in-memory graph to tens of millions of catalog objects instead of billions of file rows. Reference counting can’t collect cycles, so Raven rejects DDL that would create a referential cycle; lakehouse references are naturally acyclic, so in practice this costs little.

Results

All experiments run on four CloudLab machines (client, catalog, metadata store, GC) with data on S3 us-east-2. The clients are Iceberg 1.10 with DuckDB 1.4.4, and Delta Kernel 4.0.

Figure 7 of the paper: throughput and files created per transaction versus compaction interval for Iceberg and Delta Lake, with and without Raven. Figure: Extended MoR on single-threaded TPC-DS store_sales inserts. Throughput rises and metadata files per transaction fall as the compaction interval grows; red is the plain OTF baseline. Source: arXiv:2611.15975, Figure 7.

Small commits. At a compaction interval of 10, Iceberg throughput goes from 87 to 140 txn/min (+61%) and Delta from 40 to 62 (+55%). Files created per transaction drop from 4 to 1.33 for Iceberg and from 3.1 to 1.23 for Delta. With an interval of 2 Raven is slower than the baseline, because it does both kinds of work on every insert, so the benefit depends on batching enough appends between compactions. The gain shrinks as data files grow but is still 37% (Iceberg) and 31% (Delta) at ~9 MB files. Read latency with seven unmerged files is close to the baseline for Iceberg. The Delta read results were inconclusive because Delta Kernel’s lazy log fetching dominates.

Multi-table commits. Against Iceberg and Delta’s sequential single-table commits, Raven’s multi-table path adds latency, mostly from per-table RPCs to load metadata (176 ms load + 279 ms commit across six Iceberg tables). Batching those RPCs is left to future work. Against DuckLake on a catalog-only microbenchmark (1–5 random tables out of 1,000 per transaction), Raven scales from 23.6 txn/s at 10 threads to 146.4 at 90. DuckLake peaks at 25.4 txn/s at 40 threads and reaches a 41.6% abort rate at 90, because it serializes creation of a global catalog snapshot whether or not transactions conflict. Under heavy Zipfian skew Raven falls back toward DuckLake’s level, as table-level OCC would predict.

Figure 15 of the paper: catalog storage in GB and single-row query latency versus column count, Raven with Iceberg against DuckLake. Figure: Catalog footprint and point-query latency as column count grows on a 100,000-file table. DuckLake’s statistics live in Postgres; Raven’s stay in Iceberg manifests. Source: arXiv:2611.15975, Figure 15.

Statistics. At 320 columns DuckLake’s Postgres reaches 3.27 GB while Raven stays below 140 MB. Point-query latency rises from 1.55 s to 24.17 s for DuckLake, which scans its statistics table, and from 1.82 s to 2.38 s for Raven, which lets Iceberg prune manifests in parallel.

GC. With 60 threads doing reads, inserts and compactions across 1,000 tables, throughput with GC on is indistinguishable from GC off for read-heavy, balanced and write-heavy mixes. Expiring snapshots across 300,000 files, the collector deletes 2,183.6 files/s at the shortest client pause before S3 throttling (3,500 DELETE/s per prefix) kicks in, and 3,256.8 files/s on a local filesystem. So the S3 API, not the graph engine, is the bottleneck.

Limitations

The authors are open about the scope. Access control, especially row-level policy enforced across engines, is unaddressed. Scale testing stops well short of a large enterprise, and the paper says a deployment at that size would need a distributed transactional store (FoundationDB, Spanner) rather than one Postgres, which brings back some of the operational weight the design avoids. Extended MoR currently covers appends only. Multi-table transactions give snapshot isolation, not serializability, and still leak orphan files on client crash. The whole protocol assumes cooperative clients: every engine has to go through Raven, and an engine that reads the OTF directly sees an older snapshot without the new data files and has no protection from GC.

Why this matters

If you run Iceberg or Delta in production, the four failure modes are familiar: metadata file explosions from streaming ingest, slow commits on S3, cross-table consistency handled by convention, and VACUUM incidents. The usual fixes have been compaction jobs, longer retention, and vendor catalogs that solve some of these inside one engine (the paper notes Databricks’ multi-table transactions are limited to its own compute as of September 2026). Raven shows how to fix all four at once in an open catalog, with one design decision behind all of them: put visibility and references in a database, and leave statistics where the engines already read them.

The reachability-GC model may outlast the rest of the paper. “Sessions are roots, drops are ghosts with a TTL” is a simple and correct rule for when a file can be deleted, and it would carry over to any catalog that tracks references explicitly, including Unity Catalog and Polaris as they add non-table assets like models and volumes. Raven’s use of the ER model also makes ML models first-class assets with the same lifecycle as tables, which is where catalogs are heading anyway.

Read alongside

  • Armbrust et al., Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores (VLDB 2020): the log-based OTF design Raven integrates with.
  • Edara and Pasumansky, Big Metadata: When Metadata is Big Data (VLDB 2021): BigQuery’s experience that file statistics outgrow a transactional store.
  • Oh, Nilangekar, Deshpande, TreeCat: Standalone Catalog Engine for Large Data Systems (VLDB 2025): the same group’s all-in-database predecessor.
  • Götz et al., Interoperable ACID Transactions for Open Table Formats (VLDB 2026): LakeVilla, the marker-file alternative for cross-table transactions.
  • Jain et al., Analyzing and Comparing Lakehouse Storage Systems (CIDR 2023): the measurements behind the commit-latency complaint.
  • DuckLake (DuckDB Foundation, 2025–26): the database-only design Raven benchmarks against.

📄 arXiv abstract · 📄 PDF · 💾 umddb/raven


Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint; figures cropped from the arXiv PDF and reproduced here under fair use for educational commentary.