Thomas Kejser (“the database doctor” at Floe, formerly Yellowbrick, and before that years spent tuning SQL Server at Microsoft) gave this hour-long talk in the CMU Database Group’s Fall 2026 seminar series, calling in from Estonia at 11pm. His subject is FloeScan, Floe’s storage engine. It’s designed around a constraint most lakehouse engines get around by copying data: customers want their data to stay in Iceberg, and they want queries against it to behave like queries against a real database.
The design that results is unusual. It doesn’t ETL into a “gold layer” and it doesn’t invent a new table format. It leaves Iceberg alone, writes its own metadata next to it, and caches aggressively on local NVMe. Andy Pavlo and the audience interrupt often, which helps: the Q&A covers the parts a slide deck usually skips.
What Iceberg actually gives you
Kejser starts with a quick tour so the later design choices make sense. An Iceberg table is entirely files: Parquet for data, Avro manifest files pointing at the Parquet, manifest lists so manifests can be reused across snapshots, and a metadata JSON file at the top that names the current and historical snapshots. Object stores can’t do atomic swaps, so a REST catalog is needed for commits after all. Writers upload files out of band, then race to commit. Losers retry, which caps throughput at “somewhere south of 1,000 commits per second”, though each commit can be huge.
His blog post on this is titled “Iceberg: the wrong spec, the right idea,” and the talk explains both halves. The spec is portable and easy to parse, and it’s becoming the default precisely because nobody wants vendor lock-in.
“Every 10 years or so we have collective amnesia and we all forget everything we know about databases and we sort of start over again.”
Object storage isn’t as scalable as advertised
Lakehouse marketing assumes object storage is infinitely scalable. Kejser’s benchmarks on an i8g.48xlarge (192 vCPUs, 100 Gbit NIC, 256 threads) say otherwise. The best client library, AWS CRT, peaks around 8.8 GiB/s, which doesn’t saturate the NIC, and keeps 10 to 12 cores busy just issuing I/O. Every request is HTTP that has to be serialised and parsed. Separating storage from compute has a cost, and it’s paid in CPU.
Local NVMe is the comparison. Tuned io_uring with polling reaches about 28 GB/s on four cores, against roughly 13 cores for 8.8 GB/s from S3. Floe’s user-space NVMe driver, carried over from Yellowbrick, cuts CPU by another two to three times. Cloud NVMe is ephemeral and the machine can disappear at any time, so the design has to treat it purely as a cache. EBS is expensive and only replicates within one availability zone, so Floe keeps it for things like metadata stores and uses as little of it as possible. The resulting rule: S3 for anything that must survive, NVMe for anything you want to read fast, and minimal EBS.
Why Parquet-on-Iceberg is slow to query
The metadata problems are concrete. Min/max statistics are optional and depend on which client wrote the file: older DuckDB writes them at row-group level but not page level, Spark writes both. So last year’s data can be slower than this year’s because a different tool wrote it. Bloom filters exist in recent specs but aren’t guaranteed. Iceberg’s own statistics barely go beyond row counts (Puffin files are early), which makes query planning hard.
Parquet’s layout adds more. Compressed pages are unaligned across columns, so reading rows 0 to N from columns A and B means working out where each column’s page boundaries fall, and Parquet doesn’t record that cleanly. Cloud data engineering also tends to denormalise heavily because Spark joins are slow, and Kejser has seen customers ask for 10,000- and 20,000-column tables. Footers grow accordingly, so reading three columns means parsing a footer that describes thousands.
“Cache, don’t copy”
The usual answer is to ETL out of Iceberg into a delivery layer (a “gold layer”, or as Kejser puts it, what we used to call a data mart). Floe goes the other way and queries Iceberg directly, adding indexes and caches beside it. Part of the motivation is agents. Kejser says Floe’s customers report that 50 to 60% of their queries already come from agents, and you don’t want to build a data mart per agent; you want agents exploring the lake directly with something to speed them up. The stated goals are needle-in-a-haystack lookups at more than 10K QPS and under 10 ms, far fewer S3 requests, and streaming through petabytes with minimal I/O.
FloeScan is also separable from Floe’s own engine. There’s a DuckDB extension (LOAD floescan), an experimental Trino connector, and the client API will be open-sourced, which in effect turns the storage engine into a separate service sitting over Iceberg.
The architecture: ingester, sidecars, lookup, client
Iceberg has no commit hook, so an ingester service polls for new snapshots. For each one it reads the new Parquet files (embarrassingly parallel, several threads can even share a file) and writes sidecars, which are extra files in S3 alongside the data. S3 is cheap and replicated, so the sidecars survive anything and everything else can be rebuilt from them. There are three kinds:
- Index data maps a predicate to a set of rows, never pages. It supports min/max trees for sorted columns, hash-bloom trees for equality on sparse or small values, and inverted string indexes.
- X-rays (Kejser admits they should probably have been called leaves) map columns and row ranges to the byte offsets to fetch. They work like compressed, stripped-down Parquet footers.
- Statistics sidecars hold Apache DataSketches, which merge, so per-file sketches can be combined incrementally as data arrives.
The lookup service holds the index trees in memory and answers predicate queries. The client sits next to the compute in the same data center, owns an NVMe cache and does the actual I/O.
256K chunks and row-level intersection
The core of the design deals with a mismatch. Queries intersect and union rows, but object storage is cheapest per CPU cycle at about 256 KB reads. During ingest, each column’s pages are grouped into chunks of at least 256 KB, each labelled with its min and max row number. Pages stay unaligned, but because every chunk knows its row range, a chunk of column A can be intersected with a chunk of column B. Long Iceberg file paths are replaced by sequential 64-bit file IDs from an atomic counter, and the sequence turns out to matter later.
Index trees are partitioned by file ID across lookup nodes, which makes building them almost lock-free. For WHERE A = 42 AND B = 'abc', the lookup service takes the matching (file, rows) sets from each index and intersects them. Partitioning means only matching subtrees need intersecting, never one huge set. During the merge it also coalesces adjacent reads into larger requests. Min/max trees use range partitioning by file ID instead of hashing, because files written next to each other in time tend to have overlapping ranges, which is where the sequential IDs pay off.
Why the leaves live on the client
The lookup service returns a compact stream of files and rows, not pages. Mapping rows to byte offsets, and especially projecting the other columns of a 10,000-column table, would make the lookup service’s in-memory index enormous. The memory-heavy servers would also end up doing expensive S3 calls. So the X-rays live on the clients, which are numerous and have the NICs, CPU and NVMe to spare. Kejser’s summary: “Object storage access sucks! Share the pain.”
That gives a storage hierarchy: index pages in lookup-server memory, X-rays and transcoded data on client NVMe, and S3 behind both. Best case, with data and leaf already cached, a lookup is one network round trip. Worst case is three: index lookup, X-ray fetch, 256K block read. That’s how they reach sub-10 ms point lookups. Data is transcoded out of Parquet into a DMA-friendly format (Floe is now evaluating Vortex for this), but only after a file has been seen more than once. Scan resistance matters, and plain LRU “really hurts you.”
Keeping up with Iceberg: immutable trees and eventual commit visibility
Iceberg compaction can rewrite petabytes and invalidate an index, and in-place updates under heavy read concurrency mean lock contention. Kejser’s answer is brute force. Trees are immutable and double-buffered, rebuilt from sidecars and swapped atomically, with subtrees taking turns so memory doesn’t double. Between rebuilds, a roaring “authority bitmap” over the sequential file IDs records which files the current tree covers. Files outside it go back to the client as “fetch all rows, filter yourself”, a small brute-force tail that works much like a row-store delta in a columnar database. After a crash the bitmap is rebuilt from the sidecars.
New snapshots pass through a lockstep pipeline. Iceberg commits, then Floe ingests and builds sidecars, updates the lookup trees, and finally its catalog (FloeCat, open source) tells clients the snapshot is visible. Floe clients therefore trail plain Iceberg clients, with a target of five seconds. Kejser calls this “eventual commit visibility,” by analogy with eventual consistency. Time travel complicates it: a query AS OF TIMESTAMP '2019-01-01' can require ingesting an old snapshot that shares little with the current one, so each stage has queues, back-pressure and workload management to protect the SLA.
Asked why build on Iceberg at all given these problems, Kejser says it’s simply table stakes. Large Yellowbrick customers kept requiring it. Pavlo agrees, comparing it to branching becoming a required feature for OLTP databases.
Key takeaways
- Object storage throughput is limited by CPU spent on HTTP. Even the best S3 client didn’t saturate a 100 Gbit NIC and kept 10 to 12 cores busy, while tuned NVMe did three times the bandwidth on four cores.
- Iceberg’s portability comes with weak, writer-dependent metadata, so an engine on top can’t assume min/max values, blooms or page alignment and should build its own.
- Writing derived metadata as S3 sidecars keeps Iceberg untouched and makes every in-memory or NVMe structure rebuildable after a crash.
- Grouping pages into roughly 256 KB chunks labelled with row ranges lets an engine intersect predicates at row level while still issuing object-store-sized reads.
- Sequential file IDs are a small choice that pays off three times: compact roaring bitmaps, range-partitioned min/max trees, and cheap hash partitioning.
- Keep the memory-heavy index service away from S3 I/O and push the row-to-offset leaves to the many co-located clients that have NICs and NVMe to spare.
- Immutable, double-buffered index trees plus an authority bitmap for not-yet-indexed files avoid lock contention, and Iceberg compactions just trigger rebuilds.
- A few seconds of commit visibility lag behind Iceberg is a reasonable price for indexes that stay consistent with a known snapshot.
Source
- Talk: Inside the Guts of FloeScan and other Interesting Design Magic
- Speaker: Thomas Kejser, Floe
- Event: CMU Database Group seminar series, Fall 2026 (19 October 2026), hosted by Andy Pavlo
- Length: 1 h 04 min
- Seminar page: https://db.cs.cmu.edu/seminars/fall2026/
- Video: https://www.youtube.com/watch?v=HTHzZco2jIw