Ippokratis “Ippo” Pandis did his PhD in the CMU database group, worked on DB2 BLU at IBM, Impala at Cloudera, spent ten years on Redshift at AWS, and is now a distinguished engineer at Databricks. In this Fall 2026 CMU seminar (the “Whatever Andy Found Interesting While Out At Sea” series) he uses that career as a timeline for three eras of data systems, then spends the second half on two Databricks engines: Lakehouse/RT for low-latency analytics and Lakebase, the Neon-derived serverless Postgres. The last part shows the two meeting, with the analytics engine reading pages from Postgres storage without going through Postgres.
It’s a vendor talk and the benchmarks are anonymised and Databricks-run, so treat those numbers accordingly. The architecture is the reason to watch, and in particular the argument that separating storage from compute in OLTP is what makes real-time analytics on live transactional data possible.
Three golden ages
Pandis frames the first era as monolithic single-node RDBMSs (Oracle, DB2, SQL Server, plus Teradata and Netezza for analytics and MySQL and Postgres as the open-source pair). Hardware was scarce, engineers counted cycles and kilobytes, and shipping meant pressing a hardened CD six months after code freeze. The cow book and the red book encode what that era learned, and most intro database courses still teach it.
The second era had two drivers. Big data brought open columnar formats (Parquet and ORC, which follow the PAX layout from his advisor Natassa Ailamaki) and open engines (Hive, Impala, Presto, Spark) running on large, unreliable commodity clusters. He shows DeWitt and Stonebraker’s “MapReduce: a major step backwards” post as a reminder that the database community didn’t like much of it. What did change, he says, is that engineers and customers started believing big data had value.
The cloud was the other driver, and the hardware fact he stresses is the network. The best EC2 instance in 2015 gave about 10 Gbps; by 2023 the top instances reached 3.2 Tbps, and he says current GPU instances advertise 6.4 Tbps. The classic textbooks weren’t written for that machine. Cheap bandwidth is what made separate storage and compute and open formats on object storage practical, which in turn produced the data lake and then the lakehouse.
How Databricks SQL is put together
The baseline is the Databricks SQL warehouse. It starts with Spark’s driver/executor split and Spark SQL on top. Databricks added a proxy in front and Unity Catalog for cataloguing and governance, and replaced the Scala operators inside executors with Photon, a native vectorised C++ engine. Adaptive query execution, native shuffle and IO, and autoscaling sit on top. Pandis shows the product growing by roughly three orders of magnitude over five years as evidence that one copy of data in open formats can serve both BI and ML.
Agents hit the one-second wall
Then “the barbarians came”, meaning agents. Warehouse consumers used to be people and BI dashboards emitting one giant join. Agents bring far more concurrent sessions, and the queries come in two kinds: lots of shallow exploratory queries run in parallel to gather context, and long, verbose, badly written SQL.
Every major cloud warehouse kept getting cheaper year over year, he says, but they all hit a latency floor around one second, and his slide is a collage of user complaints about each one. The accepted fix was to copy a slice of the warehouse into a real-time serving system like ClickHouse, Pinot or StarRocks. That gives you sub-100 ms queries on data that is no longer fresh, another pipeline, and a second copy to govern. He argues the governance cost is the one agents make worst.
Lakehouse/RT
The question Databricks asked was whether you could get serving-layer latency on the one governed copy in cloud storage. Their answer is a new engine, Lakehouse/RT (the benchmark charts label it Reyden, an internal name Pandis glosses as “Reynold’s dream engine”, after co-founder Reynold Xin).
They kept Photon’s vectorised operators because they were fast and battle-tested, and rewrote everything around them in Rust: driver, executor shell, proxy, the IO path, and how the engine talks to the catalog. Metadata refresh became asynchronous and they leaned on lightweight encodings for both storage and shuffle. Once you aim for milliseconds, he says, a couple of extra network hops to the catalog become a visible cost.
In Q&A someone asks whether executors cache data on SSD in a different encoding. They don’t. With Parquet v2 and lightweight encodings they process the data as stored, and what they cache aggressively is file listings and metadata, refreshed in the background so object-store round trips stay off the query path.
“It takes between five and seven years to build a robust database engine from scratch. I think that Databricks was able to build that within one year and a half.”
He attributes the roughly 3.5x speed-up to coding agents and offers it as what teams should now expect.
The benchmarks: on TPC-H Q6 under rising concurrency, one vendor’s latency climbs steeply after 200–300 concurrent users, another holds until about 1,000 and then fails, and Lakehouse/RT is shown handling tens of thousands of concurrent queries at single-digit milliseconds. Across TPC-H scale factors it’s fastest at every size, while the competitor’s “interactive” tier stops completing queries once they exceed a 5-second quota. On TPC-DS the nearest competitor is about 3x slower and another 15–20x slower before it fails. The point he wants you to take is that one engine covers both small, fast queries and large, join-heavy ones, where other vendors sell separate products for each.
Agents changed who creates databases
For transactions Pandis brings data from Neon (acquired by Databricks) and from the Lakebase paper they presented at VLDB. Agents now create most new databases on Neon and Lakebase, more than 80% by his figure. The number of human-created databases is also rising, driven by less technical users on Replit- and Lovable-style app builders. Compute lifetimes look nothing like an Oracle install: a large share of computes live under ten seconds or under an hour, and branch trees go 15–20 levels deep as agents try a change, go deeper, and back out. Instant branching, fast spin-up and cheap rollback become baseline requirements.
Inside Lakebase
Lakebase is the Neon design. Compute is close-to-unmodified Postgres in single-tenant VMs. Storage is a multi-tenant service with three parts:
- Safekeepers receive WAL at commit through a Paxos-style quorum, three of them (one per AZ), and acknowledge once it’s durable on their SSDs.
- Pageservers consume the WAL, reorder it by page and LSN, materialise pages, cache them on local SSD, and upload page images and WAL to object storage.
- Object storage holds everything as immutable, append-only data.
Postgres’s IO calls are intercepted so a page read goes to a pageserver rather than local disk. The key API is “give me page P as of LSN L”, which the rest of the talk depends on.
Locality keeps it fast: over 98.5% of reads hit the compute-side SSD cache, nearly all of the rest are served by pageservers, and only a tiny fraction go to object storage. Pandis shows Lakebase doing about 600,000 TPC-C new-order transactions per minute at single-digit-millisecond P90, well ahead of two unnamed cloud Postgres offerings. He also claims it’s the first fully managed database that can fail over across clouds, not just across AZs or regions, with branches used to replay state on the DR side.
Analytics on live Postgres pages
The last question is how to run real-time analytics on the live transactional data without an ETL hop. Vanilla Postgres, as on RDS, is monolithic: one instance owns storage and compute, so there’s nowhere to attach another engine. The Lakebase split gives you that place.
Lakehouse/RT plans the query and then makes one cheap call to the primary Postgres compute for the catalog, stats, an MVCC snapshot (XID information) and a current LSN. Pandis says that ping can run at tens to hundreds of thousands per second. The executors then skip Postgres and call GetPage@LSN on the pageservers directly, convert heap pages to Arrow, apply MVCC visibility filtering, and run Photon on the result. The read is transactionally consistent and puts almost no load on the OLTP primary.
On TPC-H SF10, Postgres on Lakebase takes about 94 units on his chart, and Lakehouse/RT over the same live Lakebase data takes about 9, roughly 10x faster and within an order of magnitude of Lakehouse/RT on Delta. Heap pages are row-oriented, so closing that remaining gap is the open problem.
His proposed blueprint, which he wants to work out with the wider community, has the OLTP storage layer periodically settle into open table formats (Iceberg/Delta snapshots at rest, behind a REST catalog), plus a “diff API” that returns the recent changes on top. Any external engine could then read the columnar snapshot and merge in the live delta.
Q&A worth hearing
- Native code targets both x86 and ARM with hand-written intrinsics in hot loops. AVX-512 code paths exist but so far haven’t delivered much over AVX2.
- Lakehouse/RT can’t yet use Postgres secondary indexes. Pandis calls it an interesting direction for needle-in-a-haystack queries.
- Pavlo asks whether customers need real Postgres or would accept a new architecture. Pandis’s view is that agents make SQL dialects and semantics a much smaller adoption barrier than they were, since transpiling, say, a Snowflake workload is now cheap.
Key takeaways
- Network bandwidth in the cloud grew more than 300x in a decade, and that, not storage or CPU, is what made disaggregated storage and open formats viable.
- Agent workloads mean many more concurrent sessions and a mix of tiny exploratory queries and huge sloppy ones, which pushed every cloud warehouse into a roughly one-second latency floor.
- Databricks kept its proven vectorised operators (Photon) and rewrote the surrounding driver, proxy, IO and catalog paths in Rust to get to milliseconds, which shows how much latency lives outside the operators.
- Agents create most new databases on Neon/Lakebase, and many computes live under a minute, so instant branching and fast spin-up are now core OLTP features.
- A page-at-LSN API on disaggregated Postgres storage lets a separate engine do consistent reads of live data after one lightweight snapshot call to the primary.
- That gives about 10x over Postgres itself on TPC-H SF10. The remaining gap to columnar Delta comes from reading row-oriented heap pages.
- Pandis’s proposed end state is OLTP storage that writes open-table-format snapshots plus a diff API, so every engine reads one copy.
- All benchmark numbers are vendor-run with anonymised competitors, so judge the architecture on its own merits.
Source
- Talk: Lakebases, Lakehouses, and LTAP: Unifying Transactions and Analytics on Open Storage
- Speaker: Dr. Ippokratis Pandis, Distinguished Engineer, Databricks; hosted by Andy Pavlo
- Event: CMU Database Group seminar, “Whatever Andy Found Interesting While Out At Sea” (Fall 2026), 26 October 2026
- Length: 52 min
- Video: https://www.youtube.com/watch?v=xg8qksWY2eY