Andy Pavlo opens CMU’s database systems course with a claim that reframes most of what practitioners argue about online: the problems are not new. The hardware changed; the problems didn’t.
This week’s Classic of the Week is his Lecture 1 — course information plus a compressed history of database systems from the 1960s to the present. It’s the single best hour for understanding why databases look the way they do, and it comes with an implicit warning: if you don’t know this history, you will spend your career re-litigating settled debates.
The one skill industry actually asks for
Before the history, Pavlo relays what he hears when he asks industry people what they want from a graduate of this course. He expected answers like latch-free data structures or recovery protocols. He gets the same answer every time:
“They want a student that is capable of working on a large code base independently.”
His reasoning is that the specific internals are downstream of that skill. If you can drop into an unfamiliar codebase and figure out what’s going on without repeatedly interrupting your colleagues, whatever they need you to build in their database system becomes tractable.
The course is structured accordingly — documentation, test cases, and mandatory peer code reviews on the third project, on a codebase students didn’t write.
The hierarchical model: pointers all the way down
The historical arc begins in the 1960s with IBM’s IMS and the hierarchical model, built for exactly the kind of workload the Apollo program had: track a million parts, know who supplies them and how much is in stock.
The schema is two collections — supplier and part — arranged hierarchically, so every part belongs to a supplier. Instantiated, it looks superficially like a table with tuples, but each supplier record carries an extra attribute pointing at a separate data structure holding its parts. Querying means writing a for-loop to find the supplier, then iterating its parts.
Two problems follow immediately:
- Duplicate data. The same battery sold by three suppliers is stored three times.
- No data independence. The logical schema and the physical storage layout are welded together. Change one, break the other.
CODASYL: the network model
The network model (CODASYL) tried to fix duplication by allowing cross-reference sets. The supplier/part example gains a third collection, supply, and the system automatically maintains the linking sets between them.
It’s better on redundancy and worse on everything else:
- Queries are complex and unnatural. Answering “which parts does this supplier supply?” means nested for-loops traversing down the network and back up.
- Corruption is catastrophic. The cross-reference sets are the access path. Lose one to a disk head crash, and there is no longer any way to get from a supplier to its parts. The data is intact and unreachable.
Pavlo pauses to poll the room on how many students have heard of CODASYL. One, maybe two hands.
“Clearly we know who won, because you know what this one is but not this one.”
Codd, System R, Ingres, and the accidental birth of Oracle
The relational model arrives in the early 1970s, and the two foundational implementations follow: IBM’s System R and Berkeley’s Ingres.
The Oracle origin story here is worth the lecture on its own. Pavlo relays it from System R people still around: Larry Ellison would call the researchers, ask what output their database produced for a given input and what error codes came back — and the researchers, being academics with no instinct for IP protection, simply told him. He’d then go implement the same thing.
IBM never commercialized System R directly; DB2 arrived roughly a decade later, in 1983. Stonebraker commercialized Ingres, then left for Berkeley and started Postgres — which, as Pavlo notes, is named for exactly what it is: post-Ingres.
By the 1980s, the relational model had clearly won.
Object databases: the detour that got absorbed
The late 1980s and early 1990s brought object-oriented databases, riding the C++ and Smalltalk wave. The pitch was that storing objects in relational tables was a mismatch; you should write your C++ object straight to the database.
Pavlo can’t name a single one from memory, and that’s the point. They never took off, and the reason was standardization: there was no SQL equivalent. Every object database shipped its own API or its own query language, so adoption never compounded.
What’s notable is that the ideas survived. Much of what those systems pioneered now exists inside relational databases as added type and structure support — a recurring pattern in this history.
NoSQL: CODASYL’s arguments, thirty years later
The mid-2000s NoSQL wave — Cassandra, HBase, Riak, Couchbase — declared SQL slow and the relational model an obstacle.
Pavlo’s assessment is blunt: those are substantially the same arguments the CODASYL camp made in 1972–73, and they lost then. With the exception of MongoDB and things like Redis, the NoSQL systems have all added SQL support back.
The real source of their performance advantage wasn’t dropping SQL. It was dropping ACID transactions. That’s a genuine engineering tradeoff, and it got conflated with a query-language argument that didn’t hold up.
He also flags shared-nothing as the key architectural distinction of the era: each node acts independently and stores a partition of the data — as opposed to shared-disk (everyone writes to the same location) or shared-memory (everything on one box).
Good ideas that were wrong until they weren’t
The lecture’s most useful frame is that many 1970s ideas were correctly abandoned for their hardware era and correctly revived decades later:
- Query compilation — tried early, too expensive then, standard now.
- Denormalization — proposed in the first IBM system, judged a bad idea, then reintroduced at scale by Google thirty years on.
Neither idea was wrong. Both were wrong for the hardware available. This is the strongest argument for reading old database papers: the design space was explored thoroughly, and shifts in cost ratios keep making abandoned branches viable again.
Where innovation comes from
Pavlo closes on the industry/academia dynamic, which he considers unusually healthy in databases relative to other CS subfields. Ideas typically originate in academia as small prototypes, but don’t become real until a major company picks them up and builds them into a production system.
He names IBM as the vanguard of the 1970s and 1980s — System R set the template for how you build a relational system — and Google as the modern equivalent. Not because everything Google does is best, but because Spanner, F1, and Mesa get the most traction and the most eyeballs. Startups do genuinely interesting work too, he notes, but rarely publish papers, so you’re reading blog posts instead.
Key takeaways
- The problems are old; the hardware is new. Most modern database debates re-run arguments settled decades ago under different cost assumptions.
- The hierarchical model failed on duplicate data and the absence of data independence — logical schema welded to physical layout.
- CODASYL’s network model made queries unnatural and databases fragile; losing a cross-reference set orphaned reachable data permanently.
- The relational model won on data independence and a standard query language, not merely on expressiveness.
- Standardization is the moat. Object databases had good ideas and no SQL equivalent, so they died and were absorbed.
- NoSQL’s performance came from dropping ACID, not from dropping SQL — and nearly all of them added SQL back.
- Abandoned ideas deserve periodic re-evaluation. Query compilation and denormalization were both correct decisions to abandon, then correct to revive.
- The most-requested industry skill is working independently in a large unfamiliar codebase, ahead of any specific database internal.
Source
- Title: L01 — Course Information & History of Databases (CMU Database Systems)
- Speaker: Andy Pavlo
- Origin: CMU Database Group — 15-445/645 Database Systems
- Duration: ~71 minutes
- URL: https://www.youtube.com/watch?v=UGMLKsma_VU