The Google File System (2003)

Seminal Paper of the Week — the paper that quietly defined what “cloud storage” looks like from the inside.

Authors: Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung (Google) Published: SOSP ‘03 — 19th ACM Symposium on Operating Systems Principles, October 2003. Canonical link: The Google File System (Google research mirror) · ACM DOI 10.1145/945445.945450

TL;DR

In 2003, Ghemawat, Gobioff and Leung described how Google was running a multi-thousand-node, petabyte-scale distributed file system on commodity hardware — and how the design assumptions diverged so sharply from the established POSIX-file-system lineage that almost every architectural decision in the paper looks like a heresy until you read the workload section. Files are huge (multi-GB, billions of files would be the norm). Failures are continuous rather than exceptional. Writes are dominated by appends rather than random overwrites. Sequential reads dominate random ones. Under those assumptions the paper proposes:

• A single logical master that holds all metadata in memory (filename → 64 MB chunk handles → chunkserver replica locations).
• 64 MB chunks as the unit of placement and replication — 100× larger than typical Unix block sizes of the era.
• Three replicas by default stored on three different commodity Linux chunkservers, with the master orchestrating placement, re-replication, and rebalancing.
• A strict control-plane / data-plane separation: clients talk to the master only for metadata, then read and write bulk data directly to chunkservers. The master never touches file bytes.
• A lease + serial-order mutation protocol that lets the master step out of the write critical path while still giving the same byte order to all replicas.
• A new record append primitive that guarantees at-least-once atomic append into a file shared by many concurrent writers — the operation MapReduce, log aggregators, and producer/consumer queues turn out to actually want, and the one POSIX never provided.

The paper became foundational not because the implementation was particularly elegant — it cheerfully describes itself as workload-specific — but because it crystallised the assumption set that distinguishes warehouse-scale storage from filesystems-as-we-knew-them, and because almost every subsequent system at Google (Bigtable, MapReduce, Chubby’s snapshotting, the original Colossus design) was built directly on top of it.

Why the paper still matters

Twenty-three years later, GFS is gone — Google replaced it with Colossus around 2010, which broke up the single master into a sharded metadata service and adopted Reed-Solomon coding instead of triple replication. So why does this paper still get assigned in every distributed systems syllabus?

Because the three load-bearing ideas turn out to be civilisation-grade:

1. Workload-first design. GFS doesn’t try to be a general-purpose file system. It asks “what does our actual workload look like?” and then deliberately throws away guarantees the workload doesn’t need (small-file efficiency, POSIX semantics, low metadata-operation latency for random updates) in exchange for the ones it does (sequential throughput, atomic appends, cheap replication). This stance — “the workload is the spec” — is now the default for every successful warehouse-scale system. The same stance produced Dynamo (high availability over consistency), Spanner (external consistency over latency), HDFS (immutability over update), Bigtable (sorted by key over relational generality), and the modern object stores.
2. Control plane / data plane separation. Clients consult the master once per chunk to learn where the data is, then bulk traffic goes directly between client and chunkserver. The master is small, in-memory, in-RAM-fast, and never the bandwidth bottleneck. Every modern blob store works this way. S3’s frontend → backend split, Azure Blob’s namespace-vs-stream layer, HDFS NameNode vs DataNodes, Ceph’s MON+OSD split, Colossus’s metadata D-shard vs storage shard — all are direct intellectual descendants of the GFS master/chunkserver split.
3. Centralized coordination is fine if you keep it off the data path. GFS proves that a single master, in 2003, can scale to thousands of chunkservers and petabytes of data because metadata ops are six to seven orders of magnitude rarer than data bytes. The contemporaneous fashion was peer-to-peer decentralization (Chord, Pastry, CAN); GFS quietly demonstrated that the engineering and operational complexity of decentralization wasn’t actually justified if you scoped the master’s responsibilities correctly. This is the unstated foundation of Spanner, Bigtable, Colossus, and basically every centrally-coordinated warehouse-scale storage system since.

The setup

The setup section of the paper is the part most worth re-reading even if you remember the rest. It establishes the assumption set explicitly, and the system design follows almost mechanically:

• Component failures are the norm. Thousands of commodity disks and machines mean disks fail every day, machines fail every week, network partitions happen routinely. Recovery, replication, and detection have to be ambient behavior, not exception handling.
• Files are huge. Multi-GB files are common; billion-file directories are not. A 64 MB chunk size means a 1 TB file is 16 K chunks of metadata, not 250 M of inode-equivalents.
• Most mutations append. The dominant write pattern from MapReduce and log-style producers is “open file, append, close” or “many writers all appending to one shared file.” Random overwrites in the middle of a file are vanishingly rare.
• Most reads are large and sequential. Either streaming the whole file from start to end, or a large bounded range scan. Random small reads happen but are not the optimization target.
• Co-designing application and FS is allowed. Google owns both the consumers and the storage layer, so they can change the API. They add record append. They relax consistency to “defined but not guaranteed identical across replicas, with applications expected to handle duplicates idempotently.” A vendor file system can’t do this; an in-house one can, and that freedom is half the story.

From this assumption set, every subsequent design decision pre-resolves itself: the chunk size, the replication factor, the metadata-in-memory master, the lease-based mutation ordering, the absence of client-side caching for data (only for metadata), the relaxed consistency model.

The architecture

A GFS cluster has three kinds of process: one master, many chunkservers, many clients linked in as a library.

The control / data split is the load-bearing piece. A client read works like this:

1. Client asks Master: “where is chunk N of file F?”
2. Master returns the chunk handle and the list of chunkservers holding replicas, plus their version numbers.
3. Client caches that mapping for a while and pulls bytes directly from one of the replicas (typically the nearest by network topology).
4. The Master is no longer involved.

A client write to chunk c works like this:

1. Client asks the master who holds the lease for c. If nobody does, the master grants one to a replica it picks.
2. Master returns the primary’s identity and the replica list.
3. Client pushes the data (the bytes being written) to all replicas in a pipeline — the data flows along the network topology, not necessarily through the primary first. This decouples the data flow from the control flow.
4. Once all replicas have the data buffered, the client sends a write request to the primary.
5. The primary assigns a serial number to the mutation, applies it locally, and forwards the request (with the assigned serial number) to all secondaries. Secondaries apply in serial-number order, ack the primary.
6. Primary acks the client.

The lease + serial-number scheme is what lets the master stay out of the write path. The master picks the primary; the primary picks the order; all replicas observe the same order. A single lease is granted for a bounded time (60 s) and renewed implicitly while writes continue, so the master only re-enters the picture on lease expiry or chunkserver failure.

The invariants the system actually guarantees

GFS does not guarantee POSIX consistency. It guarantees something weaker that the paper carefully defines and that downstream systems were expected to handle:

• A region of a file is consistent if every client sees the same data regardless of which replica it reads from.
• A region is defined if it’s consistent AND it reflects what some specific mutation wrote (no interleaving with concurrent mutations).
• Successful serial writes to non-overlapping regions yield defined regions on all replicas.
• Successful concurrent writes to overlapping regions yield consistent but undefined regions — all replicas agree, but the bytes may be a mash-up of the concurrent writers.
• Failed writes yield inconsistent regions — different replicas may have different bytes — and applications must detect this via checksums.
• record append is at-least-once atomic: each append shows up in its entirety at some offset on all replicas, but duplicates may exist and padding may have been inserted to keep the record aligned within a chunk.

This is a much weaker contract than a POSIX file system, and applications have to be written against it (checksumming records, using unique IDs to detect duplicates, treating reads as best-effort). The bet is that this contract is sufficient for the actual workload, and that the cost in application complexity is less than the cost of providing stronger guarantees at the storage layer. That bet paid off so completely that it became the default stance for warehouse-scale storage.

The trick: record append

The single most-cited contribution of GFS for application designers is record append. POSIX gives you O_APPEND, which is racy across processes on different machines. GFS gives you a primitive where:

• Many concurrent writers on different machines can call record append on the same file simultaneously.
• The system picks an offset for each record and writes it atomically into a chunk (padding if the record won’t fit in the current chunk).
• Each writer learns the offset its record landed at.
• All replicas eventually agree on the bytes at that offset.

This is what makes MapReduce reduce-side output work. It’s what makes log aggregators work. It’s what makes producer/consumer queues over GFS work. It’s the API that POSIX has been missing for fifty years and that every modern object store has reinvented under a different name (S3’s multipart upload, Kafka’s append-only log, etc.).

Why this design has outlasted everything around it

The specific GFS implementation has been retired. The design ideas have not.

• Object stores (S3, GCS, Azure Blob) are the spiritual descendants. Same control/data-plane split. Same workload assumption (large objects, mostly written once and read many times). Same replication / erasure-coding internals. Same “frontend is a metadata service, backend is dumb storage” architecture.
• HDFS is the direct lineal descendant — Doug Cutting and Mike Cafarella reimplemented GFS plus MapReduce in Java as the storage layer for what became Hadoop, with NameNode = Master and DataNodes = chunkservers and record append reappearing as hflush/hsync semantics.
• Bigtable, Spanner, Megastore, and the rest of Google’s data infrastructure ran on GFS (and later Colossus) as their persistent substrate; everything in the L7 stack assumed the underlying storage had GFS’s semantics.
• Colossus replaced GFS by sharding the master across many “D” servers, switching from triple replication to (typically) Reed-Solomon (6,3) erasure coding for cold data, and tightening latency tails — but the API and the workload assumptions remained recognisably GFS.

The deeper lesson is methodological: the paper is a worked example of what happens when you stop trying to fit your system to a pre-existing API contract and start designing the API around the actual workload. Every line of GFS is justified by a sentence somewhere in §2 about what Google’s workload looks like. When you read modern systems papers (Dynamo, Spanner, Calvin, FoundationDB, TiKV) the same shape recurs — section 2 is always a workload characterization, and section 3 onwards is the design that falls out. That recipe didn’t exist as a default before GFS; it was the default after.

What didn’t age well

The paper is honest about its limitations even at publication time, and a few have aged into the “don’t repeat” category:

• The single master was a known scaling cliff and the source of most operational pain. Colossus split it. Any modern reimplementation should not start with a single master.
• Triple replication is wasteful at scale. Erasure coding (Reed-Solomon and successors) gives equivalent durability at ~1.4× overhead instead of 3×. Colossus and HDFS both moved to erasure coding for cold data.
• No POSIX compatibility was deliberate but limits applicability — if you want a general-purpose file system, GFS is the wrong starting point. Modern fork (Ceph, Lustre, BeeGFS) take different positions on this tradeoff.
• Weak consistency put complexity in every application. Newer object stores have tightened this — S3 now offers strong read-after-write consistency, for example — without giving up the architectural shape.

These are corrections at the parameter level. The architecture is intact.

Read alongside

• MapReduce (Dean, Ghemawat, OSDI 2004) — the canonical application of GFS; the two papers should be read together.
• Bigtable (Chang et al., OSDI 2006) — what Google built on top of GFS.
• HDFS architecture (Shvachko et al., MSST 2010) — open-source reimplementation, very close to the GFS paper structurally.
• Colossus overview (Google research blog, 2021) — the GFS successor, sharded metadata + erasure coding.
• Amazon S3 design ideas (2006 launch posts; later “Building S3” talks by Andy Warfield) — different lineage, same architectural shape.
• Dynamo (DeCandia et al., SOSP 2007) — the other end of the design space: peer-to-peer instead of centralized, eventual consistency, choose your own replication. Together with GFS, defines the two poles of warehouse-scale storage architecture.

Links

📄 Google research mirror (PDF) · 📄 SOSP ‘03 proceedings (ACM DOI)

Part of the Weekly CS Paper Digest series. The diagram above is original SVG authored for this post; the paper itself is © ACM 2003.