TigerBeetle is a database for financial transactions. It exposes debits and credits in place of SQL, supports only strict serializability, and runs a replicated state machine on top of an LSM tree. To grow from terabytes to petabytes it needed to put cold data in object storage, and the cloud vendors’ SDKs didn’t fit how TigerBeetle does I/O, memory or testing. In this 24-minute talk Georg Kreuzmayr walks through the client they wrote instead, one layer at a time. It runs on a single core with no allocation after startup, in under 40,000 lines of Zig with no dependencies outside the Zig toolchain, and on PUT it’s several times faster than anything they benchmarked against.
Why local NVMe stopped being enough
Until now each replica kept its whole LSM tree on its own NVMe, with hot data in caches and RAM. Kreuzmayr points to India’s UPI, which processes about 10,000 times the volume it did in 2015 and now handles tens of billions of transactions a month, as the kind of workload this has to absorb. Keeping everything on local NVMe breaks in three ways. Single drives top out around 100 TB, NVMe is an expensive place for cold history, and recovery time grows with the data. Re-replicating 128 TB over a 100 Gbit link takes close to three hours.
As your mean time to recovery increases, at some point you start surpassing your mean time between failure.
Past that point a second machine fails while the first is still recovering, and the cluster never catches up. His summary is “scalability is hard because survivability is hard.” TigerBeetle’s answer is what it calls diagonal scaling: the memory hierarchy gets another level below local NVMe, and that level is object storage. Object storage has no practical size limit, is cheap, is shared between replicas instead of copied, and doesn’t need to be rebuilt when a node dies.
Why not use the official SDKs
TigerBeetle processes transactions on one core, and that core can never block. The AWS C++ SDK resolves hostnames with getaddrinfo, which blocks. SDKs that offer async usually build it on thread pools rather than the kernel’s async interfaces, which adds scheduling overhead TigerBeetle avoids on purpose. The AWS C++ SDK alone pulls in more than ten dependencies, and across all vendors, transitive dependencies included, that’s a lot of code nobody on the team controls. The SDKs also allocate, free and copy memory at runtime. TigerBeetle never allocates after startup.
TigerStyle applied to network I/O
TigerStyle is TigerBeetle’s engineering guide, built on safety, performance and developer experience, which they weigh equally. Four of its rules shaped the client:
- Big performance wins come at design time, when you decide how to slice the system and what the interfaces are. Start with a back-of-the-envelope estimate of the best you could do. For this client that meant measuring raw TCP send and receive throughput with io_uring and treating it as the target.
- Every resource has a limit, so state it up front. The number of concurrent connections to the object store is fixed, and all memory is allocated during an initialization phase. Nothing is allocated at runtime.
- Don’t react to external events directly. The client refreshes DNS on a fixed interval of its own choosing and ignores the TTLs in DNS responses.
- No dependencies beyond the Zig toolchain.
Results
On PUT with 8 MB request buffers, plain HTTP and checksums disabled for every client, TigerBeetle’s client reaches 25.2 GB/s on one core. The official AWS SDKs for C++ and Go manage 1.7 and 1.9, and AnyBlob, an object storage client from database research, manages 5.3. Kreuzmayr rounds this to “around 20x” over the official SDKs and 5x over the research state of the art. GET is a closer race: about 3x faster than the official SDKs and around 30% faster than AnyBlob. The rest of the talk explains why GET can’t go further.
The stack, bottom up
The layers are async I/O on top of the kernel’s TCP/IP, then HTTP, optional TLS and DNS, and the S3, Azure and GCP APIs at the top. Each layer was written from scratch.
io_uring, and the same code running on fake I/O
All of TigerBeetle’s network and disk I/O goes through io_uring, with no blocking syscalls anywhere. Every component takes its I/O implementation as a compile-time type parameter. Production passes the io_uring implementation. Tests pass a mock with the same interface that is fully virtual and driven by a seeded PRNG. The mock injects failed connects, sends and receives, partial reads, short writes and bit flips, so each layer is tested against failures as well as the happy path. That’s deterministic simulation testing, applied to a network client.
For tuning io_uring he recommends a paper co-authored by his colleague Tobias Ziegler (High-Performance DBMSs with io_uring: When and How to Use It). The main points are to batch submissions before notifying the kernel, use zero-copy interfaces wherever possible, register buffers once at startup and reuse them, and set the flags the paper recommends. Together those gave close to 10x on a TCP receive microbenchmark.
Send and receive behave very differently. Zero-copy send takes one core from 6.9 GB/s to over 30. Receive stops at 5.6 GB/s even when optimized, because cloud NICs don’t yet have the hardware support for TCP zero-copy receive. A flame graph of the GET benchmark shows only 2% of CPU time in TigerBeetle’s own protocol code. Almost all the rest is kernel TCP receive, and about 60% of the total is a single function, skb_copy_datagram_iter, copying between kernel and user buffers. TigerBeetle’s GET numbers are waiting on the cloud providers.
HTTP, TLS, and the vendor APIs
The HTTP client supports 1.0 and 1.1 and makes GET, HEAD and PUT requests without allocating a buffer, where curl makes about 100 allocations per request. Hostnames are all known at initialization, so DNS lookups on the request path are reads from an in-memory cache. Parsing is sans-I/O (protocol logic kept separate from I/O) so it can be fuzzed in isolation. Connections are reused but deliberately re-established from time to time. There’s also an HTTP server written against the same I/O type, so the whole protocol can be tested end to end under injected faults. Swapping in an HTTP layer that allocates and copies each buffer drops PUT throughput from 25.2 to 3.1 GB/s.
TLS 1.3 follows the same pattern and has its own test server. Zig’s standard-library AES-GCM ran at about 2.2 GB/s per core, so they wrote an optimized version that is more than three times faster. TLS records put the payload and a trailing byte together, and encrypting them as one contiguous buffer means copying the payload every time. Changing the encrypt interface to take the suffix in a separate buffer removed that copy and gained about 12%. For zero-copy sends under TLS, the buffers registered with the kernel are the encryption output buffers, not the caller’s.
At the top, all three vendors share one API: fetch, store, store-no-clobber, list and erase. Checksums (CRC64-NVMe for AWS and Azure, CRC32 for GCP) verify that what was stored matches what was sent. Retries happen inside the client, which sorts failures into fatal, retryable, and retryable-after-delay (503s), and callers only see success or a fatal error. A credentials provider keeps re-reading the credentials file, fetches fresh OAuth tokens for GCP, and checks that new credentials actually work before swapping them in. Their own CRC implementation is nearly 10x faster than the standard library’s.
Owning the stack
Every optimization in the talk came from the same loop of measure, find the bottleneck, fix it, measure again, which is how AES-GCM and CRC turned up as end-to-end bottlenecks. Keeping to the features TigerBeetle actually needs kept the client under 40,000 lines. And because they own every layer, the whole client runs deterministically against the mock I/O, each protocol is tested against its own server, and every parser is fuzzed. You can’t do that with a vendor SDK.
It can be more work up front, but if you do it right, your code stops being technical debt and starts being an asset.
Key takeaways
- TigerBeetle treats object storage as one more tier under local NVMe (“diagonal scaling”), mainly so recovery time stays bounded as data grows and doesn’t overtake the time between failures.
- The official cloud SDKs failed TigerBeetle’s requirements in specific ways: blocking
getaddrinfo, async built on thread pools, more than ten dependencies, and runtime allocation and copying. - Single-core PUT throughput reached 25.2 GB/s, against 1.7 to 1.9 for the official AWS SDKs and 5.3 for AnyBlob. GET improved less because receive is limited by the kernel.
- With zero-copy, TCP send goes from 6.9 to 32 GB/s per core. Receive stays near 5.6 GB/s because cloud NICs lack zero-copy receive, and about 60% of GET CPU time goes to one kernel copy function.
- Making I/O a type parameter lets the same client code run against a seeded fault-injecting mock, which turns a network client into something you can test with deterministic simulation.
- Allocation on the request path costs a lot: an allocate-and-copy HTTP layer cut PUT throughput by about 8x.
- Profile all the way down. Standard-library AES-GCM and CRC were real end-to-end bottlenecks, and replacing them gave 3x and nearly 10x on those primitives.
Source
- Talk: “The Fastest Object Storage Client in the World”
- Speaker: Georg Kreuzmayr, Software Engineer, TigerBeetle
- Channel: TigerBeetle
- Duration: 24:28
- Video: https://www.youtube.com/watch?v=NJrqmoNpVGg