Polymath Engineer Weekly #86
Database Sharding, Kafka + eBPF, Swiss Watches, Compression, Sofas, BJJ and Leadership
I have created a survey to get feedback from you. It takes only 2 minutes.
Hello again.
Comic of the week
Links of the week
How Figma’s databases team lived to tell the scale
Given these tradeoffs, we began to explore building a horizontally sharded solution on top of our existing vertically partitioned RDS Postgres infrastructure. It didn’t make sense for our small team to re-implement a generic horizontally sharded relational database in-house; in doing so, we’d be competing with tools built by the likes of large open source communities or dedicated database vendors. However, because we were tailoring horizontal sharding to Figma’s specific architecture, we could get away with providing a much smaller feature set. For example, we chose not to support atomic cross-shard transactions because we could work around cross-shard transaction failures.
Unlocking Kafka's Potential: Tackling Tail Latency with eBPF
Slow produce requests without corresponding slow writes were always occurring around the time of some other slow write. We started wondering whether those requests were perhaps queuing and waiting for something to finish. By analyzing Kafka source code, we identified a couple of places that use synchronized blocks, including those guarding log file writes.
We set out to measure how much time Kafka’s threads, processing produce requests, spend on the aforementioned locks. Our goal was to correlate periods when they were waiting on locks with writes to the file system. We considered two approaches to do that.
The secret to luxury watch success: Tissot up. Tudor down. Why?
The Morgan Stanley report is in. It's not just numbers and statistics. For once this report gives a clear indication of what it takes to be a succesful luxury watch manufacturer.
Building Columnar Compression for Large PostgreSQL Databases
By compressing your dataset, you can grow your PostgreSQL databases further. As we’ll see across this article, this highly effective compression design allows you to reduce the size of your large PostgreSQL tables up to 10-20x. You can store much more data on smaller disks (a.k.a. saving money) while improving query performance. Timescale compression is also fully mutable, making database management and operations easy: you can add, alter, and drop columns in compressed tables, and you can INSERT, UPDATE, and DELETE data directly.
The big change in shipping logistics came when furniture manufacturers developed new ways of deconstructing and flat-packing their product, allowing sofas to be shipped via regular carriers, both more cheaply and more quickly. This innovation opened up new avenues for penny-pinching. Favorable tax situations, cheap labor, and a surging wood and wood-type-product industry in Asia made it suddenly very inexpensive to construct furniture there, and those new shipping tactics made transportation feasible.
An Evidenced-Based Training Plan for Brazilian Jiu-Jitsu
Although inherently a self-defense system, structured BJJ competitions have increased in popularity and professionalism over the past 15 years (27), with elite international and numerous regional events available to the BJJ competitor. Additionally, many international grappling events offer lucrative prizes to higher-level competitors. The shift from self-defense to sport, whereby time limits and points govern victory, has increased the reliance on the physical capacities that were once only a minimal component of BJJ. As such, there is now an increased demand for evidenced-based strength and conditioning methods to provide competitors with a physical advantage.
Book of the Week
Extreme Ownership: How U.S. Navy SEALs Lead and Win
Do you have any more links our community should read? Feel free to post them on the comments.
Have a nice week. 😉
Have you read last week's post? Check the archive.