Polymath Engineer Weekly #26
Interesting reads for the hungry mind
Hello again, I’m glad to share with you some of my recent reads =)
Links of the Week
The Distributed Computing Manifesto
“So, when I was invited by Amazon in 2004 to give a talk about my distributed systems research, I almost didn’t go. I was thinking: web servers and a database, how hard can that be? But I’m happy that I did, because what I encountered blew my mind. The scale and diversity of their operation was unlike anything I had ever seen, Amazon’s architecture was at least a decade ahead of what I had encountered at other companies. It was more than just a high-performance website, we are talking about everything from high-volume transaction processing to machine learning, security, robotics, binning millions of products – anything that you could find in a distributed systems textbook was happening at Amazon, and it was happening at unbelievable scale.“
“As more and more users interact with generative AI models, we are gaining a deeper understanding of the problems most immediately addressable by AI: ones where we have lots of training data already; where getting the correct answer 99% of the time is very useful, and the incorrect 1% won’t be disastrous; and where the underlying models can continually ingest human feedback and become better over time.”
Push-based Outbox Pattern with Postgres Logical Replication
“There are only a few patterns that I’m comfortable saying: “You should use it always if you want to build mature system”. One of them is the Outbox Pattern. Why? Because it guarantees that your business workflows and communication will not be stuck in the middle.”
“It's often necessary to access the historical values of some property. But sometimes this history itself needs to be modified in response to retroactive updates. Bitemporal history treats time as two dimensions: actual history records what history should be given perfect transmission of information, while record history captures how our knowledge of history changes.“
At Sourcegraph, writing outweighs the “water-cooler mythos”
“Writing is the tool that has allowed us to scale with clarity and consistency over the years, especially having a geographically-distributed workforce.”
Trunk and Branches Model for Scaling Infrastructure Organizations
“The solution I’ve found effective for addressing the infrastructure organization rules is an approach I call the Trunk and Branches Model. You start with a “trunk team” that is effectively your original infrastructure team. The trunk is responsible for absolutely everything that other teams expect from infrastructure, and might be called something like “Infra Eng,” “Platform Eng,” or “Core Infra.”
As the team grows, you identify a particularly valuable narrow subset of the work. Valuable here means one of three things:
it’s an exponential problem that will overrun your entire organization if you don’t solve it soon; for example, test or build instability accelerating as you hire more engineers
It’s a recurring fire that is undermining your company with users; for example, database instability causing site outages
It’s an internal workflow that’s starving your team’s ability to make investments; for example, a clunky process for manually spinning up new services in a company accelerating service adoption“
Book of the Week
The Cold Start Problem: How to Start and Scale Network Effects
Do you have any more links our community should read? Feel free to post them on the comments.
Have a nice week. 😉
Have you read last week's post? Check the archive.