Polymath Engineer Weekly #42
Let's learn something together
Hello again. Have a nice week ;)
Links of the week
“On the engineering side of things, we are building a reliable, predictable system. And predictable means really predictable. Rather than reigning in sources of non-determinism, we build the whole system from the ground up from a set of fully deterministic, hand crafted components. Here are some of our unconventional choices (design doc):
It’s hard mode! We allocate all the memory at a startup, and there’s zero allocation after that. This removes all the uncertainty about allocation.
The code is architected with brutal simplicity. As a single example, we don’t use JSON, or ProtoBuf, or Cap’n’Proto for serialization. Rather, we just cast the bytes we received from the network to a desired type. The motivation here is not so much performance, as reduction of the number of moving parts. Parsing is hard, but, if you control both sides of the communication channel, you don’t need to do it, you can send checksummed data as is.
We aggressively minimize all dependencies. We know exactly the system calls our system is making, because all IO is our own code (on Linux, our main production platform, we don’t link libc).“
“This wouldn’t work in Chrome because we release every day. We can’t tolerate huge chunks of new code suddenly showing up in trunk because it would have a high chance of taking down the canary or dev channels for an extended period. Also, the trunk of Chrome moves so fast that it isn’t practical for developers to be isolated on a branch for very long. By the time they merged, trunk would look so different that integration would be difficult and error-prone.“
▶ Illustrated Guide to Transformers Neural Network: A step by step explanation
“Transformers are the rage nowadays, but how do they work? This video demystifies the novel neural network architecture with step by step explanation and illustrations on how transformers work.“
“Most importantly, plugins will turn ChatGPT into one destination for nearly everything: search, discovery, travel planning, restaurant booking, gift shopping, first drafts, research, you name it. Importantly, unlike a traditional platform, through which users can find and download apps, ChatGPT consumes plugins and absorbs their capabilities into the main product. It’s a product as a platform, or a platform as one product.”
“Diffusion models are inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps to slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. Unlike VAE or flow models, diffusion models are learned with a fixed procedure and the latent variable has high dimensionality (same as the original data).“
Some thoughts on my presentation style
“Back when I was an academic, my conference talks were on research papers that I had co-authored, and the advice I was given was “the talk is an ad for the paper”: don’t aim for completeness in the talk, instead, get people interested in enough in the material that they’ll read the paper.
Nowadays, my talks aren’t based on papers, but the corresponding advice I’d give is “evoke a reaction”. There’s a limit to the quantity of information that you can convey in a talk, so instead go for conveying a reaction. Get people to connect emotionally in some way, so that they’ll care enough about the material that they’ll follow up to learn more some time in the future. I’d rather my audience leave angry with what I presented than leave bored and retain nothing.“
Book of the Week
Kill It with Fire: Manage Aging Computer Systems (and Future Proof Modern Ones)
Do you have any more links our community should read? Feel free to post them on the comments.
Have a nice week. 😉
Have you read last week's post? Check the archive.