Polymath Engineer Weekly #91
Blob Storage, Memory Arenas, Formal Methods, Streaming meets Blob Storage, OTP, SNES and Rust
Hello again. I think this week's links are pretty cool, enjoy ;)
Comic of the week
Links of the week
Paper Notes: f4 – Facebook’s Warm Blob Storage System
Blob storage system is built in such a manner that it abstracts away the concept of hot/warm storage from the client. Requests enter the system through a router layer where create requests are sent to the hot layer whereas delete requests are sent to the host where the blob is currently being stored. For read requests, router extracts the logical volume id & map it to the physical volume where blob is stored.
Arena allocator tips and tricks
An arena is a memory buffer and an offset into that buffer, initially zero. To allocate an object, grab a pointer at the offset, advance the offset by the size of the object, and return the pointer. There’s a little more to it, such as ensuring alignment and availability. We’ll get to that. Objects are not freed individually. Instead, groups of allocations are freed at once by restoring the offset to an earlier value. Without individual lifetimes, you don’t need to write destructors, nor do your programs need to walk data structures at run time to take them apart. You also no longer need to worry about memory leaks.
Formal Methods: Just Good Engineering Practice?
At first, this may seem counter-intuitive. Formal methods aren’t cheap, aren’t particularly easy, and don’t fit well into every software engineering approach. Its reasonable to start with the belief that a formal approach would increase costs, especially non-recurring engineering costs. My experience is that this isn’t true, for two reasons. The first is rework. Software engineering is somewhat unique in the engineering fields in that design and construction tend to happen at the same time, and a lot of construction can be started without a advancing much into design.
Tableflow: The Stream/Table, Kafka/Iceberg Duality
The data infrastructure world is undergoing multiple paradigm shifts simultaneously, all driven by S3 and the other cloud object storage services (GCS, Azure Blob Storage). While the trends we are seeing now have been in the making for a number of years, it seems that 2024 will be the year that the shift really gathers pace. As they say, change comes slowly, then all at once.
How do One-Time Passwords Work?
The algorithm itself does the following three steps. It generates a HMAC-SHA-1 value, where secret is the key and counter is the message to be hashed using the key. Then it truncates the 20-byte hash string to a shorter value.
The truncation is rather uncomplicated, the last byte from the 20-byte string is taken, the lower 4 bits are considered the “offset”. This makes the offset to be always in the range from 0 to 15.
Inside the Super Nintendo Cartridges
One of the remarkable characteristics of the Super Nintendo was the ability for game cartridges (cart) to pack more than instructions and assets into ROM chips. If we open and look at the PCBs, we can find inside things like the CIC copy protection chip, SRAM, and even "enhancement processors".
I have created a survey to get feedback from you. It takes only 2 minutes.
Book of the Week
Rust for Rustaceans: Idiomatic Programming for Experienced Developers
Do you have any more links our community should read? Feel free to post them on the comments.
Have a nice week. 😉
Have you read last week's post? Check the archive.