Recent Posts
Interesting Reads/Links 2025
Why RL is not used in industry. Other stuff on the blog is pretty level-headed, related to (mostly continuous?) optimization and machine learning.
A list of links related to system programming I find insightful.
I like the hot-takes, and honesty. (Surely the state of tech employers in Australia is not like that...)
Best short story I've read in a while.
- <https://www.scottsm...
How do destroy value
Step 1: Innovation creates technology that greatly boosts productivity, and thus creates extra wealth and value in society.
Step 2: Wealth and value is first generated bottom up, but due to winner-take-all effects, eventually one or a couple gain an oligopoly over the market.
Step 2.5: Other adversarial competitive forces effectively funnel wealth into a few middle-men (the ones selling shovels and building platforms). Like advertisers or banks in the current iteration.
Step 3: The excess ge...
Some thoughts on the newest NVIDIA GPU features
In trying to write a GPU compiler myself I had to familiarize myself with the newest features on Hopper and Blackwell, despite not owning one of those GPUs (and giving up a kidney in the process); what I learned did not sparkle joy.
I would categorize these features as either 1) communication/synchronization, 2) tensor, 3) misc. They are rather obscure and I tried to get a high-level understanding, since I couldn't actually program with them due to a lack of hardware. So here goes a birds-eye-...
CUDA Block Scheduler
The CUDA programming model supports launching dynamic number of blocks in a grid, and dynamic number of threads in a block. To make this possible, the underlying hardware must be able to dynamically schedule blocks onto actual cores - streaming multiprocessors (SMs), and dynamically schedule warps onto warp execution units.
Typically there are much more blocks than SMs, and much more warps than warp execution units. When a kernel is launched, the hardware resources used by a block is known: ...
The Open Society and its woes
The open society as described by Karl Popper is one in which normative laws are made distinct from natural laws. Where its citizens have the awareness to separate in those rules they obey the rules that are decided by people, and the rules that are inevitable. This is in contrast to the closed society where people do not have this discriminative ability and so the rules, or taboos that they obey are deemed "magical" and spawn forth from the divine, the gods, or mother earth, etc.
The magical...
The perils of Optimization
A tale of 2 orgs: As Q2 wraps up, two very senior managers, Alice and Bob, at two similar and competing companies (companies A and B) are deciding on what their team should move towards for the next two quarters. Their departments deals with systems and performance aspects of a deep learning application that is fairly central to the company's core product, and has tight latency requirements. Both Alice and Bob has been receiving numerous complaints from the department that they are working ...