This is the first post on the blog. I’ll be using this space to document what I’m learning as I dig deeper into ML systems and inference — notes, experiments, and things that trip me up along the way.
I figured the most fundamental place to start was understanding how LLMs actually work. If I want to understand how they run, I should first understand how they work under the hood.
I found 3Blue1Brown’s neural networks playlist really helpful, especially the episode on attention in transformers. He does a great job explaining the intuition and motivation. After that, I could follow along with the “Attention Is All You Need” paper.
Even then, transformers still felt like a lot, and I did not feel like I fully understood them without getting hands-on (or building a miniGPT from scratch). So I started following along with the first few videos from Andrej Karpathy’s Zero to Hero series.
I also created my own Jupyter notebook to make things more hands-on instead of only following along. Sometimes it was hard to tell whether I was just typing what he typed and taking lines of code for granted versus actually being able to do it myself. Feel free to download it if helpful. It helped me get more comfortable with PyTorch while testing my understanding of transformers.