When someone asks me, “Can you build an AI application just by vibe coding?” my answer is yes, you probably can. But building an application that works is one thing. Building an AI system that can handle a million requests without becoming too slow or too expensive is a different challenge.
There is a lot more to AI engineering than making sure your model gives the right answer or completes the task. You need to think about inference latency, GPU memory, deployment, context management, and what happens when your application starts getting more traffic than you expected. If you’re building AI agents, things get even more complicated.
There are some good courses that cover these concepts too, but I’d recommend reading books in depth if you really want to build your fundamentals. You don’t necessarily have to pay for them either. I’ve gathered 7 FREE books that I feel are worth reading. Some are older and you may have already heard of them, while others are newer. So, let’s get started.
1. Machine Learning Systems Design by Chip Huyen
This book is one of the classics and is good for beginners, so I’d start with this one. It treats machine learning as an end-to-end problem and covers its four main stages: 1) project setup, 2) preparing data pipelines, 3) modeling (selecting, training, and debugging), and 4) serving (testing, deploying, and maintaining). Each stage includes links to practical resources and case studies from machine learning engineers at large tech companies. Another good thing is that it has 27 system design questions at the end for practice. A lot has changed since 2019, so it doesn’t go deep into large language model (LLM) infrastructure, but it’s still useful for understanding how machine learning systems fit together.
2. Machine Learning Engineering Open Book by Stas Bekman
This is more of a reference than a book you read from start to finish. It’s based on Stas Bekman’s experience with large model projects like BLOOM-176B and IDEFICS-80B, and it covers GPU hardware, networking, storage, SLURM orchestration, training, inference, and debugging. I’d keep it bookmarked for when you run into problems like GPU out-of-memory errors or slow multi-node training. The GitHub repository has nearly 20,000 stars, and it’s distributed under a CC BY-SA 4.0 license, so it’s free to read and share.
3. AI Agents in Depth: Design Principles and Engineering Practice by Bojie Li
If you’re building AI agents, I’d definitely recommend checking this one out. It starts with a simple idea: an agent is an LLM, context, and tools. From there, it covers context engineering, memory, retrieval-augmented generation (RAG), tool integration, and multi-agent systems, and it explains the engineering decisions behind agents that handle longer, more complicated tasks. It also comes with lots of practical experiments. Bojie Li has another book, Understanding AI Infra, which I really like, so I’m sharing it here too.
4. The Architecture of AI Agents and LLM Systems by Aleksandr Drobiazkin
This is a much newer book that focuses specifically on building LLM applications. It covers how language models work, context engineering, RAG, structured outputs, tool calling, and different agent architectures. It also goes into the engineering side, like evaluation, observability, security, inference costs, and latency. Instead of just showing you how to use a framework, it also explains the trade-offs between approaches. It also has a companion course.
5. Inference Engineering by Philip Kiely
This is probably one of the most relevant books on this list if you’re interested in deploying LLMs and making them faster and cheaper. It walks through the whole inference stack, from GPUs and model architectures to serving software and production deployments. You’ll learn about batching, quantization, memory usage, and latency, and how to choose the right hardware or serving setup. I’d recommend it if you’re experimenting with local LLMs or engines like vLLM. It also explains why a model can run fine for one user but start struggling when hundreds of requests hit it at once.
6. LLM Inference Illustrated by Ted Kyi
This book is really good if you want to understand what happens when an LLM generates a response. It’s also full of detailed diagrams, so it works well if you learn better visually. It walks through how tokens get generated one at a time, then covers how inference systems make that faster, lighter on memory, and able to scale. Instead of treating model serving as a black box, you’ll get a clearer understanding of the engineering techniques that make it possible to serve LLMs efficiently.
7. How to Scale Your Model by Jacob Austin et al.
This book comes from researchers at Google DeepMind, and I’d highly recommend it if you want to understand how LLMs are trained and served across thousands of accelerators. It starts with how TPUs work and then goes into their architecture, memory bandwidth, model sharding, distributed training, and inference optimization. One thing I really like is that it uses LLaMA 3 as a practical example to explain how much memory, computing power, and money you need to train or serve a large model. Some of the math can get quite technical, but it’s a great resource nonetheless.
Final Thoughts
You don’t have to read all seven books from beginning to end. Pick the ones that match what you’re working on, and come back to the rest when you need them.
I think we’re reaching a point where almost anyone can build an AI application with the help of coding agents. But knowing how to make that application reliable, fast, secure, and affordable is what separates a working demo from a production system.
And that’s exactly why I think learning AI systems engineering is becoming more important, not less.
Kanwal Mehreen is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook “Maximizing Productivity with ChatGPT”. As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She’s also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.
