Also available in: Português

AI Fundamentals: Training, Inference and Quantization

In the previous article, the idea was to understand what an LLM is, how it breaks text into tokens and what parameters and the context window mean. Here the focus shifts from “what the model is” to “how it runs”: the two main phases, training and inference, and quantization, which is what makes it feasible to fit larger models on limited hardware.

How does training work?

Training is the phase in which the model adjusts its parameters based on large datasets. At this stage, it learns language patterns by trying to improve its ability to predict tokens correctly.

After training comes inference, which is when you use the model to generate responses. When you run a local model on your computer, you are usually doing inference, not full training.

Diagram comparing training and inference: in training, using large datasets, the model tries to predict the next token, measures how wrong it was and adjusts its parameters, repeating the cycle until it becomes a trained model; in inference, the trained model receives the prompt, processes the tokens and generates the output

What is inference?

Inference is running the already trained model to respond to a prompt. It is the moment when it takes the input, processes the tokens and generates the output.

If your focus is using AI locally, inference is the most important term at first. The main bottleneck is usually memory, generation speed and the model’s compatibility with your hardware.

What is quantization?

Quantization is a technique for reducing the numerical precision of the model’s weights and, sometimes, its activations, swapping heavier formats, such as FP32 or FP16, for smaller ones, such as INT8. The goal is to reduce memory usage and speed up inference.

In practical terms: quantization is a way of “compressing” the model so it runs better, especially locally. You lose a bit of precision in some cases, but you gain a lot in feasibility, cost and speed.

Diagram of quantization: in FP32 each weight uses 32 bits, in FP16 it uses 16 bits and in INT8 it uses 8 bits; the same weight becomes 0.12345679, 0.12347 and about 0.122; in a 7B model, the weights alone take about 28 GB, 14 GB and 7 GB

Why does quantization matter locally?

Without quantization, many models are too large to run comfortably on ordinary machines. With quantization, you can fit larger models into less RAM or VRAM and use cheaper or more limited hardware.

So, when you start testing local models, you will probably come across names and formats related to this. The general idea is always the same: trying to keep acceptable quality at a lower computational cost.