Artificial Intelligence
AI Fundamentals: Dense Models vs MoE - Linky Systems

AI Fundamentals: Dense Models vs MoE - Linky Systems

AI Fundamentals: Dense Models vs MoE

The previous article covered NLP and NLU, the conceptual field behind language models. This last one closes the series by going back to architecture: the difference between dense models and MoE, which parameters are actually activated for each token, and why that changes the math on cost and memory, a topic that had already come up in the article on quantization and local use.