A classic language model pushes every word through all of its parameters. The bigger the model, the more computation per word and the more expensive every answer becomes.
The idea: several specialists instead of one generalist
In a mixture-of-experts architecture the model is split into many smaller networks called experts. For each fragment of text a small component known as the router decides which experts should handle it - usually two or four out of several dozen. The rest stay idle.
The result: the model holds a large amount of knowledge, but the cost of a single answer matches a model several times smaller. Graphics card memory has to hold all the experts, while the amount of computation depends only on the active ones.
What this means for a deployment
- for the same graphics card budget you get better answers;
- more people can use the system at once, because each query consumes less compute;
- memory, not raw speed, becomes the binding constraint - and it drives the hardware choice.
What it does not solve
Mixture of experts does not make a model smarter in a field it never saw during training, and it does not remove the need to give it current data. That is what search across your organisation's documents is for, described in a separate article.