- Small models (under 15 billion parameters) do as well as giants on bounded tasks: classification, extraction, summarisation, rephrasing.
- They cost ten to fifty times less per call, answer in a fraction of a second and can run on your own servers.
- They still lag on long reasoning, synthesis of contradictory sources and complex instructions.
- The winning architecture in 2026 is a router: small model by default, large model on exception.
What changed in eighteen months
The race for size produced remarkable models, and an invoice many companies can no longer justify. Meanwhile, compact models improved faster than expected: better trained, better distilled, often specialised. In our internal evaluations, a 7 to 12 billion parameter model now reaches the level a model twenty times larger showed two years ago, on the tasks companies actually automate.
And those tasks are not essays. They are ticket classification, field extraction from invoices, meeting summaries, rephrased replies. Short, repetitive tasks with a stable output format. Exactly the home ground of small models.
When the small model is enough
Three conditions, and if they are met, look no further. The task is bounded: one input, a known output format. The context is short: a few pages at most. And you can evaluate it: a test set of two hundred examples with the right answer. Within that frame, a small model, possibly fine-tuned on your data, equals or beats a giant, for a fraction of the cost and with data that never leaves your infrastructure.
The cases where we adopted them
Sorting incoming mail at a mutual insurer, extracting data from purchase orders in manufacturing, language detection and routing in a call centre, anonymising documents before archiving, summarising meetings for a consulting firm. In each case the previous solution relied on a frontier model billed per use. The replacement cut cost by 80 to 95% with no measurable drop in quality.
When it is not enough
Multi-step reasoning, synthesis of sources that contradict each other, following long conditional instructions, complex code generation: on these tasks the gap with large models remains clear and is paid in subtle errors, harder to detect than gross ones. A small model that is wrong is wrong with confidence.
Evaluating a small model on ten hand-picked examples, seeing that it "works", and discovering a 12% error rate on edge cases in production. Two hundred representative examples, including the hard cases, are the minimum.
The architecture that wins: the router
Rather than choosing a model, choose a routing rule. The small model handles everything by default. A confidence score, a context length or a request category triggers the call to the large model. In our deployments, 80 to 90% of the volume stays on the small model, and the invoice follows. The large model becomes a specialist you consult, not a generalist you pay at every sentence.
| Task | Small model | Large model |
|---|---|---|
| Classify, label, route | Yes, by default | Unnecessary |
| Extract structured fields | Yes, often after fine-tuning | On unusual documents |
| Summarise a short document | Yes | Unnecessary |
| Reason across several sources | No | Yes |
| Write with a precise tone at length | Sometimes | Yes |
| Act as a multi-step agent | For simple steps | For planning |
"The question is no longer which model is best. It is which model is sufficient, and for which task."
Conclusion
Small models do not replace large ones; they put each in its place. For a company this means predictable costs, response times users accept and control over its data. The method fits in three words: evaluate, route, measure.
FAQ
What hardware to host a small model?
A server with a 24 GB GPU is enough for a 7 to 12 billion parameter model in ordinary professional use. For large volumes, two GPUs and a dedicated inference server.
Should we fine-tune the model on our data?
For classification and extraction tasks, fine-tuning often adds 5 to 15 points of accuracy with a thousand examples. For summarisation or rephrasing, a good prompt is usually enough.
How do we check quality does not drop?
With a frozen test set, re-run at every change of model or prompt, and an accuracy dashboard checked weekly. Without it, you will not know quality has dropped until a customer tells you.
Are small models safer for sensitive data?
Hosted on your premises, yes: the data does not leave. That is often the decisive argument in healthcare, finance and legal.
