Redistributed

The prevailing narrative of AI progress runs in one direction: bigger. More parameters, more training data, more compute, more money. GPT-4 reportedly cost over $100 million to train. Google's Gemini Ultra required clusters of thousands of TPUs. The arms race has a clear logic — scale up and performance follows. But a counter-trend is quietly reshaping who gets to use AI, and on what terms.

Small language models — roughly 1 to 8 billion parameters — are delivering performance that would have been considered impossible at their size just two years ago. They run on laptops. They run on phones. They run without an internet connection, without an API key, without sending a single byte to a cloud provider. This is not a curiosity. It is an infrastructure shift with profound implications for how AI capability gets distributed.

What Counts as "Small"

The term is relative. In 2020, GPT-3's 175 billion parameters seemed enormous. Today, models with 400 billion or more parameters define the frontier, and anything under 10 billion counts as small. The category includes some of the most capable models per parameter in existence:

Microsoft Phi series. Phi-2 (2.7B), Phi-3 (3.8B), and Phi-4 (14B) demonstrated that aggressive data curation could close much of the gap with models ten times their size. Microsoft's approach — training on carefully filtered, high-quality data including synthetic textbook-style material — challenged the assumption that scale was the primary lever.
Google Gemma. The Gemma 2 family (2B, 9B, 27B) brought Google's research infrastructure to bear on efficient model design. The 9B variant matches or exceeds many earlier models with far more parameters.
Meta Llama 3.2. The 1B and 3B variants are designed explicitly for on-device deployment. Meta released them with mobile-optimized architectures, signaling that small models are a strategic priority, not an afterthought.
Mistral 7B. When Mistral released its 7B model via a torrent link in September 2023, it outperformed Llama 2's 13B model on most benchmarks. The message was clear: parameter count is an increasingly poor proxy for capability.
Qwen 2.5 series. Alibaba's Qwen models span from 0.5B to 72B parameters, with the smaller variants performing remarkably well on multilingual tasks — a detail that matters for global access.

These are not toys. On standard benchmarks for text generation, summarization, classification, code completion, and translation, the best 7-8B models now approach what GPT-3.5 delivered when it was the commercial frontier in early 2023.

Why Small Models Got Good

Three years ago, the scaling laws seemed settled. The 2020 Kaplan et al. paper from OpenAI established what appeared to be a universal relationship: make models bigger, and they get better, predictably. The implication was that size was destiny.

Then Hoffmann et al. at DeepMind published the Chinchilla paper in 2022, and the story changed. Their finding was deceptively simple: most large language models were undertrained. A 70B model trained on the standard amount of data could be matched by a much smaller model trained on significantly more data. The optimal ratio, they argued, was roughly 20 tokens per parameter — meaning a 7B model should ideally see around 140 billion tokens of training data. Many existing large models had seen far less data per parameter than this ratio suggested.

This insight unlocked the small model revolution. If you could achieve comparable performance by training smaller models on more, better data, then the constraint shifted from model size to data quality and training efficiency. Several techniques converged:

Data curation. Microsoft's Phi research showed that carefully selected, high-quality training data — including synthetic data generated by larger models — could dramatically improve small model performance. The insight: a billion parameters trained on excellent data beats ten billion parameters trained on internet noise. Quality of training signal, it turned out, mattered as much as quantity of parameters.

Knowledge distillation. Larger models can teach smaller ones. By training a small model to mimic the outputs and intermediate representations of a large model, you transfer capability without transferring size. The student learns compressed versions of what the teacher knows. This works because large models contain enormous redundancy — much of their parameter space encodes overlapping or rarely-used information.

Architectural improvements. Grouped-query attention, rotary positional embeddings, sliding window attention, and mixture-of-experts routing all improve the ratio of capability to parameter count. These are engineering innovations that make each parameter work harder. The Mistral 7B model, for instance, used sliding window attention to handle longer contexts without the quadratic cost of full attention.

Better tokenization. Improved tokenizers that encode text more efficiently mean the model processes more meaning per token. This sounds mundane but compounds across billions of tokens of training data. A 15% improvement in tokenization efficiency is equivalent to 15% more training data for free.

What They Can Do, and Where They Struggle

The honest answer is: small models handle most everyday tasks competently. Text summarization, email drafting, code completion for common patterns, classification, sentiment analysis, translation between major languages, question answering over provided context — a well-tuned 7B model does these adequately and sometimes well.

Where they fall short is equally important. Complex multi-step reasoning — the kind that requires holding many constraints in working memory simultaneously — degrades faster in small models than in large ones. Nuanced creative writing that requires maintaining voice, tone, and narrative consistency over thousands of words remains a frontier model strength. And broad knowledge recall — the ability to answer questions about obscure topics from parametric memory alone — is directly proportional to model size. A 3B model simply cannot store as much world knowledge as a 400B model. The information has to go somewhere.

The practical implication: for many production use cases, small models suffice. A customer service bot answering questions about a known product catalog. A code completion tool suggesting the next few lines of Python. A document classifier routing incoming requests. These tasks are well-defined, contextually constrained, and do not require the full breadth of a frontier model. The gap between "good enough for this task" and "best possible performance" is often not worth the 50-100x difference in inference cost.

The Access Revolution

Here is where the redistribution story sharpens. A quantized 7B-parameter model — compressed from 16-bit floating point to 4-bit integer precision — fits in approximately 4 gigabytes of RAM. That is less than most modern laptops allocate to a web browser. It runs on a MacBook Air. It runs on a midrange Android phone. It runs on a Raspberry Pi, slowly but functionally.

Quantization is the compression technique that makes this possible. Full-precision model weights use 16 bits per parameter (FP16). Reducing to 8-bit integers (INT8) halves the memory requirement with minimal quality loss. Going to 4-bit (INT4) halves it again, with some degradation that is often acceptable for practical use. A 7B model at full precision requires around 14 GB of memory. At INT4 quantization, it needs roughly 4 GB. Tools like llama.cpp, Ollama, and LM Studio have made running quantized models locally a matter of downloading a file and clicking a button.

This changes the economics fundamentally. No API key. No per-token charge. No data leaving the device. No internet connection required. No terms of service governing what you can ask or what the model can say. The inference tax — the perpetual rent paid to cloud AI providers for every token processed — drops to zero. The marginal cost of a query becomes the electricity to run your own hardware, which is negligible.

For specific communities, this is not an incremental improvement. It is a category shift:

Developers without cloud budgets. A student in Nairobi or a freelancer in rural Brazil can run a capable language model without a credit card attached to a cloud provider. The barrier to experimentation drops from recurring expense to one-time hardware that they may already own.

Organizations with privacy requirements. Healthcare providers, law firms, financial institutions — anywhere data cannot legally or ethically leave the premises — can deploy AI capability without the compliance nightmare of sending sensitive information to third-party APIs. On-device inference means the data stays where the law says it must.

Low-connectivity environments. Large parts of the world lack reliable high-speed internet. Offline-capable AI models work in field hospitals, remote schools, disaster zones, and the vast stretches of the Global South where connectivity is intermittent. As Crawford documents in Atlas of AI, the infrastructure requirements of cloud AI systematically exclude populations already marginalized by existing technology distribution (Crawford, 2021).

Users who want autonomy. Anyone who objects to their queries being logged, their data being used for training, or their access being subject to a provider's content policy gains genuine independence with local models. This is not paranoia. It is a reasonable preference for tools you control.

The Tension: Democratized Access, Concentrated Origins

The redistribution picture is not uncomplicated. Most high-performing small models owe their capabilities to large models in some way. Phi's training data included synthetic data generated by GPT-4. Distilled models are, by definition, compressed versions of larger ones. The open-weight models from Meta and Mistral were trained on compute infrastructure that cost tens of millions of dollars.

The value chain still begins with concentrated resources. The democratization is real, but it occurs downstream. As Widder et al. observe, "open" in the AI context often means open enough to use but not open enough to reproduce — the training data, the RLHF process, and the full compute recipe remain proprietary (Widder et al., 2023). Small models extend access without dismantling the concentration that produces them.

This matters. If the small model ecosystem depends on large model research, then the companies controlling frontier models retain structural power even as their outputs become commodity goods. They set the ceiling that small models approach. They choose what to release, when, and under what license. Meta's Llama license permits most uses but reserves the right to restrict competitors with more than 700 million monthly active users. Generosity with limits is still control.

Acemoglu and Johnson argue that the direction of technology is a choice, not an inevitability — and that choices about who benefits from innovation are made by those who control its development (Acemoglu & Johnson, 2023). The small model trend expands the beneficiary pool, but the choice remains upstream.

What This Means for You

If you build software, develop content, or make decisions about AI adoption, small models deserve serious evaluation. Here is practical guidance:

When SLMs suffice: Any task with a well-defined scope and available context. Classification, extraction, summarization, translation, code completion for common patterns, conversational interfaces over specific knowledge bases. If you can clearly describe what "good" looks like for the task, a 7B model can probably deliver it.

When they do not: Open-ended creative work at high quality. Complex analytical reasoning with many variables. Tasks requiring broad world knowledge without provided context. Anything where the difference between 90% accuracy and 98% accuracy matters — medical diagnosis, legal analysis, safety-critical applications.

Which to try first: Llama 3.2 3B for the smallest usable general model. Mistral 7B or Qwen 2.5 7B for the best balance of capability and size. Phi-4 if you need strong reasoning in a compact package. All are available through Ollama or LM Studio with minimal setup.

Hardware requirements: For a 7B quantized model, any machine with 8 GB of RAM and a reasonably modern processor. Apple Silicon Macs perform particularly well due to unified memory architecture. For the 1-3B models, most smartphones manufactured after 2022 have sufficient capability.

The calculation: If your application processes fewer than roughly 10 million tokens per month, API pricing may still be cheaper than the engineering effort to self-host. Above that threshold, self-hosting economics become favorable. But cost is only one variable. Privacy, latency, reliability, and autonomy all enter the equation.

The rise of small language models does not end the concentration of AI power. The largest models will continue to improve, and the most capable AI will remain expensive to run. But SLMs redraw the floor — the minimum level of AI capability available to anyone with a modest computer. That floor has risen dramatically, and it keeps rising.

In the history of infrastructure, this pattern has precedent. Mainframes gave way to personal computers. Centralized telephone exchanges gave way to mobile networks. Each transition expanded access without eliminating the advantages of scale. The powerful retained their advantages, but the minimum viable participant gained genuine capability.

Small language models are personal computers for AI. Not as powerful as the mainframe. But yours.

Sources

Crawford, Kate. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, 2021. Publisher
Hoffmann, Jordan et al. "Training Compute-Optimal Large Language Models." arXiv, 2022. arXiv
Widder, David Gray et al. "Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI." SSRN, 2023. SSRN
Acemoglu, Daron & Johnson, Simon. Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity. PublicAffairs, 2023. Publisher
Abdin, Marah et al. "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone." Microsoft Research, 2024. arXiv
Kaplan, Jared et al. "Scaling Laws for Neural Language Models." arXiv, 2020. arXiv

General

The Rise of Small Language Models