Redistributed

Every time you send a prompt to ChatGPT or Claude, your words travel to a distant data center, get processed on hardware you don't own, and generate a response that's logged on servers you can't inspect. The transaction is convenient, capable, and entirely dependent on someone else's infrastructure.

There's another option. You can run language models on your own hardware — your laptop, your desktop, your home server. The models are smaller, the capabilities more limited, but the tradeoffs reveal something important about how AI access actually works.

Why This Matters for Redistribution

Running AI locally is an access question. Cloud APIs extract data; local models don't. Corporate services charge per token; local inference costs electricity. Platform models require internet; local runs offline. These aren't just technical differences — they're power differences.

As Yochai Benkler documented in The Wealth of Networks, commons-based alternatives to proprietary systems can redistribute capability away from concentrated gatekeepers. Local AI is one such alternative: imperfect, constrained, but independent.

The question isn't whether local models match frontier services. They don't. The question is whether they're good enough for specific needs, and what you gain when they are.

What "Local" Actually Means

When we say "local AI," we mean inference — running a trained model to generate outputs — on consumer hardware. You download model weights (the learned parameters, often gigabytes in size), load them into memory, and generate responses without any network connection.

This is different from training. Training a model from scratch requires massive compute clusters and datasets. But running a trained model? That's increasingly feasible on ordinary machines.

The economics split here. Training remains concentrated in a handful of well-funded labs (costing upwards of $1 billion per run). Inference is becoming democratized. You can't build GPT-4 at home, but you can run a surprisingly capable open-weights model on a gaming laptop. Understanding this split — between who can create models and who can run them — is key to understanding the redistribution dynamics of AI infrastructure.

What You Need: Hardware Requirements by Model Size

Language models come in sizes measured by parameter count — the number of learned values in the model. More parameters generally means more capability, but also more hardware required.

Small Models (1B-7B parameters)

These run on most modern laptops. A 7-billion parameter model, quantized (compressed), needs roughly 4-8GB of RAM. An M1 MacBook Air handles them fine. So does a mid-range gaming PC.

What they can do: Basic text generation, simple Q&A, code completion, summarization. Quality is noticeably below frontier models, but often good enough for drafting, brainstorming, or local search.

Example models: Llama 3 8B, Mistral 7B, Phi-3 Mini, Qwen2 7B.

Medium Models (13B-30B parameters)

These need more muscle. Budget 16-32GB of system RAM, or a GPU with at least 8GB of VRAM for reasonable speeds. Apple Silicon Macs (M1 Pro and up) handle this tier well because GPU and RAM share the same memory pool.

What they can do: Noticeably better reasoning, longer coherent outputs, better instruction-following. Many users find 13B-class models sufficient for daily tasks.

Example models: Llama 2 13B, CodeLlama 34B, Mixtral 8x7B (a "mixture of experts" that activates different parameter subsets per query).

Large Models (70B+ parameters)

Here, consumer hardware starts to strain. A 70-billion parameter model needs 35GB+ of RAM just to load the quantized weights. Running it fast enough to be useful requires either a very high-end GPU (RTX 4090 with 24GB VRAM), multiple GPUs, or an M2/M3 Max/Ultra Mac with 64GB+ unified memory.

What they can do: Approach (but don't match) frontier model quality for many tasks. Genuinely useful for complex writing, code, and analysis.

Example models: Llama 3 70B, Qwen2 72B, DeepSeek 67B.

The Apple Silicon Factor

Apple's M-series chips changed local AI economics. Unified memory means the GPU can access all system RAM, not just dedicated VRAM. A Mac Studio with 192GB unified memory can run models that would require thousands of dollars of GPU hardware on Windows.

This isn't marketing; it's architectural. The M3 Ultra's memory bandwidth (800 GB/s) beats many discrete GPUs for memory-bound inference. For running large local models, Apple Silicon currently offers the best consumer-accessible option.

The PC Path

On Windows/Linux, the path runs through NVIDIA GPUs. AMD and Intel support is improving but CUDA remains the best-supported framework. Key considerations:

VRAM matters most. An RTX 4090 (24GB) outperforms two RTX 3080s (10GB each) for large models.
System RAM matters for CPU inference. If you're running on CPU (slow but possible), 64GB+ helps.
Quantization lets you run bigger models on smaller hardware — more on this below.

The Software Landscape

You don't need to compile C++ to run local models. Several tools have made local inference approachable.

Ollama

The simplest path. Ollama packages models into a single downloadable unit and provides a command-line interface. ollama run llama3 downloads and runs Meta's Llama 3. It handles quantization, memory management, and provides an API compatible with OpenAI's format, so existing tools often work.

Best for: Getting started quickly, API-compatible local inference, Mac users.

llama.cpp

The engine under much of the local AI ecosystem. llama.cpp is a C++ implementation of model inference optimized for CPUs and diverse hardware. It introduced the GGUF format for quantized models and runs on everything from Raspberry Pis to server clusters.

Best for: Maximum hardware compatibility, embedding in other applications, advanced users.

LM Studio

A desktop application with a graphical interface. Browse and download models, adjust parameters, chat — no command line required. Cross-platform, though best on Mac.

Best for: Users who want a visual interface, model exploration, non-technical users.

text-generation-webui

A web interface for local models with extensive configuration options. More complex to set up than LM Studio but more flexible.

Best for: Power users who want fine-grained control, running multiple models, advanced inference settings.

Open WebUI

A self-hosted web interface that can connect to Ollama or other backends. Provides a ChatGPT-like experience for local models, including conversation history and user accounts.

Best for: Self-hosting, multi-user setups, teams wanting shared local AI.

Quantization: How Compression Enables Access

Here's where local AI gets interesting. A model's full-precision weights might total 140GB. But do you need full precision?

Quantization compresses weights by reducing their numerical precision. Instead of storing each parameter as a 16-bit floating-point number, you might use 8 bits, 4 bits, or even 2 bits. The model gets smaller and faster at some cost to quality.

The Formats

GGUF: The dominant format for CPU and consumer inference, developed by the llama.cpp project. GGUF files come in multiple quantization levels: Q8 (highest quality, largest), Q5, Q4 (common balance), Q3, Q2 (smallest, most degraded). A "Q4_K_M" variant of Llama 3 70B fits in about 40GB instead of 140GB.

AWQ and GPTQ: GPU-focused quantization methods that often achieve better quality-per-bit than naive quantization. Commonly used for GPU inference in production.

INT8/INT4: Simpler quantization used by some frameworks. Less sophisticated than AWQ/GPTQ but widely supported.

The Quality Tradeoff

Quantization isn't free. Models lose information. But the loss is often acceptable:

Q8 (8-bit): Nearly indistinguishable from full precision for most tasks.
Q4 (4-bit): Noticeable quality loss on some tasks, but often acceptable. The most common compromise.
Q2 (2-bit): Significant degradation. Useful for fitting models on constrained hardware, but expect worse outputs.

The practical implication: quantization is a form of accessibility. A 70B model that would require a $10,000 hardware setup at full precision becomes runnable on a $3,000 laptop at Q4. The compressed model is worse, but it exists for people who can't afford the uncompressed version.

This echoes a pattern in infrastructure access. Water can be delivered by pipe or by truck. Pipe is better but requires infrastructure investment. Truck is worse but reaches more people. Quantization is the truck — a compromise that extends reach.

Choosing a Model: Capability vs. Requirements

Not all models are created equal. The open-weights ecosystem has stratified into tiers.

The Frontier-Adjacent Tier

Models that approach closed-API capabilities for specific tasks. Llama 3 70B and its derivatives, Mixtral 8x22B, Qwen2 72B. These require serious hardware but deliver serious results.

Use for: Tasks where you'd otherwise pay for GPT-4 or Claude. Code generation, analysis, long-form writing.

The Sweet Spot Tier

7B-13B models that punch above their weight. Mistral 7B, Llama 3 8B, Phi-3 Medium. These run on laptops and often suffice for daily tasks.

Use for: Local coding assistants, summarization, brainstorming, Q&A against your own documents.

The Specialized Tier

Models fine-tuned for specific domains. CodeLlama for programming, BioMistral for biomedicine, Nous Hermes for instruction-following. Smaller than general models but often better within their domain.

Use for: When you know what you need. A 7B coding model might outperform a 70B general model on code.

Model Sources

Hugging Face hosts most open-weights models. The Ollama library and LM Studio provide curated selections. TheBloke (on Hugging Face) maintains an extensive collection of quantized versions.

Licensing varies. Llama models have acceptable-use restrictions (The Llama License). Mistral models are Apache-licensed. Phi models have Microsoft's terms. The distinction between truly open-source and open-weights licenses matters — see Open Weights Are Not Open Source for why. Check before deploying commercially.

What You Give Up

Running locally isn't all upside. The limitations are real.

No Frontier Capabilities

The best open-weights models trail the best closed APIs by a significant margin. GPT-4, Claude 3 Opus, Gemini Ultra — these remain inaccessible locally. For tasks requiring genuine frontier capability (complex reasoning, specialized domains, safety-critical applications), local models may not suffice.

This gap may narrow or widen. It's currently wide. But the Rise of Small Language Models — purpose-built models that punch above their weight on specific tasks — is narrowing it for many practical use cases.

Update Lag

When OpenAI ships a new capability, you have it immediately. When Meta releases a new Llama, you wait for quantization, testing, and community validation. The local ecosystem runs months behind the frontier.

Maintenance Burden

Cloud APIs handle infrastructure invisibly. Local models require you to manage updates, hardware, storage, and troubleshooting. This is work. If your laptop's GPU crashes, that's your problem.

No Ecosystem Integration

ChatGPT plugins, Claude's artifacts, Gemini's Google integration — these don't exist locally. You're running a model, not a product. Building integrations is possible but requires effort.

Energy and Environmental Costs

Running inference locally consumes electricity. For occasional use, this is negligible. For heavy use, it accumulates. The environmental argument for local AI isn't straightforward — corporate data centers can be more energy-efficient per inference than consumer hardware.

The Autonomy Question

Ursula Franklin distinguished "holistic" technologies (that extend human capability under human control) from "prescriptive" technologies (that reduce humans to following machine outputs). Where does local AI fall?

Local models offer one form of autonomy: freedom from corporate platforms. Your prompts aren't logged on someone else's servers. Your usage isn't metered by someone else's pricing. Your access isn't contingent on someone else's terms of service.

But this autonomy has limits. The models themselves emerged from corporate labs (Meta, Mistral, Microsoft). The training data was scraped from the web without consent. The hardware supply chain runs through conflict minerals and exploited labor. Running locally doesn't make AI innocent; it just changes which dependencies you accept.

Kate Crawford's work in Atlas of AI reminds us that no AI is truly "local" in the full sense — the supply chains are global, the training data is collective, the research lineages are institutional. What local running does is shift the operational control from remote platforms to local hardware. That's meaningful but partial.

Privacy: The Strong Argument

The clearest case for local AI is privacy. When you run locally:

Your prompts stay on your machine. No corporate logging, no training on your data, no third-party access.
Sensitive information can be processed safely. Medical notes, legal documents, proprietary code — local inference poses no exfiltration risk.
Offline operation is possible. Air-gapped systems can run AI without network connectivity.

For individuals working with sensitive data, local AI may be the only acceptable option. For organizations with strict data governance, local deployment solves problems that cloud APIs cannot.

This matters for redistribution. Privacy is often a luxury good — those with resources can afford to protect their data; those without cannot. Local AI democratizes one form of privacy protection, extending it to anyone with adequate hardware.

Getting Started: Practical Paths

Path 1: Casual Exploration

Install Ollama or LM Studio. Download a 7B model (Llama 3 8B or Mistral 7B). Start chatting. This takes minutes on most modern machines.

Good for: Seeing what local AI feels like, basic tasks, learning.

Path 2: Daily Driver

Invest in hardware that handles 13B-30B models comfortably (M2 Pro Mac or RTX 4070+). Run Mistral or Llama 3 8B for daily tasks. Use cloud APIs when you need frontier capability.

Good for: Most users who want local AI as a practical tool.

Path 3: Maximum Local

Get serious hardware (M3 Max with 96GB or multi-GPU workstation). Run 70B-class models. Build integrations with local tools. Accept slower inference for full autonomy.

Good for: Privacy-critical applications, research, true independence from cloud services.

Path 4: Server Deployment

Run models on a home server or cloud instance you control. Use Open WebUI for multi-user access. Connect to existing workflows via APIs.

Good for: Teams, organizations, self-hosted infrastructure.

The Bigger Picture

Local AI is a redistribution mechanism, but an imperfect one. It shifts control from platforms to users — for users who can afford adequate hardware. It creates independence from corporate APIs — while depending on corporate-trained models. It enables privacy — for those with technical capability.

The pattern echoes other infrastructure questions. Rooftop solar redistributes energy generation but requires upfront capital. Home water filtration redistributes treatment but requires purchase and maintenance. Local AI redistributes inference but requires hardware investment.

These are real redistributions, meaningful improvements in autonomy and access. They're also partial. They don't eliminate power asymmetries; they rearrange them. Understanding both the gains and limits helps make informed choices about where local AI fits in a broader strategy for more equitable AI access.

The frontier will stay in the clouds. But the useful middle — the AI capable enough for most daily tasks — is increasingly runnable in your own hands. What you do with that capability is up to you.

Sources

Benkler, Yochai. The Wealth of Networks. Yale University Press, 2006. Publisher
Crawford, Kate. Atlas of AI. Yale University Press, 2021. Publisher
Franklin, Ursula. The Real World of Technology. House of Anansi Press, 1989. Publisher
Ollama. Official documentation. ollama.ai
llama.cpp. GitHub repository. GitHub
LM Studio. Official site. lmstudio.ai
Hugging Face. Model hub and documentation. huggingface.co
Meta AI. Llama model documentation. llama.meta.com
Mistral AI. Model documentation. mistral.ai

General

Running AI Locally