Redistributed

When a company plugs a language model into its internal knowledge base, something shifts. The model didn't get smarter. The company got more leverage from the intelligence it already had. This is Retrieval-Augmented Generation — RAG — and it is quietly rearranging who needs what in the AI economy.

The Mechanism

A language model, on its own, answers questions from what it absorbed during training (see How Language Models Actually Work for the mechanism). That training data is frozen in time — a snapshot of the internet, books, and code from months or years ago. It knows nothing about your company's internal procedures, your customer records, or the policy document updated last Tuesday.

RAG changes this. Here is what happens when you query a RAG system:

Your question gets converted to an embedding — a numerical representation that captures its meaning, not just its words. The query "What is our refund policy for enterprise clients?" becomes a vector in high-dimensional space.
That embedding is compared against a document store — a collection of your organization's documents, also converted to embeddings. A vector database finds the chunks of text whose embeddings sit closest to your query.
The relevant chunks are retrieved and passed to the model as context — alongside your original question. The model now generates its answer using both its general training and your specific documents.

The model itself hasn't changed. It hasn't learned anything new. It has been handed the right reference material at the right moment, like a well-briefed analyst given a stack of relevant files before a meeting.

This matters because it decouples two things that used to be bound together: model capability and domain knowledge.

Why This Changes the Economics

Before RAG, there were essentially two ways to make a language model useful for your specific domain. You could use a massive frontier model and hope its training data covered your area well enough. Or you could fine-tune a model on your own data — retraining its parameters to encode your specific knowledge.

Fine-tuning works, but it is expensive, slow, and creates a brittle artifact. Every time your underlying data changes, you need to retrain. You need machine learning engineers who understand training pipelines. You need compute budgets. As Lewis et al. documented in the original RAG paper, retrieval-augmented approaches offer a way to get domain-specific performance without the overhead of retraining (Lewis et al., 2020).

RAG sidesteps most of this. Instead of encoding knowledge in model weights, you store it in a searchable database and retrieve it on demand. The trade-offs look roughly like this:

	Fine-Tuning	RAG
Cost	High (GPU hours for training)	Moderate (embedding + storage + retrieval)
Freshness	Stale until retrained	As current as your document store
Control	Opaque (encoded in weights)	Transparent (you can inspect what was retrieved)
Accuracy on domain tasks	High if well-tuned	High if well-indexed
Setup complexity	Needs ML expertise	Needs data engineering

The practical result: organisations that could never afford to fine-tune a model can now get domain-specific AI performance by organising their existing documents and connecting them to a retrieval pipeline. A 7-billion-parameter open-source model with good RAG can outperform a model ten times its size on questions about your internal knowledge base.

The Embedding Economy

RAG has created a new infrastructure layer. Documents need to be converted to embeddings, stored in specialised databases, and retrieved at speed. An entire ecosystem has sprung up to serve this need.

Vector databases — Pinecone, Weaviate, ChromaDB, Qdrant, and the pgvector extension for PostgreSQL — are the storage layer. They are optimised not for rows and columns but for similarity search across high-dimensional vectors. Pinecone raised $100 million in 2023 at a $750 million valuation, a measure of how much capital is flowing into this infrastructure (Pinecone, 2023).

Embedding models turn text into vectors. OpenAI's embedding API, Cohere's Embed, and open-source alternatives like Sentence Transformers each make different trade-offs between quality, speed, and cost. Your choice of embedding model shapes what your RAG system can find, because it determines how "meaning" gets represented numerically.

Chunking strategies determine how documents are split before embedding. Chunk too large and retrieval loses precision. Chunk too small and you lose context. This unglamorous engineering decision has an outsized effect on RAG quality — and it is one reason that organisations with well-structured knowledge bases get dramatically better results than those with messy document stores.

This is a new dependency stack. Where once you needed a model and a prompt, you now need an embedding model, a vector database, a chunking pipeline, a retrieval strategy, and monitoring to ensure quality doesn't silently degrade. Each layer has its vendors, its lock-in risks, and its pricing pressure.

Who Benefits

Enterprises with structured knowledge. The organisations that benefit most from RAG are those that already have well-organised internal documentation, clean knowledge bases, and structured data. RAG rewards prior investment in information architecture. A company with a well-maintained Confluence instance or a curated technical library gets immediate value. A company whose institutional knowledge lives in a thousand untagged email threads gets frustration.

Vector database startups and infrastructure providers. RAG has created a market that barely existed three years ago. Every company implementing RAG needs somewhere to put its embeddings. The vector database vendors are the picks-and-shovels play of the current AI deployment wave.

Consultancies and system integrators. RAG implementation is not trivial. It requires decisions about chunking, embedding models, retrieval strategies, prompt construction, and evaluation. A growing industry of AI consultancies is building RAG pipelines for enterprises that cannot or will not build their own.

Data holders. This is the deeper shift. RAG moves the competitive advantage from model access — which is increasingly commoditised — toward proprietary data. If anyone can use the same base model, the differentiator becomes what you can retrieve. Your proprietary documents, customer interactions, and internal research become the moat. As Gao et al. note in their survey of retrieval-augmented generation, the quality and relevance of the knowledge source is the primary determinant of system performance (Gao et al., 2024).

Who Pays

Organisations without technical capacity. RAG is simpler than fine-tuning, but it is not simple. It requires data engineering, infrastructure management, and ongoing evaluation. Organisations without these capabilities — small nonprofits, local governments, under-resourced schools — cannot easily build RAG pipelines. The democratisation is real but partial: RAG lowers the floor for AI deployment, but the floor is still higher than "install an app."

Anyone with messy data. RAG creates a strong incentive to organise your knowledge base, but organisation is expensive and ongoing. The garbage-in-garbage-out principle applies with force: retrieve the wrong documents and the model will confidently generate wrong answers grounded in irrelevant context. This is arguably worse than a model that simply says it doesn't know.

Workers whose knowledge was tacit. When an organisation builds a RAG system over its internal documentation, it is operationalising institutional knowledge. The employee who was valuable because they knew where to find things and how processes actually worked now competes with a system that can surface the same information instantly. RAG doesn't automate jobs in the dramatic sense, but it erodes the advantage of accumulated organisational knowledge — a dynamic that also plays out in fine-tuning, where worker expertise gets encoded into model weights. The broader pattern is explored in The Shape of AI Displacement.

The Quality Problem

RAG systems fail in ways that are different from — and sometimes less visible than — base model failures.

A base model that doesn't know something might hallucinate, but the hallucination is at least recognisably untethered. A RAG system that retrieves the wrong documents will generate answers that look well-sourced and authoritative but are grounded in irrelevant context. The failure mode is not nonsense but plausible misdirection.

Evaluation is hard. You need to measure not just whether the final answer is correct but whether the right documents were retrieved, whether the most relevant chunks were selected, and whether the model faithfully used the retrieved context rather than falling back on its training data. Barnett et al. identified seven distinct failure points in RAG pipelines, from missing content in the knowledge base to incorrect specificity in retrieval (Barnett et al., 2024). Most organisations deploying RAG are not measuring any of these systematically.

What Readers Should Know

When RAG makes sense: You have a body of domain-specific knowledge that changes over time, and you need a system that can answer questions about it. Customer support, internal knowledge management, regulatory compliance, technical documentation — these are the strong use cases.

When it doesn't: If your questions are general-knowledge, a base model is probably fine. If your data is highly structured and queries are predictable, a traditional database with a good interface might serve you better than the overhead of a RAG pipeline.

Rough cost estimates: A minimal RAG setup using an open-source embedding model, pgvector on an existing PostgreSQL instance, and a hosted LLM API can cost under $100/month for moderate usage. An enterprise deployment with Pinecone, OpenAI embeddings, and a production retrieval pipeline runs into thousands per month, scaling with document volume and query load.

The key question to ask: Before investing in RAG, ask whether your organisation's knowledge is in a state where retrieval will actually help. If the answer to most questions lives in three people's heads rather than in documents, RAG won't help — but the process of preparing for RAG (documenting, structuring, organising) might be valuable in its own right.

RAG is not a revolution in AI capability. It is a revolution in AI plumbing — and as with compute and hardware, in infrastructure the plumbing determines who gets water.

Sources

Lewis, Patrick, et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020. arXiv
Gao, Yunfan, et al. "Retrieval-Augmented Generation for Large Language Models: A Survey." arXiv preprint, 2024. arXiv
Barnett, Scott, et al. "Seven Failure Points When Engineering a Retrieval Augmented Generation System." arXiv preprint, 2024. arXiv
Pinecone. "Pinecone Raises $100M Series B." 2023. Pinecone Blog
Sentence Transformers Documentation. "Pretrained Models." 2024. SBERT.net

General

How Retrieval-Augmented Generation Changes Who Needs What