Before a language model can answer your questions, it goes through a multi-stage process that shapes its behavior. Understanding this pipeline reveals where decisions get made and whose values get embedded.
Stage 1: Pre-training
This is the most expensive step. The model sees billions of text examples and learns to predict the next word. By the end, it's absorbed patterns from across the internet, books, code repositories, and whatever else was in the training data.
Who decides what goes in? The company building the model. They choose sources, apply filters, and determine what counts as "quality." These choices shape what the model knows and how it thinks. The question of Who Owns the Training Data — and whether its creators have any say in how it's used — is one of the most contested in AI today.
What patterns emerge? The model learns the statistical structure of text — grammar, facts, reasoning styles, biases, everything. It doesn't distinguish "true" from "plausible." It learns what humans have written, which includes a lot of things that are false, offensive, or harmful.
What it costs: Tens of millions of dollars in compute. Months of training time on thousands of specialized chips. This is why only a handful of organizations can do it — a dynamic explored in Economics of Training a Frontier Model.
Stage 2: Supervised Fine-Tuning
Raw pre-trained models are useful but crude. They'll complete any prompt, including harmful ones. They'll produce plausible-sounding garbage. They need shaping.
In supervised fine-tuning (SFT), the model learns from curated examples of good behavior (Ouyang et al., 2022). Human contractors (or other AI models) produce demonstrations: "For this prompt, here's what a helpful, harmless response looks like."
Who decides what's "good"? The company, through contractors, guidelines, and quality control. This is where values start getting imposed explicitly — what the model should refuse, what tone it should adopt, what topics it should handle carefully.
What it costs: Much less than pre-training but still significant. Thousands of examples, each requiring human judgment.
Stage 3: RLHF (Reinforcement Learning from Human Feedback)
This step refines the model further using comparative judgments. Humans (or other AI) see two or more model outputs and pick the better one. A reward model learns from these comparisons, and then the language model is trained to maximize rewards.
Why this helps: Direct demonstration is limited — you can only show so many examples. RLHF lets the model explore behaviors and get feedback on a larger space of possibilities.
Why this is tricky: The model optimizes for what gets rewarded, not for what's actually good. If raters prefer confident answers, the model becomes more confident — even when it shouldn't be. If raters prefer certain tones or styles, the model adopts them regardless of content quality.
Who decides what's rewarded? The company, through rater guidelines, quality control, and the choice of who does the rating. This is value embedding at scale. The workers doing the rating — often contractors in the Global South — are themselves part of the story (see The RLHF Workforce and Data Labeling in the Global South).
Stage 4: Constitutional AI and Other Variants
Some labs use variations on RLHF. Anthropic's Constitutional AI, for instance, uses AI-generated feedback based on explicit principles rather than (only) human preferences (Bai et al., 2022). The model is trained to critique and revise its own outputs according to a "constitution" of rules.
Who writes the constitution? The company. The principles sound neutral — "be helpful, harmless, and honest" — but their interpretation in specific cases is anything but.
Stage 5: Ongoing Adjustments
Training isn't over at launch. Models get updated based on usage patterns, user feedback, safety incidents, and changing priorities. Entire categories of outputs get suppressed or enabled based on company policy.
Who decides what changes? The company, often with minimal transparency. You might notice that a model's behavior shifted, but you probably won't know why or when it will shift again.
The Infrastructure View
Each stage of the pipeline embeds decisions:
| Stage | Key Decision | Who Decides |
|---|---|---|
| Pre-training | What data to use | Company |
| SFT | What behavior to demonstrate | Company + contractors |
| RLHF | What to reward | Company + raters |
| Constitutional | What principles to enforce | Company |
| Updates | What to change | Company |
At every step, the model is being shaped according to someone's values and incentives. The output isn't neutral or natural — it's engineered.
Understanding the pipeline means understanding that "the AI" isn't a single thing making independent judgments. It's the product of a supply chain, each link involving human decisions that could have been made differently. Transparency tools like model cards and Datasheets for Datasets propose making these decisions visible — but adoption is voluntary and uneven. Proposals for Democratic AI Governance aim to give the public more say in how these decisions get made. Whether any of this changes the concentration of decision-making depends on whether transparency translates into accountability, which so far it largely has not.
Related
- How Language Models Actually Work
- Compute Is the New Oil
- The Autonomy Illusion
- What Fine-Tuning Really Does
- What Model Cards Tell You
- The RLHF Workforce
- Who Owns the Training Data
- Democratic AI Governance
Sources
- Anthropic. "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback." 2022. arXiv
- OpenAI. "GPT-4 Technical Report." 2023. arXiv
- Bai, Yuntao et al. "Constitutional AI: Harmlessness from AI Feedback." 2022. arXiv
- Ouyang, Long et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022. arXiv
- Perez, Ethan et al. "Red Teaming Language Models with Language Models." 2022. arXiv