Doc Review: Gemini's Safety Documentation

Documents: Gemini Model Cards (3.1 Pro, 3 Flash, 2.5 series) and Frontier Safety Framework Report Source: Google DeepMind Model Cards

Google is the world's largest AI company by revenue and reach. When Google publishes safety documentation, it establishes industry patterns. Other companies look to Google's format, depth, and omissions as benchmarks for what "adequate" disclosure looks like.

So what do Google's Gemini model cards actually reveal? The answer follows a familiar pattern: strong on safety testing, detailed on benchmarks, silent on training data, absent on environmental costs. But Google's documentation has a distinctive character — elaborate process descriptions paired with minimal concrete data. Understanding this pattern helps clarify not just what Google discloses, but what better disclosure from any lab could look like.

What Model Cards Are Supposed to Do

Mitchell et al. (2019) introduced model cards at Google itself. The framework calls for:

• Model details (architecture, training procedure)
• Intended use and out-of-scope applications
• Training data composition and sources
• Performance across demographic groups
• Limitations and ethical considerations
• Quantitative evaluation metrics

Gebru et al.'s (2021) Datasheets for Datasets standard extends this to training data, requiring documentation of data sources, collection methods, language composition, labeler information, and consent considerations.

Google helped create these standards. The question is whether Google follows them. For a deeper look at these frameworks, see What Model Cards Tell You and Datasheets for Datasets.

What Google Discloses

Safety Evaluations: Process Over Substance

Google's Gemini documentation describes a four-tier evaluation framework:

Development evaluations occur throughout training to assess progress against launch criteria. Assurance evaluations happen at key milestones, conducted by teams outside model development. Red teaming involves specialist adversarial testing. External evaluations engage independent domain experts.

The Gemini 3.1 Pro model card states the model "outperforms Gemini 3 Pro across both safety and tone, while keeping unjustified refusals low." It notes the framework addresses "five risk domains: CBRN (chemical, biological, radiological and nuclear information risks), cyber, harmful manipulation, machine learning R&D and misalignment."

The Frontier Safety Framework report describes threshold-based evaluation where "models' performance across various dangerous capability categories" determines deployment readiness.

This sounds comprehensive. But notice what is provided: categories, not numbers. Process descriptions, not outcomes. We learn that Google tests for CBRN risks. We don't learn what they found. Compare this to Anthropic's approach — as documented in Claude's Model Card — which includes before-and-after mitigation measurements. Google's four-tier framework could be genuinely valuable if it came with data to match its structure.

The cards note that "all critical capability level thresholds remain unmet" and the model "remains below alert thresholds for the CBRN, harmful manipulation, machine learning R&D, and misalignment CCLs." This is reassuring—but without knowing what the thresholds are, we can't assess whether they're meaningful.

Red Teaming: Described, Not Detailed

The Gemini 3.1 Pro card states:

"We conduct manual red teaming by specialist teams who sit outside of the model development team."

It notes that for child safety evaluations, Gemini 3.1 Pro "satisfied required launch thresholds, which were developed by expert teams to protect children online."

But the documentation doesn't reveal:

• The size of the red team
• How many attack attempts were tested
• What specific vulnerabilities were found
• What the "required launch thresholds" actually are
• Which attacks succeeded before mitigations

The cards describe a rigorous process. They don't share its results.

Capabilities: The Benchmark Tour

On capabilities, Google provides substantial detail. The Gemini 3.1 Pro card includes:

• Humanity's Last Exam (with search/code): 51.4%
• ARC-AGI-2 (abstract reasoning): 77.1%
• GPQA Diamond (scientific knowledge): 94.3%
• SWE-Bench Verified (agentic coding): 80.6%
• LiveCodeBench Pro Elo: 2887
• MMMLU (multilingual Q&A): 92.6%

These numbers are specific, verifiable, and comparable across models. This is what meaningful disclosure looks like — and it demonstrates that Google is capable of rigorous, quantitative transparency when it chooses to be. For context on what these benchmarks actually measure (and what they miss), see AI Benchmarks.

Why the asymmetry between detailed benchmark data and absent training data composition? Benchmarks support marketing and product comparison. Training data composition invites legal scrutiny and competitive risk. The incentives, not the capabilities, explain the gap.

What Google Withholds

Training Data: The Security Excuse

The Stanford Foundation Model Transparency Index gave Google a score of 14 out of 58 indicators on their 2025 assessment—among the lowest in the industry. Google scored zero on multiple data categories:

• Data sources (Score: 0): Google states "we do not publicly disclose further details for security reasons"
• Data size (Score: 0): No token counts or parameter information
• Language composition (Score: 0): Not disclosed
• Domain composition (Score: 0): Only vague reference to "wide range of domains"

The Gemini 3.1 Pro card defers specifics, stating: "For more information about the training dataset for Gemini 3.1 Pro, see the Gemini 3 Pro model card." The linked card provides no additional detail.

What we learn from various sources: Pre-training included "publicly-available web-documents, code, images, audio and video" with "deduplication, safety filtering, and quality filtering" applied. Google acquires publicly available data from "The Pile and CommonCrawl."

What we don't learn:

• Dataset size
• Proportion from each source
• What copyrighted material is included
• What personal data is included
• Selection and filtering criteria
• Which websites contributed data

Google's justification, per the FMTI report: "specific numbers of FLOPs, along with parameters, could give competitors an idea of our proprietary approach" and facilitate "reverse engineering."

This justification deserves scrutiny. While there are legitimate security considerations in some disclosure contexts, training data composition is not a meaningful attack vector for language models. It is, however, a meaningful accountability vector — enabling questions about copyright, consent, and bias that Google would prefer to manage privately. The Who Owns Training Data analysis explores why these questions matter so much.

Labor Conditions: Invisible

Google's model cards don't document:

• Who labels training data
• Where labelers are located
• How they're compensated
• What working conditions they face
• Which contractors are used

Compare this to the detail Google provides about its evaluation partners (external experts, AI Safety Institutes) or its hardware (TPUv5p architecture, multiple 8,960-chip pods). External partnerships get named. Labor gets erased.

As Kate Crawford documented in Atlas of AI, the AI industry systematically renders labor invisible while celebrating engineering achievements. Google's documentation exemplifies this pattern. The Ghost Work and Hidden Labor reviews examine why this invisibility matters and what alternatives exist — including third-party labor audits and supply chain transparency standards.

Environmental Impact: Complete Absence

Google's Gemini documentation provides zero information on:

• Training compute in FLOPs
• Energy consumption
• Carbon emissions
• Water usage for cooling

The FMTI found that Google discloses none of the key information related to environmental impact—joining AI21 Labs, Alibaba, Amazon, Anthropic, DeepSeek, Midjourney, Mistral, OpenAI, and xAI in complete non-disclosure.

Google publicly commits to carbon neutrality and renewable energy. But without baseline numbers, these commitments are unfalsifiable. How much energy did training Gemini 3 consume? We don't know. For the scale of what is being left undisclosed, see Environmental Cost and Data Center Energy.

Architecture: The Black Box

The cards note that Gemini uses a "sparse Mixture-of-Experts Transformer" trained across "multiple 8,960-chip pods." Beyond this, architecture remains opaque:

• No parameter counts
• No training duration
• No specific technique details beyond high-level descriptions

The technical report for Gemini 2.5 exists. It provides benchmark results and safety categories. It does not provide the technical details that would enable independent evaluation or reproduction.

The Datasheets Standard vs. Google's Practice

Gebru et al.'s Datasheets for Datasets standard—published in Communications of the ACM with Google co-authors—specifies over 50 questions datasets should answer:

Category	Datasheets Requirement	Google Disclosure
Motivation	Why was the dataset created?	Not disclosed
Composition	What does it contain?	Vague categories only
Collection	How was data collected?	"Publicly available" + crawling
Preprocessing	How was it cleaned/filtered?	"Deduplication, safety filtering"
Uses	Intended applications?	Documented
Distribution	How is it shared?	N/A (not shared)
Maintenance	Who maintains it?	Not disclosed
Legal	Consent and rights?	Not disclosed
Labor	Who did annotation work?	Not disclosed

Google participated in creating the standard. Google does not follow it.

Comparison to Other Labs

Anthropic (Claude)

Anthropic's cards are similarly strong on safety, weak on data. Training data gets one paragraph: "a proprietary mix of publicly available information on the Internet... as well as non-public data from third parties." Neither company names specific datasets.

Anthropic provides something Google doesn't: specific external evaluation partners named (Apollo Research, Deloitte, NNSA). Google describes external evaluation without naming evaluators.

OpenAI (GPT-4)

OpenAI's GPT-4 System Card provides comparative analysis between early and launch versions, showing specific safety improvements. The card includes before-and-after examples demonstrating how mitigations changed behavior.

Google describes improvement ("outperforms Gemini 3 Pro across both safety and tone") without showing the specific changes.

Meta (Llama)

Meta's Llama cards are more structurally complete on some dimensions. They provide data mix percentages—not sources, but proportions. The original LLaMA card specified "67% from CCNet; 4.5% from GitHub."

Google provides no comparable breakdown. Google does publish a robots.txt opt-out mechanism (Google Extended) that Meta doesn't offer, but this addresses future data collection, not past training data disclosure.

The Industry Pattern

All major labs:

• Document safety evaluations extensively
• Provide benchmark performance
• Describe acceptable use policies
• Omit specific training data sources
• Obscure labor practices
• Ignore environmental costs

The 2025 Stanford FMTI found that five Frontier Model Forum companies (Amazon, Anthropic, Google, Meta, OpenAI) "all score in the exact middle" of the index. The mean score is 41 out of 100—and declining. "The entire industry is systemically opaque about four critical topics: training data, training compute, how models are used, and the resulting impact on society."

Google is not unusually opaque. Google is normally opaque — and the normal standard across the Frontier Lab Oligopoly is inadequate. The FMTI's declining scores suggest that without regulatory pressure, the industry is drifting toward less transparency, not more.

The Redistribution Angle

Documentation as Power

Who can evaluate Google's AI systems? Those with access to information Google chooses to release. This is structural control over accountability.

When documentation is incomplete:

• Regulators can't verify compliance
• Researchers can't reproduce findings
• Affected communities can't trace harms to sources
• Competitors can't assess claims
• The public can't make informed decisions

Google decides what counts as adequate disclosure. Google decides who can hold Google accountable.

The Security Rationalization

Google's stated reason for opacity is security: training data details might enable "reverse engineering" or "data poisoning."

This is implausible. Knowing that Gemini was trained on CommonCrawl doesn't enable attacks. What it enables is questions: Which websites? Did you have permission? What copyrighted material did you include? What personal data?

Security is the excuse. Liability is the concern.

Whose Harms Get Tested

The Frontier Safety Framework focuses on CBRN, cybersecurity, persuasion, and "machine learning R&D" risks. These are serious categories. They also share a common feature: they represent harms to powerful actors (states, corporations, infrastructure).

What's less visible in the documentation:

• Bias affecting marginalized communities
• Labor exploitation in training data production
• Environmental justice concerns
• Copyright and creator rights
• Surveillance capabilities

As Virginia Eubanks argued in Automating Inequality, automated systems concentrate risk on the vulnerable while protecting the powerful. The categories Google tests reflect this pattern: advanced biosecurity threats get dedicated evaluations; algorithmic discrimination gets vague assurances. The Gender Shades study demonstrated concretely how this plays out — when evaluation categories omit equity, the resulting systems fail marginalized communities first. To Google's credit, their evaluation framework is more comprehensive than many competitors'; the issue is that comprehensiveness of categories without transparency of results is process without accountability.

Documentation Labor Shifts to Critics

The Stanford FMTI team spent months gathering information that Google could provide in hours. Journalists, researchers, and civil society organizations do the accountability work that documentation should enable.

This is a redistribution of labor. Google externalizes transparency costs onto the public.

What This Means for You

If you're deploying Gemini:

• You have good information about capabilities and benchmarks
• You have process descriptions for safety evaluation
• You don't have data to audit training sources, assess biases, or evaluate supply chain ethics
• You're trusting Google's internal processes without external verification

If you're affected by Gemini:

• You can't determine if your work was used in training
• You can't assess labor conditions in the production chain
• You can't evaluate environmental claims

If you're evaluating AI companies:

• Google's documentation matches industry norms
• Industry norms are inadequate
• Process descriptions without outcomes data is theater, not transparency

If you're advocating for AI accountability:

• The Datasheets standard exists — push Google to meet it
• Support the Stanford FMTI and similar accountability projects
• Demand concrete data, not process descriptions
• Note that Google helped create the standards it doesn't follow — this gives advocates leverage
• The EU AI Act and Who Regulates Foundation Models provide the emerging regulatory context that may compel disclosure

The Bottom Line

Google's Gemini documentation tells you what the model can do and how Google thinks about safety testing. It does not tell you what Gemini was trained on, who did the training labor, or at what environmental cost.

The cards are procedurally elaborate and substantively thin. They describe evaluation frameworks without sharing results. They reference thresholds without specifying them. They invoke external expertise without naming partners. They assure without demonstrating.

Google co-authored the model card and datasheet standards. Google does not meet them. This is not a technical limitation — it is a policy choice shaped by the same competitive and legal pressures facing every major lab.

The largest AI company in the world has decided that adequate disclosure means process descriptions without data, safety categories without findings, and benchmark numbers without training details. When Google makes this choice, it establishes what "normal" looks like for the industry.

The current normal is insufficient. But the path forward is visible: Google's own benchmark disclosure shows they can do rigorous, quantitative transparency. Their safety framework shows they can build structured evaluation processes. The challenge is extending that rigor to the domains — data, labor, environment — where disclosure is uncomfortable rather than commercially useful. External pressure from regulators, civil society, and projects like the FMTI is what closes that gap.

Related

• Claude's Model Card — Anthropic's parallel approach, stronger on mitigation data
• What Model Cards Tell You — The disclosure framework Google helped create
• Datasheets for Datasets — The standard Google co-authored but does not follow
• AI Benchmarks — What Gemini's benchmark numbers actually measure
• Gender Shades — Why equity evaluations need more than vague assurances
• Ghost Work — The labor Google's documentation renders invisible
• Environmental Cost — The impact Google's cards do not report
• Frontier Lab Oligopoly — The competitive dynamics driving industry-wide opacity
• EU AI Act Explained — Regulatory pressure that may reshape disclosure norms
• AI Market Concentration — Why Google's choices set industry patterns

Sources

• Google DeepMind. "Gemini 3.1 Pro Model Card." February 2026. DeepMind
• Google DeepMind. "Gemini 3 Pro Frontier Safety Framework Report." November 2025. PDF
• Google DeepMind. "Gemini 3 Pro Model Card." December 2025. PDF
• Google DeepMind. "Model Cards." DeepMind
• Stanford CRFM. "Google: Gemini 2.5 Pro Transparency Report." December 2025. Report
• Stanford CRFM. "Foundation Model Transparency Index." December 2025. FMTI
• Wan, Alexander et al. "The 2025 Foundation Model Transparency Index." Stanford CRFM, December 2025. Paper
• Mitchell, Margaret et al. "Model Cards for Model Reporting." FAT Conference, 2019. arXiv
• Gebru, Timnit et al. "Datasheets for Datasets." Communications of the ACM, 2021. arXiv
• Crawford, Kate. Atlas of AI. Yale University Press, 2021. Publisher
• Eubanks, Virginia. Automating Inequality. Macmillan, 2018. Publisher