LLM Structured Output Failures: When JSON Mode Breaks and How to Repair It (2026)

LLM Structured Output Failures: When JSON Mode Breaks and How to Repair It (2026)

Structured output mode was supposed to end LLM JSON errors. The pitch was clear: enable JSON Mode, add a schema, and your LLM will never again return a markdown-fenced code block where you expected a clean {.

The reality is messier. Structured output reduces failure rates dramatically — from 8–15% to under 1% in typical cases — but it does not eliminate them. When they do fail in production, the failures are subtler and harder to debug than the raw-output errors everyone knows. Truncated arrays that look valid until you count the elements. Refusal objects that are valid JSON but not your schema. Schema drift on complex nested structures at high concurrency.

This guide covers every provider-specific failure mode, with exact repair code for Python and TypeScript, so you can handle them before they break your pipeline.

> Instant repair: When structured output fails, paste the broken JSON into AI JSONMedic's repair tool for a one-click fix. For provider-specific errors, use the OpenAI JSON Mode fixer or Claude JSON Fixer — or use the repair API in your CI to catch it before it hits production.

---

The Myth of Zero-Error Structured Output

In 2026, the major providers offer:

None of these reach zero. Production pipelines that call LLMs at scale — thousands of calls per day — will hit these edge cases regularly. A 1% failure rate on 5,000 daily calls means 50 broken JSON responses that need handling.

---

OpenAI: Where Strict Mode Still Breaks

OpenAI's Structured Outputs (GA since Nov 2024) with strict: true is the most reliable schema enforcement available from any major provider. But three scenarios still cause failures.

Failure 1: Truncation at max_tokens

The most common remaining failure in OpenAI Structured Outputs. If max_tokens is too low for the schema's required output, the model truncates mid-JSON:

{
  "items": [
    {"id": 1, "name": "Widget A", "price": 9.99},
    {"id": 2, "name": "Widget B

The fix is increasing max_tokens, but when truncation has already happened, repair is needed:

import json
import requests

def parse_openai_response(content: str, schema: dict) -> dict:
    """Parse OpenAI structured output with repair fallback."""
    # Step 1: Try clean parse
    try:
        return json.loads(content)
    except json.JSONDecodeError:
        pass

    # Step 2: Attempt repair via aijsonmedic API
    try:
        response = requests.post(
            "https://aijsonmedic.com/api/repair",
            json={"json": content},
            timeout=5
        )
        if response.ok:
            repaired = response.json().get("repaired", "")
            return json.loads(repaired)
    except Exception:
        pass

    # Step 3: Retry the LLM call with explicit repair prompt
    raise ValueError(f"Could not repair structured output: {content[:200]}")

Failure 2: Refusal Objects

When the model declines to answer, OpenAI Structured Outputs returns a valid JSON refusal object instead of your schema:

{
  "refusal": "I can't help with that request."
}

This is valid JSON — your JSON.parse() will succeed — but it doesn't match your schema. Always check for the refusal key before using the output:

def handle_openai_output(message) -> dict:
    if message.refusal:
        raise ValueError(f"Model refused: {message.refusal}")
    return json.loads(message.content)

Failure 3: Streaming Mode

Streaming mode disables Structured Outputs enforcement. If you're using stream=True, the model falls back to unguided generation — which means all the raw LLM JSON errors (markdown fences, trailing commas, Python literals) can reappear.

When you must stream + parse JSON, assemble the full stream first, run repair, then parse:

from openai import OpenAI

client = OpenAI()

def stream_and_parse(prompt: str) -> dict:
    stream = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},  # JSON Mode, not Structured Outputs
        stream=True
    )

    # Assemble full output
    full_content = ""
    for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        full_content += delta

    # Repair before parse — strip fences, normalize quotes
    cleaned = full_content.strip()
    if cleaned.startswith("

"):

cleaned = cleaned.split("```")[1]

if cleaned.startswith("json"):

cleaned = cleaned[4:]

return json.loads(cleaned)

---

## Anthropic Claude: Schema Enforcement Edge Cases

Claude's `output_config.format` (available since Claude 3.5 Sonnet) enforces JSON schema via guided generation, similar to OpenAI's mechanism. Three edge cases still break it.

### Failure 1: Deeply Nested Schemas

Claude's schema enforcement becomes unreliable above 8–10 levels of nesting. If your schema has a structure like `order → items[] → product → attributes → pricing → tiers[]`, you may get outputs where a deeply nested required field is omitted or has the wrong type.

The fix: flatten your schema where possible. Instead of deep nesting, use parallel top-level arrays with ID references. If you can't restructure, add validation after parse:

python

from anthropic import Anthropic

import jsonschema

client = Anthropic()

def claude_structured_call(prompt: str, schema: dict) -> dict:

message = client.messages.create(

model="claude-sonnet-4-6",

max_tokens=4096,

output_config={

"format": {

"type": "json_schema",

"json_schema": schema

}

},

messages=[{"role": "user", "content": prompt}]

)

data = json.loads(message.content[0].text)

# Validate against schema — structured output doesn't guarantee business logic

try:

jsonschema.validate(data, schema)

except jsonschema.ValidationError as e:

# Log the violation, retry with correction prompt

correction = f"Your previous output violated the schema: {e.message}. Return corrected JSON only."

# ... retry logic

raise

return data

### Failure 2: tool_use Union Type Violations

When using Claude's `tool_use` API (as opposed to `output_config`), JSON schema enforcement on `input_schema` has a known edge case with union types. A field defined as `{"type": ["string", "null"]}` can cause Claude to return the field with value `None` (Python literal) instead of `null` (JSON):

python

Claude tool_use output — looks valid, isn't

'{"user_id": 42, "nickname": None}'

Run a targeted repair before parsing tool_use results:

python

import re

def repair_claude_tool_output(raw: str) -> str:

"""Fix Python literal leakage in Claude tool outputs."""

fixed = re.sub(r'\bNone\b', 'null', raw)

fixed = re.sub(r'\bTrue\b', 'true', fixed)

fixed = re.sub(r'\bFalse\b', 'false', fixed)

return fixed

Or use the [AI JSONMedic repair tool](/json-repair) which handles Python literals and 16+ other corruption patterns automatically.

### Failure 3: Context Window Pressure

Near Claude's 200k context limit, output quality degrades. Structured output enforcement is among the first capabilities to weaken. If you're passing very long context + requesting a complex structured output, truncation or schema violations become much more likely. Keep structured output calls under 100k input tokens for reliable enforcement.

---

## Google Gemini: responseSchema Failures

Gemini's `responseMimeType: "application/json"` with `responseSchema` uses constrained decoding (via grammar-based generation) for well-supported schema patterns. But failure rates are higher than OpenAI/Claude, particularly in three scenarios.

### Failure 1: Recursive Schema References

Gemini's constrained decoding doesn't support `$ref` recursive schema patterns. A schema like `TreeNode { children: TreeNode[] }` will cause the schema enforcement to silently fall back to unconstrained generation:

python

import google.generativeai as genai

import json

def gemini_structured_call(prompt: str, response_schema) -> dict:

model = genai.GenerativeModel(

"gemini-2.0-flash",

generation_config=genai.GenerationConfig(

response_mime_type="application/json",

response_schema=response_schema

)

)

response = model.generate_content(prompt)

# Always repair before parse — Gemini falls back silently

content = response.text

try:

return json.loads(content)

except json.JSONDecodeError:

# Strip fences, repair, retry

content = content.strip("` \n")

if content.startswith("json"):

content = content[4:]

return json.loads(content)

### Failure 2: Experimental Model Versions

Preview and experimental Gemini model versions have less reliable schema enforcement than GA versions. Pin to GA model versions (`gemini-2.0-flash`, not `gemini-2.0-flash-exp`) in production.

### Failure 3: Integer vs Number Type Coercion

Gemini sometimes returns integer values as floats even when the schema specifies `"type": "integer"`. A field defined as `{"type": "integer"}` may come back as `1.0` instead of `1`. This passes JSON.parse() but fails strict schema validators. Always use a lenient validator for Gemini output:

python

Instead of strict integer check:

jsonschema.validate(data, schema) # Fails on 1.0 vs 1

Use coercion-aware validation:

from pydantic import BaseModel

class MyOutput(BaseModel):

count: int # Pydantic will coerce 1.0 → 1

result = MyOutput(**data)

---

## Mistral and Groq: json_object Mode Limitations

Mistral's `response_format: {type: "json_object"}` and Groq's equivalent are JSON Mode (syntax enforcement only) — not Structured Outputs (schema enforcement). They guarantee valid JSON syntax but not schema conformance. This is the same as OpenAI's legacy JSON Mode.

Common failures:
- Model returns `{}` or `{"error": "..."}` instead of your expected structure
- Missing required fields without any error
- Integer fields returned as strings

The repair pattern here is schema validation + retry, not JSON syntax repair:

typescript

import Groq from 'groq-sdk';

import { z } from 'zod';

const client = new Groq();

const OutputSchema = z.object({

title: z.string(),

items: z.array(z.string()),

count: z.number()

});

async function groqStructuredCall(prompt: string) {

const response = await client.chat.completions.create({

model: "llama-3.3-70b-versatile",

messages: [

{

role: "system",

content: Return ONLY valid JSON matching this schema: ${JSON.stringify(OutputSchema.shape)}

},

{ role: "user", content: prompt }

],

response_format: { type: "json_object" }

});

const content = response.choices[0].message.content ?? "{}";

// Parse (JSON Mode guarantees syntax)

const raw = JSON.parse(content);

// Validate schema — JSON Mode doesn't enforce this

const result = OutputSchema.safeParse(raw);

if (!result.success) {

throw new Error(Schema violation: ${result.error.message});

}

return result.data;

}

---

## Local Models: The Fallback Stack

Local models via Ollama, vLLM, or llama.cpp have optional schema enforcement — and it's much less reliable than cloud providers. Even with `format: "json"` in Ollama or `--guided-json` in vLLM, smaller models (under 7B parameters) fail schema compliance at much higher rates.

See [local LLM JSON output debugging](/blog/local-llm-json-output-debugging) for the full playbook. The short version: always run the validate-and-repair-and-retry pattern for local models, and use Q8 quantization or higher for complex schemas.

---

## Universal Repair Pattern (Python)

Here is a production-ready pattern that works across all providers:

python

import json

import re

import requests

from typing import TypeVar, Type

from pydantic import BaseModel, ValidationError

T = TypeVar("T", bound=BaseModel)

def parse_llm_structured_output(

content: str,

schema: Type[T],

max_retries: int = 1

) -> T:

"""

Parse LLM structured output with repair fallback.

Works across OpenAI, Claude, Gemini, Groq, and local models.

"""

# Pass 1: Clean and parse

cleaned = _clean_content(content)

try:

data = json.loads(cleaned)

return schema(**data)

except (json.JSONDecodeError, ValidationError):

pass

# Pass 2: Remote repair via aijsonmedic

repaired = _repair_json(cleaned)

if repaired:

try:

data = json.loads(repaired)

return schema(**data)

except (json.JSONDecodeError, ValidationError):

pass

raise ValueError(f"Structured output repair failed after {max_retries} attempts. Raw: {content[:500]}")

def _clean_content(content: str) -> str:

"""Strip markdown fences and normalize Python literals."""

content = content.strip()

# Strip ``json ... `` fences

if content.startswith("```"):

parts = content.split("```")

content = parts[1] if len(parts) > 1 else content

if content.startswith("json\n"):

content = content[5:]

# Normalize Python literals

content = re.sub(r'\bNone\b', 'null', content)

content = re.sub(r'\bTrue\b', 'true', content)

content = re.sub(r'\bFalse\b', 'false', content)

return content.strip()

def _repair_json(content: str) -> str | None:

"""Attempt repair via aijsonmedic API."""

try:

response = requests.post(

"https://aijsonmedic.com/api/repair",

json={"json": content},

timeout=5

)

if response.ok:

return response.json().get("repaired")

except Exception:

pass

return None

### TypeScript Version

typescript

import { z } from 'zod';

async function parseLLMOutput(

content: string,

schema: z.ZodType

): Promise {

// Clean: strip fences, normalize literals

let cleaned = content.trim();

if (cleaned.startsWith('```')) {

const parts = cleaned.split('```');

cleaned = parts[1] ?? cleaned;

if (cleaned.startsWith('json\n')) cleaned = cleaned.slice(5);

}

cleaned = cleaned.trim();

// Parse

let data: unknown;

try {

data = JSON.parse(cleaned);

} catch {

// Repair via API

const repaired = await repairJSON(cleaned);

data = JSON.parse(repaired);

}

// Validate schema

return schema.parse(data);

}

async function repairJSON(broken: string): Promise {

const res = await fetch('https://aijsonmedic.com/api/repair', {

method: 'POST',

headers: { 'Content-Type': 'application/json' },

body: JSON.stringify({ json: broken })

});

if (!res.ok) throw new Error('Repair API failed');

const { repaired } = await res.json();

return repaired;

}

```

---

Failure Rate by Schema Complexity

The more complex your schema, the higher the failure rate — even with strict enforcement:

Source: Aggregated from production pipeline data shared across dev.to, dasroot.net, and community benchmarks.

---

When to Validate vs. When to Repair

Both matter, but they catch different problems:

JSON repair fixes syntax: unclosed brackets, trailing commas, Python literals, markdown fences, truncated output. Use it after JSON.parse() fails. Schema validation catches business logic violations: missing required fields, wrong types, values outside allowed ranges. Use it after JSON.parse() succeeds.

The full production pipeline: raw output → clean (strip fences) → parse → [fail: repair → re-parse] → validate against schema → [fail: retry LLM with correction prompt]

You need both stages. Repair handles what the parser can't read; validation handles what the schema requires.

---

Quick Reference: Provider Failure Patterns

---

FAQ

Does OpenAI Strict Mode guarantee valid JSON?

OpenAI Strict Mode (response_format: {type: json_schema, json_schema: {..., strict: true}}) guarantees schema-conformant JSON in most cases, but still fails when the output is truncated at max_tokens, when the model issues a refusal object instead of JSON, or in streaming mode where enforcement is not applied to the assembled stream. Always validate after parsing.

Why does Claude's structured output still produce malformed JSON?

Claude's output_config.format: {type: 'json_schema', json_schema: {...}} can still produce malformed JSON in three scenarios: very complex nested schemas exceeding 10+ levels of nesting, outputs approaching the 200k context window limit (truncation), and tool_use JSON schema violations when input_schema has a union type without explicit null. Run a repair pass before JSON.parse() in production.

What's the fastest way to repair LLM structured output in Python?

Use a try/repair/retry pattern: parse with json.loads() first, then on failure apply jsonrepair or aijsonmedic's repair API, then re-parse. If the repaired output still fails schema validation, log and retry the LLM call with an explicit error correction prompt. This pattern handles 90%+ of structured output failures in production.

Does Gemini's response_schema prevent JSON errors?

Gemini's responseSchema with responseMimeType: 'application/json' provides constrained decoding for most outputs but fails when the schema contains recursive references, when the output token limit is hit, or when using preview/experimental models. Repair is still required in production pipelines.

How do I handle structured output failures in streaming mode?

Streaming mode disables structured output enforcement in all major providers as of mid-2026. Assemble the full stream, then run the complete string through repair (strip fences, normalize quotes, close truncations), then parse and validate. Never attempt JSON.parse() on partial stream chunks.

What's the difference between JSON Mode and Structured Outputs in OpenAI?

JSON Mode (response_format: {type: 'json_object'}) guarantees valid JSON syntax but does NOT enforce any schema — the model can return any valid JSON. Structured Outputs (response_format: {type: 'json_schema'} with strict: true) enforces both JSON syntax and schema conformance. In 2026, JSON Mode is considered legacy; use Structured Outputs for all new production code.

---

Summary

Structured output enforcement reduces LLM JSON errors to under 2% in most cases. That's good enough for demos, not good enough for production. Every provider has specific edge cases — truncation, refusal objects, streaming mode gaps, schema complexity limits — that require a repair layer.

The pattern that works: clean → parse → repair on failure → validate schema → retry LLM on validation failure. Apply it consistently and you'll handle the long tail of structured output failures that enforcement misses.

Use AI JSONMedic's JSON repair tool for one-click repair, or the JSON validator to check your structured output against your schema before it hits your business logic.

For related guides, see LLM Structured Output Libraries (2026) for library selection, Local LLM JSON Output Debugging for self-hosted model issues, and Pydantic Validation Errors from LLM JSON for the Python validation side.