Structured Output & JSON Mode

Why This Matters

Every production system that calls an LLM to extract data, fill a form, or drive a downstream API eventually hits the same wall: the model is a next-token predictor, not a data-entry clerk. Ask it for JSON and, most of the time, it produces JSON. But "most of the time" is not a contract a payments pipeline, a database write, or a function call can run on. A stray trailing comma, an extra sentence of preamble ("Sure, here's the JSON:"), a missing required field, or a hallucinated enum value breaks the parser. Structured output closes that gap by making schema conformance a property of how the model generates tokens, not something you hope for and validate after the fact.

Everyday Analogy

Think of the difference between asking someone to "fill out a form in your own words" versus handing them an actual form with labeled boxes: name here, date there, checkbox for yes or no. The first approach produces something readable but inconsistent; the second produces something a machine can process every time because the shape of the answer is fixed before the person starts writing. Structured output is the actual form: the model fills in labeled boxes instead of writing free-form prose that happens to resemble one.

Why Raw Text Output Fails in Production

Early LLM integrations asked for JSON purely through prompting: "Respond only in JSON matching this schema." This works often enough to demo but fails in predictable ways at scale:

• Formatting drift. Models add markdown code fences, explanatory preambles, or trailing commentary that breaks a naive json.loads() call.
• Schema drift. A field name gets slightly renamed, a number is returned as a string, or an optional field is omitted when your code expects it present.
• Invalid enums. Asked to classify sentiment as positive, negative, or neutral, the model occasionally emits mixed or somewhat positive, values your downstream code was never built to handle.
• Nesting errors. Deeply nested schemas increase the odds of a missing bracket or malformed array as generation length grows.

None of this is a training gap the next model version reliably fixes; it is a structural mismatch between free-text generation and machine-readable contracts.

Two Approaches: Prompting vs. Constraining

There are two fundamentally different ways to get structured output, and the distinction matters:

• Prompt-based JSON mode. You ask nicely and the model tries its best. Many providers offer a response_format: json_object flag that biases the model toward valid JSON syntax but does not guarantee it matches your specific schema, only that it is syntactically valid JSON of some shape.
• Schema-enforced constrained decoding. The generation process itself is restricted so that at every token step, only tokens consistent with the target schema are eligible. The model cannot produce a token that would create an invalid enum value, a wrong type, or a malformed structure, because that token is never in the sampling pool to begin with.

The second approach is the meaningful upgrade: it moves validation from "check after the fact and retry" to "make invalid output unreachable."

Real Tools That Do This

A Worked Example: Extracting a Support Ticket

Say a support inbox forwards raw customer emails and you need { "category": "billing" | "technical" | "account", "urgency": "low" | "medium" | "high", "summary": string } for each one.

1. Naive prompting. You write "extract this as JSON" in the prompt. It works for 95 out of 100 emails; the other 5 return malformed JSON, an extra field, or an urgency value like "urgent" that isn't in your enum, and your pipeline crashes on ticket 37.
2. Schema-enforced call. You define the schema once (in JSON Schema or a Pydantic/Zod model) and pass it to OpenAI's Structured Outputs, Anthropic's tool use, or wrap the call with Instructor. The model's decoding is constrained so urgency can only ever be one of your three literal strings, category can only be one of your three literal strings, and all three fields are always present.
3. Downstream code. Your code calls .category, .urgency, .summary directly on a validated object, no defensive try/except around a JSON parse, no regex to strip markdown fences, no retry loop for malformed output.

This is the same underlying mechanism whether you call it "JSON mode," "structured outputs," or "tool-based extraction": at generation time, the model's next-token choices are filtered to only those consistent with your schema.

Real-World Showcase

• OpenAI's Structured Outputs feature reports near-100% schema conformance in evaluation benchmarks once strict mode is enabled, versus a meaningfully lower rate for prompt-only JSON requests on the same tasks.
• Instructor has become a default starting point for Python teams doing LLM-based data extraction specifically because it turns "define a Pydantic model" into "get a validated instance back," with automatic retries that re-prompt the model using the validation error itself as feedback.
• Outlines is widely used by teams self-hosting open-weight models (via vLLM or Hugging Face's transformers) who need schema guarantees but don't have access to a managed provider's structured-output endpoint.

Try It Yourself

1. Take a JSON schema you'd use for a real extraction task (invoice line items, resume fields, or support ticket triage) and write it out as a Pydantic model or Zod schema.
2. Compare, on paper, what happens with prompt-only JSON mode versus schema-enforced constrained decoding when the model tries to emit an enum value that isn't in your schema — one silently produces bad data, the other structurally cannot.
3. Sketch how you'd wire Instructor's automatic retry-with-validation-feedback loop into a support-ticket extraction pipeline, including what happens after three failed retries.

Common Mistakes

• Treating provider "JSON mode" (syntactically valid JSON) as equivalent to schema-enforced structured output (valid JSON matching your exact schema) — they are not the same guarantee.
• Skipping runtime validation entirely because "the API guarantees the schema," when a schema can still contain semantically wrong values (a fabricated invoice number that is technically a valid string).
• Designing schemas with ambiguous or overlapping enum values that confuse the model even when decoding is constrained to that enum's token set.
• Using structured output for tasks that genuinely need free-form reasoning, then complaining the output feels stilted, when a lighter prompt-based approach with post-hoc parsing would fit better.
• Not versioning schemas alongside prompts, so a schema change silently breaks every downstream consumer parsing the old shape.

Key Takeaways

• Raw LLM text output is unreliable for programmatic use because next-token generation was never a hard contract with your schema.
• Prompt-based JSON mode biases toward valid syntax; schema-enforced constrained decoding makes invalid structure unreachable at the token level.
• OpenAI Structured Outputs, Anthropic tool use, Instructor, and Outlines are the real tools teams use today, spanning managed APIs to self-hosted open models.
• Structured output builds directly on the function-calling machinery from AI-018 and depends on picking a model, per AI-024, that supports it well.
• Validate schema conformance is not the same as validating semantic correctness; you still need both.