On-Device LLMs and JSON Output: Structured Output Reliability at 14MB (Needle2)

Needle2 landed on Hacker News on August 11, 2026 and immediately hit the top of the feed: a 14MB agentic LLM that runs a full session in 28MB of RAM, designed for phones, wearables, smart home devices, and microcontrollers. At 45M parameters compressed to 2-bit with Cactus Quants, it benchmarks competitively against models 5× to 70× larger on tool calling tasks.

The HN thread surfaced a question that developers building agentic pipelines on tiny hardware have been wrestling with for months: when an on-device LLM calls tools and returns JSON, how reliable is that structured output — and what do you do when it breaks?

This guide is the answer.

Why Needle2 Matters for JSON Output

Needle2 isn't a general-purpose chat model. Cactus built it explicitly for tool calling, device use, and structured extraction — all of which live or die on reliable JSON output.

An agentic pipeline on a phone or wearable might look like this:

User: "Turn off the lights in the kitchen and set a 10-minute timer."
→ On-device LLM (Needle2) parses intent
→ Emits JSON tool calls: [{"tool": "smart_home", "action": "lights_off", "room": "kitchen"}, {"tool": "timer", "duration": 600}]
→ Device runtime executes the calls

If the JSON is malformed, the pipeline breaks silently. There's no cloud API to absorb the error. There's no 8K-parameter heavyweight to retry with. The JSON has to be right the first time, or the device has to recover gracefully.

That recovery behavior is the focus of this guide.

The Structured Output Challenge at 14MB

Before diagnosing failures, it helps to understand what makes tiny models structurally different from cloud models when it comes to JSON output.

Quantization and JSON Reliability

Needle2 uses CQ2-bit (Cactus Quants), fitting 45M parameters into a 14MB binary. That's approximately 2 bits per weight value. For comparison, most cloud-serving models run in bfloat16 (16 bits) or at minimum int8 (8 bits).

What does bit-depth have to do with JSON? More than you'd expect.

The model's ability to distinguish between similar tokens — "action" vs "actions", true vs "true", null vs "" — depends on weight precision. At 2-bit quantization, the decision boundaries between similar tokens compress. The model still "knows" the schema structure it was trained on, but under adversarial prompts or ambiguous context, it confuses similar-looking fields more often than a higher-precision model would.

This failure mode is called schema drift: the output is valid JSON, but the field names or values deviate from your expected schema.

Context Window Constraints

Tiny models have shorter context windows. When your prompt (system instructions + user message + tool schema) approaches the context limit, the model starts compressing earlier reasoning steps. This is when truncated JSON appears:

{"action": "lights_off", "room": "kitchen", "confirm

The model ran out of effective context and stopped generating mid-string. This is distinct from schema drift — truncation produces syntactically invalid JSON that fails immediately at parse time.

No Server-Side Retry

Cloud APIs (OpenAI, Anthropic, Gemini) handle retry logic server-side and give you structured output guarantees. On-device inference has no such guarantee. Every retry is a new inference pass — battery burn, latency, and often not an option on time-sensitive agentic tasks.

This changes the math on validation strategy: on-device pipelines must tolerate more failures at the repair layer rather than the retry layer.

What Breaks and How to Fix It

Failure Mode 1: Truncated JSON

Symptoms: JSON ends mid-string, mid-key, or mid-array. json.parse() throws immediately. Cause: Context window exceeded, or max_tokens too low relative to expected output length. Fix:

import json

def safe_parse(raw_output: str) -> dict | None:
    """Attempt to parse; try repair on failure."""
    try:
        return json.loads(raw_output)
    except json.JSONDecodeError:
        # Attempt structural repair before giving up
        repaired = repair_json(raw_output)  # jsonrepair or AI JSONMedic API
        try:
            return json.loads(repaired)
        except json.JSONDecodeError:
            return None  # escalate to retry

For Needle2 specifically, keep your tool schemas as flat as possible. Deeply nested JSON requires more tokens to complete, and the last few tokens are where truncation bites.

Failure Mode 2: Schema Drift

Symptoms: JSON parses successfully but fails Pydantic/Zod validation. Fields have wrong names, wrong types, or missing required keys. Example:

// Expected
{"tool": "timer", "duration": 600}

// Actual (schema drift)
{"tool": "timer", "time": 600}

Both parse. Only one matches your schema.

Cause: 2-bit quantization compressing the weight space between similar field tokens (duration vs time vs seconds). Fix: Constrained decoding at the token level when your runtime supports it, plus post-generation validation.

import { z } from "zod";

const TimerCall = z.object({
  tool: z.literal("timer"),
  duration: z.number().int().positive(),
});

function validateToolCall(raw: unknown): z.infer<typeof TimerCall> {
  const result = TimerCall.safeParse(raw);
  if (!result.success) {
    // Log drift, attempt repair or retry
    console.error("Schema drift:", result.error.flatten());
    throw new Error("Tool call schema drift");
  }
  return result.data;
}

For on-device deployments where constrained decoding isn't available, keep enum values and field names highly distinctive in your schema. The more similar duration and time look to the model's compressed weight space, the more often it will drift.

Failure Mode 3: Markdown Contamination

Symptoms: JSON is wrapped in markdown code fences:

json

{"tool": "timer", "duration": 600}

Cause: The model was trained on markdown-formatted data and injects fences by default. Common across all model sizes but more frequent in models trained on code-heavy data. Fix:

import re

def strip_markdown_fences(text: str) -> str:
    """Remove

json ... `` or ` ... `` wrappers."""

text = text.strip()

# Remove opening fence

text = re.sub(r'^```(?:json)?\s*\n?', '', text)

# Remove closing fence

text = re.sub(r'\n?```\s*$', '', text)

return text.strip()

This should run before your JSON parse attempt, not after. Fence stripping is cheap; failed parses are not.

### Failure Mode 4: Boolean and Null Confusion

**Symptoms:** JSON parses but booleans appear as strings, nulls appear as `"null"`, or Python-style `True`/`False`/`None` appear instead of JSON-standard `true`/`false`/`null`.

json

// Broken

{"enabled": "True", "value": "None"}

// Expected

{"enabled": true, "value": null}

**Cause:** Under aggressive quantization, the model confuses JSON literals with their Python/JavaScript string representations. This is especially common in models fine-tuned on Python-heavy corpora.

**Fix:**

python

import json

import re

def normalize_json_literals(raw: str) -> str:

"""Replace Python-style literals with JSON-standard equivalents."""

raw = re.sub(r'\bTrue\b', 'true', raw)

raw = re.sub(r'\bFalse\b', 'false', raw)

raw = re.sub(r'\bNone\b', 'null', raw)

return raw

Apply this before JSON parse. [Boolean and null errors from LLMs](/blog/llm-python-boolean-json-errors) are one of the most common failure modes across all model sizes, and tiny models hit them more frequently.

## The Repair-First Pattern for On-Device Pipelines

Given the constraints of on-device inference — no server-side retry, battery cost, latency sensitivity — the recommended pattern is:

Raw LLM output

│

▼

Strip markdown fences

│

▼

Normalize literals (True→true, None→null)

│

▼

Try json.parse()

│

┌──┴──────────┐

Success? No

│ │

▼ ▼

Validate Attempt repair

(Zod/Pydantic) (jsonrepair / AI JSONMedic)

│ │

├─OK ▼

│ Try json.parse()

▼ │

Done ┌──┴──────────┐

Success? No

│ │

Validate Retry

(Zod/Pydantic) inference

│

┌──┴──────────┐

OK Drift

│ │

Done Log + fallback

The key insight: **repair is cheaper than inference retry**. Fixing a truncated bracket costs microseconds. Re-running inference on a phone CPU costs seconds and battery. Invest in the repair layer before escalating to retry.

For online repair, [AI JSONMedic's repair tool](/json-repair) handles all four failure modes above — truncation, extra commas, unquoted keys, and boolean normalization — without sending data to a server. This matters for on-device use cases where privacy is a concern and network is unavailable.

## Needle2's Constrained Decoding Advantage

Needle2 was explicitly designed for tool calling, and its engine includes constrained decoding for tool call schemas. This means it enforces structural validity at the token level for predefined tool schemas — the model physically cannot emit tokens that break the schema structure.

This is a significant advantage over models adapted for tool calling as an afterthought. Constrained decoding eliminates Failure Modes 1 (truncation mid-structure) and 3 (markdown contamination) for defined schemas.

What it doesn't eliminate: **semantic drift**. The model can emit a perfectly valid JSON object that passes schema validation but contains a hallucinated value — `"room": "bedroom"` when the user said kitchen. This is an inference quality problem, not a formatting problem, and repair tools cannot fix it. Your application logic needs to handle semantic validation separately.

## Practical Patterns for Agentic Mobile Pipelines

### Keep Schemas Flat

Nested JSON requires more tokens and more "memory" across the context window:

json

// Avoid deep nesting on tiny models

{

"action": {

"type": "smart_home",

"parameters": {

"command": "lights",

"state": "off",

"room": "kitchen"

}

}

}

// Prefer flat schemas

{

"action": "smart_home_lights_off",

"room": "kitchen"

}

Flat schemas reduce token count, reduce schema drift risk, and are easier to validate.

### Use Distinctive Field Names

Field names that sound similar increase drift risk at 2-bit precision. Choose names that are semantically distinct:

json

// Risky (duration, time, seconds — too similar)

{"duration_seconds": 600}

// Better (unique compound names)

{"timer_seconds": 600}

### Pre-validate at Prompt Time

If you know the model will always emit one of N tool schemas, send the exact schema in the system prompt and instruct the model to match it exactly. For Needle2's tool call mode, this is handled automatically — but for general JSON extraction, an explicit example in the prompt reduces drift.

### Repair as the Safety Net

The combination of constrained decoding (for structure) + semantic validation (Pydantic/Zod) + repair-on-failure (AI JSONMedic or jsonrepair) gives you three layers of defense:

1. **Layer 1:** Constrained decoding prevents syntax errors
2. **Layer 2:** Schema validation catches semantic drift
3. **Layer 3:** Repair recovers truncated or partially broken outputs before retry

This three-layer approach means the retry path — expensive on mobile hardware — is a last resort rather than the first response to failure.

## The Broader Trend: Edge LLMs and the JSON Problem

Needle2 is part of a wave of ultra-small models designed for on-device agentic use. Apple Intelligence, Gemini Nano, and now Cactus Needle2 all target the same use case: local inference for real-time, privacy-sensitive, latency-critical tasks.

All of them share the JSON problem. The smaller the model, the more aggressive the quantization, the higher the drift rate for structured output. The [LLM structured output landscape in 2026](/blog/llm-structured-output-json-repair) has solved the cloud-side problem (every major provider offers token-level enforcement). The on-device side is still catching up.

For now, the practical answer for developers building on Needle2 and similar models is the same as it's always been for [local LLM JSON debugging](/blog/local-llm-json-output-debugging): **don't trust raw output, validate everything, and keep a repair fallback ready**.

The model is doing its best in 28MB of RAM. Your application stack needs to cover the rest.

## Quick Reference: On-Device JSON Failure Modes

| Failure Mode | Symptom | Fix |
|---|---|---|
| Truncation | JSON ends mid-string | Repair + retry |
| Schema drift | Valid JSON, wrong fields | Constrained decoding + Zod/Pydantic |
| Markdown fences | `

json `` wrapper | Strip before parse |

| Boolean/null confusion | True, None literals | Normalize before parse |

| Semantic hallucination | Right schema, wrong values | Application-layer semantic validation |

Wrapping Up

Needle2 landing on HN with 110 points is a signal that on-device agentic LLMs are transitioning from research curiosity to production reality. The 14MB binary that runs a full agentic session in 28MB of RAM is genuinely impressive engineering.

For developers building on it: the JSON problem is solved with the same tools and patterns that work for larger local models, just with tighter tolerances. Constrained decoding for structure, semantic validation for values, and a repair fallback for the cases where both miss.

Try the repair step online at AI JSONMedic — paste any broken on-device LLM output and it repairs client-side, no data leaves your browser.

---

Related: LLM JSON Repair Guide · Local LLM JSON Debugging (Ollama, vLLM, llama.cpp) · LLM Structured Output + JSON Repair