Computer Use Models and JSON: When GUI Automation Agents Produce Malformed Action Schemas

Computer-use models don't browse the web or click buttons directly. They emit JSON.

Every action a computer-use agent takes — click this element, type this string, scroll to this position, take a screenshot — is encoded as a structured JSON object before it reaches a GUI automation framework. When that JSON is malformed, the automation stops. There is no error recovery built into most frameworks, and the agent typically has no way to know whether its action was executed at all.

This post, written by The JSON Medic, examines how computer-use models like CUA-S1 produce action schemas, what causes those schemas to break, and what repair strategies work in production.

---

What CUA-S1 Is

In September 2026, the team at trycua.com published CUA-S1 on Hacker News — a "System One model" for computer use that reached 57 points and an active discussion. The project (github.com/trycua/cua) positions itself as a fast, low-latency computer-use agent: observe screen state, decide on an action, emit it as a typed schema, repeat.

The "System One" framing is deliberate. Traditional computer-use approaches (Anthropic's computer use tool, OpenAI's CUA operator) use large general-purpose models that reason through each step. CUA-S1 is designed for heuristic, fast-path decisions — the kind of routing a human expert does reflexively, not deliberatively. Low latency is the primary design goal, which means it trades raw reasoning depth for throughput.

This framing has a direct consequence for JSON reliability. Fast models under latency pressure are more likely to emit partial or abbreviated schemas. The faster the inference, the shorter the sampling window, and the more likely a long structured output gets truncated at a cost boundary.

---

The JSON Schema of a GUI Action

Computer-use models typically emit one of a small set of action types. A representative schema for a click action looks like this:

{
  "action": "click",
  "coordinate": [847, 412],
  "button": "left",
  "clicks": 1
}

A type action looks like:

{
  "action": "type",
  "text": "[email protected]"
}

A scroll action:

{
  "action": "scroll",
  "coordinate": [760, 500],
  "direction": "down",
  "amount": 3
}

Each action schema has strict requirements. coordinate must be a two-element integer array — not a string, not an object, not three elements. button must be one of "left", "right", or "middle". direction must be exactly "down" or "up", not "downward", not "Down", not a number.

The automation layer that consumes these actions — whether that's a custom headless browser controller, a pyautogui wrapper, or a commercial RPA framework — typically has zero tolerance for schema deviations. A missing key raises a KeyError. A wrong type raises a TypeError. A coordinate stored as "847, 412" (a string) instead of [847, 412] (an array) silently fails in JavaScript JSON parsers that don't validate structure.

---

Five Ways Computer-Use JSON Breaks

1. Coordinate Format Drift

Models trained on mixed data sometimes emit coordinates as strings, objects, or comma-separated values instead of integer arrays:

{"action": "click", "coordinate": "847, 412"}
{"action": "click", "coordinate": {"x": 847, "y": 412}}
{"action": "click", "coordinate": [847.3, 412.0]}

The first two are structural type mismatches. The third (floats instead of ints) is often handled by frameworks, but some strict parsers reject it.

2. Truncated Action Objects

Under latency pressure or token budget constraints, a model may stop generating mid-object:

{"action": "type", "text": "noman.ahmed@

This truncation pattern is particularly common when the text field contains long strings — email addresses, URLs, code snippets, or multi-line inputs. The JSON parser throws SyntaxError: Unexpected end of JSON input, which is also the most common error in our repair logs from structured output pipelines.

3. Invented Action Types

Models hallucinate action names that don't exist in the target framework's action space. Common examples include "hover" when the framework expects "mouse_move", "submit" when the framework expects a "click" on a submit button, or "navigate" when the framework has no navigation action.

This class of error cannot be repaired structurally — a "hover" action is valid JSON, but semantically wrong. It requires either schema validation against the allowed action space or fallback handling.

4. Missing Required Fields

The model emits a valid JSON object but omits a required field:

{"action": "click"}

No coordinate. The framework either throws on missing key access or, worse, clicks at coordinate [0, 0] — the top-left corner of the screen — silently doing the wrong thing.

5. Nested Schema Confusion

Some computer-use frameworks use nested action schemas, particularly for complex interactions like drag-and-drop or keyboard shortcuts:

{
  "action": "key",
  "key": "ctrl+c"
}

Models sometimes emit these as separate sequential actions instead of the compound form, or flatten the nested structure:

{"action": "key", "modifier": "ctrl", "key": "c"}

This looks reasonable but breaks any framework that expects the compound "ctrl+c" string format.

---

Why Repair Is Part of the Production Loop

A computer-use agent operating at CUA-S1 speeds needs to handle schema errors in the same inference loop, not after the fact. The standard recovery pattern has three tiers:

Tier 1 — Structural repair. Fix malformed JSON before parsing: close unclosed brackets, normalize string formats, remove trailing commas. This is what AI JSON Medic's repair API does — the same repair logic that handles LLM output in traditional pipelines applies directly to action schemas, because the failure modes are identical. Tier 2 — Schema validation. After structural repair, validate the action object against the known action schema. Flag missing required fields, type mismatches, and invalid enum values. Return a typed error with enough context for the retry logic to issue a corrective prompt. Tier 3 — Semantic fallback. For hallucinated action types ("hover", "navigate", "submit"), maintain a mapping from common variants to canonical names. Treat this as an alias table, not a repair: no structural fix needed, just a lookup.

In practice, most production computer-use pipelines implement Tier 1 inline (a try/catch wrapping JSON.parse with a repair call on failure) and skip Tiers 2 and 3 until a specific failure mode is observed in production logs.

---

CUA-S1 in Context: System One Models and JSON Reliability

CUA-S1's "System One" framing is interesting from a JSON reliability perspective. Anthropic's own research on structured output failure rates shows that smaller, faster models have higher error rates on complex schemas — not because they're less capable, but because they're less willing to "spend" tokens on careful formatting when the primary reward signal is task completion, not schema fidelity.

This matches what we see in the repair logs for on-device automation models. The Cactus Needle 3 analysis found that smaller automation models — the 8–29MB range that targets low-latency local execution — show a structural error rate of roughly 2–5% on complex multi-field schemas, dropping to under 0.5% on simple single-action schemas like click and scroll. For a CUA-S1 system running at 100ms per action, even a 1% structural error rate means a failure every 100 actions — roughly every 10–15 seconds in an active automation session.

This is not a rejection of fast models. It's a case for wrapping them with lightweight repair logic that adds <5ms of overhead but eliminates the hard stops that truncated JSON causes.

For computer-use pipelines built on Anthropic's API, the Claude JSON fixer handles Claude-specific output patterns — including partial tool use payloads and action schemas truncated at context limits.

---

What a Production CUA Repair Pipeline Looks Like

Here is a minimal Python implementation of the three-tier repair pattern for a computer-use action parser:

import json
from json_repair import repair_json

VALID_ACTIONS = {"click", "type", "scroll", "key", "screenshot", "mouse_move"}
ACTION_ALIASES = {
    "hover": "mouse_move",
    "navigate": None,  # No mapping — must prompt for correction
    "submit": "click",
    "press": "key",
}

def parse_cua_action(raw: str) -> dict | None:
    # Tier 1: structural repair
    try:
        action = json.loads(raw)
    except json.JSONDecodeError:
        repaired = repair_json(raw)
        try:
            action = json.loads(repaired)
        except json.JSONDecodeError:
            return None  # Tier 3 fallback: re-prompt the model

    # Tier 2: schema validation
    if "action" not in action:
        return None  # Missing required field — re-prompt

    action_type = action["action"]

    # Tier 3: semantic alias resolution
    if action_type in ACTION_ALIASES:
        if ACTION_ALIASES[action_type] is None:
            return None  # Unknown — re-prompt
        action["action"] = ACTION_ALIASES[action_type]

    if action["action"] not in VALID_ACTIONS:
        return None

    # Type coercion for coordinate field
    if "coordinate" in action:
        coord = action["coordinate"]
        if isinstance(coord, str):
            try:
                parts = [int(x.strip()) for x in coord.split(",")]
                action["coordinate"] = parts
            except ValueError:
                return None
        elif isinstance(coord, dict) and "x" in coord and "y" in coord:
            action["coordinate"] = [int(coord["x"]), int(coord["y"])]

    return action

This parser handles the five failure modes described above. The repair_json call (using the json-repair Python library) handles structural issues. The alias table handles semantic drift. The coordinate normalization handles type mismatch. The whole thing adds under 2ms of overhead on average for well-formed input.

---

Internal Linking Into the CUA Content Cluster

If you're building computer-use automation pipelines, the repair pattern described here is part of a larger structured-output toolchain:

• Validate your action schemas before they reach the automation framework
• Inspect the repair output to confirm the structural fix is semantically correct
• Review the LLM structured output validation inspection layer guide for the three-layer pipeline pattern
• See on-device LLM JSON output for the Needle2 analysis of small model reliability

The pattern is the same whether the JSON describes a database record, an API response, or a GUI action. The failure modes are structurally identical. The repair tooling transfers directly.

---

Summary

Computer-use models emit JSON. That JSON breaks in predictable ways: coordinate format drift, truncation, invented action types, missing fields, and nested schema confusion. CUA-S1's "System One" framing — fast, heuristic, low-latency — increases the probability of truncation under token pressure, which makes structural repair more important, not less.

The three-tier pattern — repair, validate, alias-resolve — adds minimal overhead and eliminates the hard stops that malformed action schemas cause in GUI automation pipelines. For systems running at CUA speeds (100ms per action), this is the difference between a 1% failure rate and a 0.1% failure rate. At scale, that matters.

Use AI JSON Medic's repair API as the Tier 1 repair layer in your CUA pipeline. The REST API handles structural repair with <10ms round-trip and returns both the repaired JSON and a structured diff of what changed.

---

FAQ

What is a computer-use model?

A computer-use model is an AI system that controls a computer by observing screen state (screenshots) and emitting structured JSON action schemas (click, type, scroll) that a framework executes. CUA-S1, Anthropic's computer use tool, and OpenAI's CUA operator are all examples.

Why does computer-use JSON break more often than API JSON?

Computer-use models operate under latency pressure and observe complex, variable screen states. Long action sequences and compound schemas (drag-drop, keyboard shortcuts) are more likely to be truncated or misformatted than simple API response objects.

Can I use AI JSON Medic's repair API in a computer-use pipeline?

Yes. The REST API endpoint accepts any malformed JSON string and returns the repaired version. Wrap it around your action parser's JSON.parse call — if the parse fails, call the repair endpoint, then re-parse.

What is the overhead of adding a repair layer?

The AI JSON Medic API adds under 10ms for a typical action schema repair (well under the 100ms inference time of fast models like CUA-S1). Local repair libraries like json-repair (Python) add under 2ms.

Does schema validation need to happen after repair?

Yes. Structural repair fixes syntax errors — missing brackets, truncated strings, trailing commas. It does not fix semantic errors like invented action types or missing required fields. Schema validation is a separate step after repair.

What should I do when repair fails completely?

If both JSON.parse and the repair attempt fail, re-prompt the model with the raw output and a correction instruction: "Your previous action schema was invalid. Return only a valid JSON object with these fields: action, coordinate. Do not include markdown or explanation." This handles the rare case of completely garbled output.