Structured Output Beats Prompt Tweaking: A Practical Eval Loop

We spent two weeks rewriting prompts. Then we changed the response format, and the failure rate dropped by half.

That is not a story about prompt engineering being useless. It is a story about measuring the wrong thing.

The problem

Our extraction endpoint returned prose that we parsed with a regex. When the model changed its phrasing, the regex broke quietly, and the pipeline wrote empty fields into the database.

We counted those as prompt problems for ten days. They were parsing problems wearing a costume.

The change

We switched the endpoint to structured output and made the schema the contract:

schema = {
    "type": "object",
    "properties": {
        "vendor": {"type": "string"},
        "amount": {"type": "number"},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
    },
    "required": ["vendor", "amount", "currency"],
}

Now a malformed response is a validation error we can count, instead of a silent empty field.

The eval loop

The part that actually mattered was the loop around it:

1. Fifty hand-labelled examples, including the ugly ones.
2. Run the new prompt and the old prompt over the same set.
3. Compare field-level accuracy, not exact string match.

A human judged the first pass, because the disagreements were the interesting part.

What I would do differently

I would build the eval set before touching the prompt. Two weeks of prompt work is cheap to run and expensive to interpret, especially when you have no baseline to interpret it against.