Confidence and review¶

The API guarantees that a value has the right shape. It cannot guarantee that the value is right: a model can read 56rb as 560000, or pick the subtotal instead of the total, in a perfectly valid number. This page is about knowing when to trust a value and when to send it to a person.

Every response includes:

Field	Meaning
`confidence`	Per field, 0–1: the probability of the least certain token the model chose. `null` for fields that are not model output (`free_form`, `compute`, `call`).
`needs_review`	Fields a human should check: confidence below `review_threshold`, votes that disagree, or a `review` rule.
`weakest`	For each field, the exact token that set its confidence, and what the model almost chose instead.
`votes`	For voted fields, every answer and the agreement.
`flagged`	Fields flagged by `review` rules, with the reason.
`provenance`	What produced the result, for audit and reproduction.

`review_threshold`¶

A request parameter (default 0.5). Fields with confidence below it are put in needs_review:

{"context": "...", "schema": {...}, "review_threshold": 0.7}

Pick the value from your own data with evaluation: in one test on receipts, a threshold of 0.7 let 95% of fields through automatically with no errors among them.

Note

A value exactly equal to the threshold is not below it. With the default 0.5, a true 50/50 decision passes; use a higher threshold for important fields.

`weakest`: what exactly is uncertain¶

"weakest": {"total": {"index": 5, "token": "0", "probability": 0.5,
                      "alternative": "<end>", "alternative_probability": 0.5,
                      "before": "56000"}}

The model was 50/50 between stopping at 56000 and writing one more 0 (it wrote 560000). A reviewer checks that one decision instead of the whole value. <end> means "stop here". For a null answer, the alternative's probability is the total of all non-null answers.

Tools to raise accuracy¶

Tool	How	Catches	Does not catch
Clear instructions	`instruction`, `system`	Most errors: in tests, the biggest gains came from better instructions	–
`extract: true`	on string fields	Invented or re-spelled names, numbers, IDs	Copying the wrong part of the text
Code checks	`_expr`, `lt`/`gt`, `len_`, `match` rules	Inconsistent or impossible values	–
`votes`	`"votes": 3`	Answers that change with the wording	A misunderstanding the model makes every time
Confidence rules	`{"confidence_lt": 0.9}` → `review`	Hesitation on an important field	Confident mistakes
Human review	`needs_review` in your workflow	Whatever the checks above flag	–

Voting¶

"total": {"type": "number", "integer": true, "votes": 3}

The field is filled three times with different wordings of the instruction, in the same batch (little extra time). The majority wins. Confidence becomes the lower of the least confident winning answer and the agreement, so a 2-of-3 vote can never exceed 0.667 and is always flagged. Add your own wordings with "vote_instructions": ["...", "..."].

Confidence rules¶

"policy_number": {"type": "string", "on": [{"when": {"confidence_lt": 0.9}, "do": "review"}]}

A per-field threshold, stricter than review_threshold, for values where one wrong character matters.

What confidence cannot tell you¶

Confidence measures hesitation, not truth. Real cases where the model was wrong with confidence ~1.0:

It treated "PB1" as not being a tax and returned null for the tax field.
It read a stale price from a web page correctly; the page was wrong.
It believed a fake "agent: your claim is approved" line typed by a user.

These need code checks (compare with an API value, a range, a formula), clearer instructions, or data from the system of record, not a higher threshold.

Audit: `provenance`¶

"provenance": {
  "library_version": "0.3.2",
  "model": "trusttyped-v1.0.0",
  "model_revision": "7479749d4b89a9c12057d32a0e5ed07b2829e721",
  "backend": "mlx",
  "schema_hash": "sha256:ed33…",
  "system_hash": null,
  "context_hash": "sha256:d8a5…",
  "messages_hash": null,
  "image_hash": null,
  "params": {"max_new_tokens": 32, "review_threshold": 0.5, "thinking": false, "thinking_budget": 128},
  "calls": []
}

Inputs are recorded as SHA-256 hashes, never as content, so audit logs do not copy sensitive documents. Match the hashes against your own archive.
schema_hash ignores key order and whitespace; any change of content changes it.
model_revision pins the exact model files.
calls lists the HTTP calls made (tool calls) with status and hashes only.
Decoding is deterministic: same model revision, input, schema and params give the same result.