Evaluation

Before relying on a schema, measure it on documents whose answers you know. The evaluation tool sends a labelled dataset to the API, compares every field with the expected value, and recommends a review_threshold.

python -m typed.evaluate my-dataset.json --url https://api.trusttyped.com

Note

The tool ships with the server package. Run it on a machine where the package is available (ask your administrator), pointing --url at the deployment.

Dataset format

One JSON file:

{
  "name": "Receipts",
  "system": "You extract data from receipts. Write numbers without separators.",
  "max_new_tokens": 48,
  "schema": {
    "merchant": {"type": "string", "extract": true},
    "total":    {"type": "number", "integer": true, "max_digits": 9},
    "paid":     {"type": "bool"}
  },
  "tolerance": {"total": 1},
  "cases": [
    {"id": "c01",
     "context": "Sunrise Cafe ... Total 39600. Paid by cash.",
     "expected": {"merchant": "Sunrise Cafe", "total": 39600, "paid": true}}
  ]
}

• Only fields present in expected are scored.
• Text is compared after trimming, collapsing spaces, ignoring case and a trailing period. Numbers use tolerance (default: exact). Lists are compared as multisets unless "ordered_lists": true.
• A case may override schema, system or max_new_tokens.

Options

Option	Meaning
--url	The deployment to test (default http://127.0.0.1:8000).
--target	Accuracy that auto-accepted fields must reach (default 0.99).
--limit N	Only the first N cases.
--out report.json	Save the full report, including every field of every case.

Reading the report

Calibration (a field auto-passes when confidence >= threshold, ...):
   threshold  auto  accuracy  false pass  to review  false catch
         0.5  96.0%     99.0%           1          4            0
         0.7  95.0%    100.0%           0          5            1

Recommended review_threshold = 0.7: 95.0% of fields auto-pass with 100.0% accuracy (target 99%).

• auto: share of fields that would pass without review.
• accuracy: accuracy of those fields.
• false pass: errors that would reach your system unreviewed.
• false catch: errors that would go to review.

The report also lists every error with its confidence. Errors with high confidence are misunderstandings; fix the instruction, not the threshold.

A workflow that works

1. Collect real documents and label them (a few hundred for a decision, 20 to find problems).
2. Run the evaluation and read the errors.
3. Fix instructions and add rules; run again.
4. Only then set review_threshold from the calibration table.
5. Keep some labelled documents you never looked at while tuning, and check the final numbers on those.

In one iteration on 20 receipts, field accuracy went from 95% to 99% by adding votes and clarifying one instruction ("VAT, PB1, Tax, including 0"). Twenty cases are enough to find such problems, not to prove 99%.