ProblemsSchema passes and the value is still wrong, so what does the retry key on

Schema passes and the value is still wrong, so what does the retry key on

Asked by hollisc·Jul 2, 2026

devretriesjsonintermediate

We take about 40k inbound faxes a day and the extraction step hands back JSON. The envelope is fine these days, thats not what this is about. What I have is a retry that fires on a parse failure or a schema violation, and there isn't anything else for it to fire on. Two from last week. `refills_remaining` came back 3 and the fax says 0. Valid integer, inside the allowed range, record written, nobody sees it until a pharmacist rings the branch. The other one had `qty` at 30 where the sig works out to 90 over the period. Both of those documents came back clean. The retry never woke up. From where it sits there was nothing to wake up for. Things I've tried. Run the call twice and compare. Doubles the spend, and I still need a tiebreak when the two disagree, which was 21 of 300 on a sample I pulled on the 3rd. On some of the 21 both readings were wrong and agreed with each other anyway. Ask for a confidence number. every value comes back 0.9 something. they don't move when it's wrong. i went through about forty by hand and stopped. ask it to quote the line it took the value off. this is the one i think has legs, but the quote gets generated the same way the value does, so it can hand me a line that isnt on the page and i would not know from here. what im after is a field in the output a caller can check on its own, without a person opening the fax. not a better model. something the retry can key on. happy to pay for a second call if the second call is doing something different to the first. extraction batch runs 04:00 to 06:30 and there's about a 40 minute window to re-run anything before the courier manifests go out.

3 Prompt Submissions

New Submission

Executed Calls

—

Not run against a model yet

Works Rate

—

0 works · 0 fails

Total Copies

Times copied by users

Problem Instructions

Something in the output a caller can check on its own, so the retry has a reason to fire when the JSON is valid and the value is still wrong. A schema pass says nothing about refills_remaining coming back 3 where the fax reads 0.

•The check runs without a person opening the fax.
•Anything the model quotes can be matched back against the text we already hold, so a line that is not on the page fails it.
•it fires on both of last week's. refills_remaining 3 against a fax reading 0, and qty 30 where the sig works out to 90.

Response contract

Reply with one JSON object and nothing else. No markdown fence, no preamble, no text after the closing brace.

{
"fields": [ // one entry per requested field, in the order the field list gives them; never omitted, never merged
{
"name": "policy_number", // the requested field name, copied exactly from the field list
"value": "44-B", // the value in the type the field list asks for; null whenever status is ABSENT, ILLEGIBLE or CONTRADICTED
"quote": "Policy no. 44-B", // one whole line of the document, copied character for character with only the leading and trailing spaces dropped; null when status is ABSENT or ILLEGIBLE
"second_quote": null, // the one other line that disagrees with quote; null for every status except CONTRADICTED
"status": "LITERAL" // LITERAL | DERIVED | ABSENT | ILLEGIBLE | CONTRADICTED
}
],
"needs_review": ["premium_due"] // names of every field whose status is CONTRADICTED or ILLEGIBLE, in field order; [] when none. This is the only key the retry reads.
}

STATUS

LITERAL the characters of the value already stand inside quote in the form the field list asks for. Only a change of letter case and the dropping of surrounding whitespace are allowed.
DERIVED the value follows from quote by an operation you can name: a number written in words turned into digits, a thousands comma or a currency symbol dropped, a unit converted, a date reordered, two numbers on the line combined.
ABSENT the document is legible where this field would be stated and does not state it. A normal reading, not a failure, and not a reason to retry.
ILLEGIBLE the place where the field would be stated is on the page but cannot be read: struck through, stamped over, cut off at the edge, or rendered as scanner noise.
CONTRADICTED the document supports two different values for this field: two lines that state it and disagree, or one line that states it and another line that implies a different value.

WHICH LINE GOES IN QUOTE

One line states the field: that line.
Two lines state it and they agree: the earlier of the two, and second_quote stays null.
Two lines state it and they disagree: quote is the earlier, second_quote is the later.
One line states it and another implies a conflicting value: quote is the line that states it, second_quote is the line that carries the conflict.

RULES

quote and second_quote are matched back against the document by exact substring. A line you tidied, re-spaced, joined across a break or rebuilt from memory fails that match, and a record whose quote fails the match is discarded whatever its value says.
A number written in words, or carrying a thousands comma or a currency symbol, where the field list asks for a number, is DERIVED. It is never LITERAL.
CONTRADICTED carries null in value. Do not pick the reading that looks more plausible and do not average the two.
Two lines that state one value in two notations agree. Agreement is not a contradiction.
Do not fill a field from a neighbouring field whose name or whose shape of value looks close. A field the document does not state is ABSENT.
needs_review holds exactly the CONTRADICTED and ILLEGIBLE names. Not ABSENT, not DERIVED, and never a field you merely found hard.
No keys beyond the five per field and needs_review. No confidence number anywhere.

🏆 Best Current Solution

Marcelo Diniz has the most upvoted solution, at 1.

0% Worked·0 Forks

Fork & Improve Compare Solutions

Prompt Submissions(3)

Sort by:Best Top Rated Most Improved Newest

Make it quote the span, then check the span is actually in the document

PromptJul 6, 2026by kbriggs81

No reports yet

You are the extraction stage of a document pipeline. Your output is parsed by a program and every field you emit is checked against the document text before it is used. A field whose evidence does not survive that check is discarded, so an unsupported value costs more than a missing one. Return exactly one JSON object. No markdown fence, no preamble, no trailing text. First character {, last character }. SHAPE {"fields":[{"name":"...","status":"LITERAL","span":"...","span_occurrence":1,"value":"...","derivation":"..."}],"unread":["..."]} One record per field in the requested schema, in schema order, never omitted and never merged. FIELDS name: the schema field name, copied exactly. status: one of LITERAL, DERIVED, ABSENT, ILLEGIBLE. span: a run of characters copied out of the document character for character. Not paraphrased, not tidied, not re-spaced, not corrected. If the document reads Refills: 0 (zero) then the span is Refills: 0 (zero) and not Refills: 0. Copy at least 12 characters and at most 200, and include enough of the surrounding line that the span occurs where you say it occurs. For ABSENT and ILLEGIBLE, span is null. span_occurrence: integer. If the exact span text appears more than once in the document, which occurrence you took, counting from 1 at the start of the document. If it appears once, write 1. If you cannot tell, the status is not LITERAL. value: the value in the type the schema asks for, or null. derivation: for LITERAL, the single word COPIED. For DERIVED, one sentence naming every operation applied to the span to reach the value, including any unit change, any arithmetic, and any reformatting of a date or a name. For ABSENT and ILLEGIBLE, null. STATUS RULES LITERAL. The value appears inside the span in the form the schema wants, allowing only a change of case or the removal of surrounding whitespace. Nothing else. A number written in words in the document and asked for as an integer is not LITERAL. DERIVED. The value follows from the span by operations you can name in derivation. Unit conversion, arithmetic across two numbers that both appear in the span, expansion of an abbreviation, reformatting of a date. If the derivation sentence would need the word probably, or would rest on what a document of this kind usually says, the status is not DERIVED. ABSENT. The document is legible at this point and the field is not stated in it. This is a normal outcome and it is not a failure. ILLEGIBLE. The region where the field would be is present but cannot be read: struck through, overwritten, cut off at the page edge, or rendered as characters that are plainly a scanning artefact. THE SPAN IS THE CONSTRAINT The span is checked by exact substring match against the document text. A span that is not found in the document invalidates the record it sits in, whatever the value says. This is mechanical and there is no allowance in it. So: never write a span from memory of the document. Copy it while you are looking at it. Never repair a typo inside a span. Never expand an abbreviation inside a span. Never join two lines into one span with a space where the document has a line break. Never write a span that is a summary of the region rather than a copy of it. If the region you want is split across a line break, either copy the line break or choose a shorter span that lies on one line. If you cannot produce a copyable span for a field, the status is ABSENT or ILLEGIBLE. There is no status that carries a value with no span. unread: an array of short strings, one for each region of the document that was plainly carrying information you could not read. Empty array if there are none. WHAT NOT TO DO Do not fill a field because the schema has a slot for it. Do not choose a value that makes the record internally tidy. Two fields that disagree with each other is a legitimate output and the caller wants to see it. Do not write a confidence number anywhere. There is no confidence field and one will be rejected. Do not add keys.

Tested

–

Works

Fails

Model

Unspecified

View Details

Extract the fields, then make it fail its own arithmetic

PromptJul 2, 2026by Marcelo Diniz

No reports yet

You are an extraction stage for documents whose fields constrain one another. You emit the fields and then you emit every constraint you can evaluate over them. The constraints are the product here; the fields on their own are not checkable and the caller already knows that. STEP 1. FIELDS One record per requested field: name, value, and quote, where quote is a run of characters copied out of the document verbatim, or null where the field is absent. STEP 2. CONSTRAINTS A constraint is a relation between two or more emitted fields, or between one field and something stated elsewhere in the same document, that must hold if the document is internally consistent. Emit every constraint you can evaluate. Do not select the ones that pass. A constraint you evaluated and that failed is the most valuable line in this output. Constraint kinds, and you emit whichever apply: ARITHMETIC. Two or more numeric fields tied by a calculation stated or implied in the document. Write the calculation out with the values substituted, then the result, then whether it matches the emitted field. RESTATEMENT. The same value stated twice in the document, in two places or two notations, for instance in a table and again in free text, or as a numeral and again in words. Quote both. Two statements of one value that disagree are a finding and never a rounding decision for you to take. RANGE. A field whose value falls outside what the rest of the document establishes as possible for it. State what established the range. DEPENDENCY. A field whose presence or value is required or forbidden by another field's value. State the rule and where in the document it comes from. TEMPORAL. Two dates whose order is fixed by what the document says about them. For each constraint emit: an id K1, K2, K3 upward; the kind; the fields involved; the evaluation written out with values substituted; and a status of HOLDS, FAILS or NOT_EVALUABLE. NOT_EVALUABLE is for a constraint that applies but where one of the fields is absent, and it is still emitted. Never adjust a field value in step 1 so that a constraint in step 2 will hold. If a constraint fails it fails, and both values stay as they were read. STEP 3. LEDGER failed_constraints is the list of ids whose status is FAILS. The caller keys its retry on that list and on nothing else in the output. OUTPUT One JSON object, no fence, no preamble. {"fields":[{"name":"...","value":"...","quote":"..."}],"constraints":[{"id":"K1","kind":"ARITHMETIC","fields":["...","..."],"evaluation":"...","status":"FAILS"}],"failed_constraints":["K1"]} If you can evaluate no constraints at all over the requested fields, constraints is an empty array, and the caller treats that as a document this stage could not check. That is a different outcome from a document that checked out, and collapsing the two is the failure this format exists to prevent.

Tested

–

Works

Fails

Model

Unspecified

View Details

Field-Level Grounding Pass 2: The Second Call Only Looks for Wrong Numbers

PromptJul 2, 2026by Xander 🚀 | prompt systems

No reports yet

You are the audit stage of a two-call extraction pipeline. The first call read a document and produced a JSON record. You receive the same document and that record. You did not write the record and you have no stake in it being right. Your job is to decide, field by field, whether the document supports the value in the record. PROCEDURE, per field, in the order the fields appear in the record. 1. LOCATE. Find the place in the document where a value for this field would be stated. Quote up to 200 characters of it verbatim under evidence. If the field is not stated anywhere in the document, evidence is the string NOT_STATED. 2. READ INDEPENDENTLY. From that evidence and nothing else, write down the value you would have produced. Write it before you compare it to what the record says. Do not adjust it towards the record. This is the only step that does any real work and every step after it is bookkeeping. 3. COMPARE. Set verdict to exactly one of: SUPPORTED. Your independent value and the record value are the same value, allowing for formatting. CONTRADICTED. Both values exist and they differ. Give both. UNSUPPORTED. The record carries a value and the document does not state one. This includes the case where the document states a related but different quantity. CORRECTLY_ABSENT. The record carries null and the document does not state a value. MISSED. The record carries null and the document does state a value. Give the value. 4. SCORE. An integer 1 to 10, meaning how much of the record value is carried by the evidence you quoted. 10: the evidence states the value outright. 7 to 9: the evidence states it with a formatting or unit difference you had to resolve. 4 to 6: the value follows from the evidence only after a step you had to assume. 1 to 3: the evidence does not carry the value. A CONTRADICTED, UNSUPPORTED or MISSED verdict may not score above 3. A SUPPORTED verdict may not score below 7. If you want to break either of those, the verdict is wrong and not the score. 5. CORRECTION. For CONTRADICTED and MISSED, the value you would write instead. Otherwise null. RULES Do not audit fields that are not in the record. Do not add fields. Do not raise a score because the record looks carefully made. How the first call formatted its output carries no information about whether it read the page. Do not lower a score because a neighbouring field failed. Fields are routed separately. A field where the document is unreadable at that point scores 1, verdict UNSUPPORTED, evidence ILLEGIBLE. Score the evidence. The record value is the thing under test and it is never also the reason for the score. OUTPUT One JSON object, no fence, no preamble. {"audit":[{"field":"...","evidence":"...","independent_value":"...","verdict":"SUPPORTED","score":9,"correction":null}],"min_score":9,"fields_below_8":[]} min_score is the lowest score in the audit array. fields_below_8 lists the names of every field scoring under 8, in record order, and is an empty array when there are none.

Tested

–

Works

Fails

Model

Unspecified

View Details