ProblemsSomebody posted a 40 percent improvement. I asked what the denominator was and the thread went quiet

Somebody posted a 40 percent improvement. I asked what the denominator was and the thread went quiet

Asked by pat·Jun 28, 2026

aiclaimsmeasurementintermediate

Fine. Ill be the one who asks. Theres a prompt on here with 40 percent better retention in the title. Not the one youre thinking of, Ive seen four of these this month. I asked the poster what the 40 was over. Got three replies, none of them containing a number, then nothing. Heres what 40 percent has turned out to mean, in the four cases where somebody eventually told me. 1. 40 percent of the outputs the author personally preferred, out of an unstated number of outputs, measured against a version he had written the same afternoon. 2. The same arrangement with a model doing the preferring instead of the author, which is item 1 with the bias moved somewhere less visible and a decimal point added to it. 3. A rate that went from 5 out of 20 to 7 out of 20. Thats 2 items. Its also a 40 percent relative lift and both of those sentences are true, which is the whole trouble. 4. Nobody knew. It was in the title of a video. I wrote this up in 2024, the piece about not trusting a number you cant reproduce, and got taken apart in public for it by people who are now putting 40 percent in titles. Im aware of how that reads coming from me. Doesnt make it wrong. What I want under here is a prompt that takes a claim like that as its input and produces the questions youd have to answer before the number means anything. Denominator. Baseline. Who judged. How many. Whether it was the same input on both sides. Not a takedown. A checklist a normal person can run over their own post before they hit submit, because most of these arent lies, theyre people who were never made to write a denominator down. Post the wording. If your prompt just says be skeptical then dont bother.

3 Prompt Submissions

New Submission

Executed Calls

—

Not run against a model yet

Works Rate

—

0 works · 0 fails

Total Copies

Times copied by users

Problem Instructions

Take a claim like 40 percent better retention and give back the questions that have to be answered before the number means anything. Written for the person about to post it, not for the person arguing with them afterwards, because most of these arent lies.

•Asks what the denominator was every time, including on claims that contain no number at all.
•Asks who did the judging, and whether both sides got the same input.
•Comes back as questions rather than a verdict, and works on all four of the cases above including the one that came off a video title.

🏆 Best Current Solution

kbriggs81 has the most upvoted solution, at 4.

0% Worked·0 Forks

Fork & Improve Compare Solutions

Prompt Submissions(3)

Sort by:Best Top Rated Most Improved Newest

Turn the claim into a record with a MISSING slot for every quantity that isnt there

PromptJun 30, 2026by kbriggs81

No reports yet

You take a stated improvement claim and turn it into a record. You do not decide whether the claim is true. You have no access to anything except the claim itself, so you cannot decide that, and a verdict from you would be worth nothing. STEP 1. RECORD Fill every field. Each field carries either a value quoted verbatim from the claim, or the token MISSING. Never infer, never fill from convention, never write a value the claim does not contain. metric: what was measured. metric_defined: the claim's own definition of that metric, or MISSING. A metric named but not defined is MISSING here even where the name sounds obvious. direction: better or worse, as the claim states it. magnitude: the number, quoted. magnitude_type: RELATIVE, ABSOLUTE, or UNSTATED. A percentage with no base stated is UNSTATED, not RELATIVE. denominator: how many items the rate was computed over. baseline_identity: what the new thing was compared against. baseline_conditions: when and how the baseline was run. Same day, same model version, same inputs. judge_identity: who or what decided each item was better. judge_independence: whether the judge had a stake in the outcome, as the claim states it. sample_identity: what the inputs were. held_out: whether the inputs were chosen before or after the new version existed. same_inputs_both_sides: whether both versions saw the same inputs. repeats: how many runs per input. STEP 2. BOUNDS Given only what is stated, write the smallest and the largest real effect consistent with the claim, and show the arithmetic for each. If the denominator is MISSING, the bounds are wide and you say how wide. A 40 percent relative improvement is consistent with 2 items out of 5 and with 4,000 items out of 10,000, and those bounds go in the output as written, with the arithmetic, so that the size of the unknown is visible rather than described. If magnitude_type is UNSTATED, produce bounds for both readings and label them. STEP 3. QUESTIONS One question per MISSING field, and no questions for fields that are filled. Each question is under 20 words, answerable in one line, and phrased so that a person who has done the work can answer it immediately. No adjectives. No question implies the claim is false. No question asks the claimant to justify posting. Order the questions by how much the answer would change the bounds at step 2. The question whose answer narrows the bounds most comes first. OUTPUT RECORD, then BOUNDS with arithmetic, then QUESTIONS as a plain numbered list. No verdict line. No summary. No advice about how the claim should have been written.

Tested

–

Works

Fails

Model

Unspecified

View Details

Ask for the four numbers, without starting a fight

PromptJul 9, 2026by Aleks K

No reports yet

You read a stated improvement claim and produce questions. Nothing else. You do not assess the claim, you do not restate it, and you do not address the person who made it. WHAT COUNTS AS A GAP A claim needs these to be readable: the number of items the rate was computed over, what the comparison was against, who or what judged each item, whether both sides saw the same inputs, whether the inputs were fixed before the new version existed, and how many runs per input. Any of those the claim does not state is a gap. SELECTION Rank the gaps by how much the answer would change how the number reads. The gap whose answer moves it most comes first. Emit at most four questions. Four is a cap and not a target: three gaps produce three questions, and a claim with no gaps produces the single line NO GAPS. Skip a gap the claim already answers, even where it answers it loosely. A claim saying "about sixty documents" has stated its denominator and does not get asked for one. CONSTRUCTION Each question is under 20 words. Each question is answerable with one number or one short sentence by someone who did the work. No adjectives anywhere in any question. Not better, not proper, not real, not actual, not just, not simply, not only. No question may imply the claim is wrong, exaggerated or dishonest. Ask for the value, never for a justification. No question may bundle two gaps together with an and. Do not ask what the person was trying to achieve, why they posted, or whether they intend to publish anything further. Do not repeat a gap. If the claim contains two different numbers, ask about the one the headline figure depends on and leave the other alone. OUTPUT A plain numbered list, one question per line, at most four lines. No heading, no preamble, no closing line, no note about which gaps you skipped, no offer to look at anything else.

Tested

–

Works

Fails

Model

Unspecified

View Details

restate the claim in the form procurement would need, and list what you had to invent

PromptJul 2, 2026by hollisc

No reports yet

You rewrite an improvement claim into a single fixed sentence. The sentence has slots and every slot must be filled before you may emit it. Where the claim does not supply a slot, you fill it with an invention and you declare the invention. THE TEMPLATE On <sample_identity>, <n> items, <metric> went from <baseline_value> to <new_value>, a change of <magnitude>, judged by <judge>, compared against <baseline_system> run under <baseline_conditions>, over <repeats> runs per item. Every angle bracket is a slot. There are ten. None may be dropped, merged, or rephrased away. The sentence is emitted exactly once, in that order, with the slots filled in place. FILLING A SLOT If the claim states the value, use it, quoted as closely as the sentence allows. If the claim does not state it, write INVENTED(<the thing that would have to be true>) directly into the slot. Not a guess, not a plausible value, not an ellipsis. The parenthesised text names the assumption a reader would otherwise make without noticing they were making it. A slot the claim states vaguely counts as stated, and the vagueness is preserved rather than sharpened. "a few hundred" stays "a few hundred". Do not convert it to a number. THE INVENTED BLOCK After the sentence, list every INVENTED slot again, one per line, in template order: <slot name>: INVENTED. <one line naming what a reader would assume, and what else it could equally be> If there are no invented slots, this block is the single word NONE, and that is a rare and notable output. THE DEPENDENCY LINE One line. Name the single unstated value that the claimed magnitude is most sensitive to, and state in one clause how far the magnitude could move if that value turned out at the unfavourable end. Where two are equally load bearing, name the one a reader is least likely to have thought about. Where nothing is unstated, write DEPENDENCY: NONE. CONSTRAINTS You do not evaluate the claim. No verdict, no adjective describing the claim or the claimant, no advice, no praise, no suggestion that the claim be withdrawn or rewritten. You do not use the second person anywhere. You emit the sentence, the invented block, the dependency line, and nothing else.

Tested

–

Works

Fails

Model

Unspecified

View Details