Kenneth W. BinghamAI Prototyping Engineer · Full-Stack
A benchmark · measured, not estimated

The fewest tokens that still get the job done.

Every request to an AI is paid for by the token. Shortening a prompt is easy. Knowing whether the shorter one still works is the part people skip.

This is a test bench for that question. It writes the same instruction six ways, gives each to three AI models many times, marks every answer by fixed rules, and reports one number: tokens per correct answer. A wrong answer costs its tokens and counts for nothing, so cutting too far is punished as hard as waste.

3,600answers, every one marked by rule, not by an AI
3models, run on a laptop, reporting exact token counts
40tasks of four kinds, each with a known right answer
41%lower cost per correct answer, best way of writing against polite prose

The result

All models and all tasks together. The instruction column is the length of the instruction alone. The last column is the measure: everything spent, divided by the answers that were right. Lower is better.

Way of writing the instructionInstruction, tokensAnswers rightTokens per correct answer
Mixed: plain direction plus the answer's shape3359%179
Outline with the answer's shape2955%182
Short plain lines3147%191
Abbreviated3442%232
Polite prose9249%304
Function signature4236%350

The top three are close. The clear results are at the bottom: polite prose costs about three times the tokens to say the same thing, and the two forms that look most efficient to a programmer, abbreviation and code, did worse than plain short English.

Same model. Same task. Same data.

One model, asked to pull a date and an amount out of an invoice line and return them as JSON. The only difference is how it was told not to add anything else.

JSON only.0%usable. It found the right values every time, then wrote "Here is the response" in front of them, which breaks any program reading the answer.
Please give me your answer as a JSON object ... and please do not add any other words or explanation.100%usable. Longer, and on this model every extra word was doing work.

The model did not change between those two columns. What it was given did. That is the whole argument of What I put in, as a measurement.

No single way of writing wins

Split by kind of task, the best form changes. That is the practical finding: choose the form to suit the job, and measure, because the right choice is not the one that looks shortest.

Kind of taskBest way of writingTokens per correct answerWorst that still worked
Classify a reviewShort plain lines60 (99% right)Function signature: 329
Pull fields from an invoice lineMixed: plain direction plus the answer's shape145 (75% right)Abbreviated: 379
Do a sumNo way of writing worked. See finding 4.1% at best
Turn a note into a recordOutline with the answer's shape153 (71% right)Polite prose: 376

Four things the numbers showed

Each was checked by reading the wrong answers themselves before it was written down here.

1. Written as code, the models wrote code

Given a function signature and asked for the sentiment of a review, two of the three models produced a sentiment program instead of an answer: 0% and 8% right. An instruction that looks like something to be built gets built.

2. Abbreviating lost twice

"Clsfy rvw sntmnt" is shorter on the page than "Classify the review's sentiment". It cost more tokens, because a model splits unfamiliar spellings into more pieces, and it scored 42% against 47%. What looks minimal to a person is not minimal to a model.

3. Sometimes the padding is carrying the load

The comparison above. A short "JSON only." was ignored by one model every time; the long polite version was obeyed every time. Cutting words is only a saving while the answer stays usable.

4. No wording fixed the arithmetic

Asked for a sum and told "number only", the models were right 1% of the time in every form. When they answered with a bare number they were right 1% of the time. When they broke the rule and showed their working, the right figure was in the answer 55% of the time. Forbidding the working removed the tokens that were doing the job. A sum belongs in code, not in a prompt.

How it works

Three small stages. The output of each is the input of the next, and only the first one uses an AI. The thing being tested is never the judge.

StageGoes inDoes the workComes out
AnswerA task, and one way of writing its instructionThe AI modelAn answer, with the exact count of tokens used
MarkThe answer, and the right answer already knownFixed rules, covered by 65 testsRight or wrong
ReportEvery mark and every token countArithmeticTokens per correct answer

The marking is strict on purpose. If the instruction said "JSON only" and the answer has a sentence in front of it, that is a fail, because a program reading the output would fail too. Each task is asked five times in each form, with fixed seeds so the run can be repeated exactly.

What this does not show

It is a real result for these models and these tasks. It is not a law.

Three small models

llama3.1:8b, llama3.2:3b, qwen2.5-coder:7b, all run locally. Larger commercial models follow instructions better and would likely narrow the gaps. The ranking may differ on them.

Forty tasks

Enough to show the method works and to find the effects above. Not enough to claim a general rate for any way of writing.

One wording of each form

Each way of writing was tried with one phrasing per kind of task. A different phrasing of the same form could score differently, which is itself the point.

It measures. It does not yet rewrite

The next version feeds the result back: shorten a prompt, measure it, keep the change only if the cost per correct answer falls, and judge on tasks it was not tuned on.

The model did not change. The input did.

People say AI produces rubbish. Sometimes it does. This page is a measurement of how much of that is decided before the model reads a word of the task.