case study · AI reliability · measured on public test sets
Asking the same AI twice does not make it right. Asking a different one helps.
A common way to make an AI answer more reliable is to ask the same model several times and take the answer it
gives most. I built a reliability engine and tested that idea against another one on two public question sets
I did not write. Asking the same model five times cost five times as much and barely moved its wrong answers.
Checking it against a model from a different family cut them from 10.0% to 2.7%.
A wrong answer from an AI looks exactly like a right one. In regulated or customer-facing work, that is the
whole risk: nobody can check every answer, so the system has to know when it should not answer. The usual
fixes, asking the model again or having a second call check the first, cost more tokens. Whether they actually
catch the wrong answers is rarely measured.
What was tested
Two public sets, chosen because they are built to catch a model out:
Documents: 300 questions from SQuAD 2.0,
each about one paragraph. Half cannot be answered from the paragraph, and were written on purpose to look
as if they can. The right response to those is "the paragraph does not say".
Misconceptions: 200 multiple-choice questions from TruthfulQA,
where the wrong choices are things many people, and many models, believe.
Three model families ran locally on one laptop: Llama (llama3.1:8b), Qwen (qwen2.5-coder:7b) and DeepSeek
(deepseek-coder-v2 16B lite). Every answer was marked by fixed rules, never by a model: 3,256 model calls in all.
Results: documents
Design
Right
Wrong
No answer
Tokens per correct answer
One call to Llama
82.7%
10.0%
7.3%
350
Llama asked five times, majority answer
83.3%
9.7%
7.0%
1,731
Two families, Llama and Qwen, must agree
79.0%
2.7%
18.3%
845
Two families plus a word-matching check
76.7%
7.0%
16.3%
871
Results: misconceptions
Design
Right
Wrong
No answer
Tokens per correct answer
One call to Llama
49.5%
50.5%
0%
241
Llama asked five times, majority answer
50.0%
50.0%
0%
1,192
Three families must all agree
22.0%
13.5%
64.5%
1,732
Two of three families agree
38.5%
42.5%
19.0%
990
"No answer" means the system declined and handed the question to a person. It is never counted as right
on a question that has an answer.
What it shows
The same model, asked again, mostly repeats itself. Five samples changed wrong answers by less than
half a point on both sets, at five times the tokens. A model's mistakes are not random noise that averaging
removes; they are what the model believes.
A different family is a genuinely independent check. Models trained by different teams make
different mistakes, so where they agree the answer is far more often right. On documents that cut wrong
answers by about 3.7 times, at half the cost of asking one model five times.
A check is only useful if it can be wrong for a different reason. I expected a word-matching check
with no AI in it to settle disagreements between the models. It made things worse: the trick questions are
built from the paragraph's own words, so it found the very sentence the trap came from and backed the wrong
answer, all 13 times it settled a trick question. It shared the trap's blind spot.
Different families still share popular myths. On 13.5% of the misconception questions all three
families agreed on the same wrong answer. Agreement is evidence, not proof. Where it matters, only a source
can settle it.
Fewer wrong answers cost more referrals to a person. The two-family design declined about a third
of the answerable document questions. Whether that trade is worth it depends on what a wrong answer costs.
How the engine works
The design principles, in the order the engine applies them:
Software first. Where an answer can be computed or read exactly (a sum, a date in a fixed format),
code does it and no model is called. In an earlier benchmark this alone took one task from 0% to 100% right.
Every claim must be found in the source. A model answering from a document must copy the sentence it
relied on. If that sentence is not in the document, or the answer is not in that sentence, the answer is refused.
Independent checks, not repeated ones. A second opinion comes from a different family of model, never
the same model asked again.
Disagreement goes to a person. When the checks split and nothing independent settles it, the engine
says so instead of picking one.
Only checked answers are remembered. An answer confirmed against its source is kept as a fact; an answer
that is merely agreed on is kept, but marked unverified, so it is never built on as if it were proven.
The document path. Questions without a document go to three families instead, and agreement there is marked "consistent", never "verified".
How the test was kept fair
Marks set first. The pass marks were written down before any engine or test code existed, so the result
could not be argued into a win afterwards.
Questions I did not write. Both sets are public, written by other people, and drawn with a fixed seed.
One set of calls for every design. Designs that share a step share the same saved model calls, so the
comparison is between designs, not between lucky and unlucky samples.
No model as judge. Answers were marked by fixed rules (SQuAD's own word-overlap scoring), with 31 unit
checks on the engine's decision rules.
Did it pass?
Not yet. Three marks had to pass for the engine to count as finished; one did.
Mark
Result
Why
Halve wrong answers against one call, documents
No
The full engine went from 10.0% to 7.0%. The word-matching check was the cause; the two-family design alone reached 2.7%, but it was not the design the mark was set for.
Fewer wrong answers than five samples, at lower cost
Yes
7.0% against 9.7%, at half the tokens per correct answer.
Fewer wrong answers on misconceptions, while still answering enough
No
Wrong answers fell from 50.0% to 13.5%, but it answered too few questions: 22.0% right against a floor of 34.6%.
The next step follows from the result: drop the word-matching check as a tie-breaker, make the two-family
design the engine for documents, and test it on a fresh draw of questions with the marks set first again.
Why it matters to an employer
It shows how I evaluate an AI system: on questions built to break it, against baselines, with the cost of every
answer counted, and with a plan written before the results exist.
It reports what failed. The most useful finding here, that a check is only worth having if it fails for a
different reason, came from a part of my own design that did not work.
The principles carry over to any AI feature that must not make things up: support every claim from a source,
get second opinions that are really independent, and send the hard cases to a person.
Technologies and skills
Node.js, local models through Ollama with structured (JSON schema) output, LLM evaluation design,
retrieval-style grounding, benchmark statistics.
What it does not claim
Three small open models on one machine. Large commercial models are more accurate, and the gaps may differ on them.
SQuAD paragraphs are short and clean; real documents are longer and messier.
Multiple choice is easier to mark than free answers. A result on it says nothing about free-text answers.
No design here makes an AI never wrong. The goal is fewer wrong answers, and knowing which answers to check.
The engine's code is private. The results, the design and the test sets are here so the claims can be checked.
10.0% to 2.7%wrong answers, one model against two families
10.0% to 9.7%wrong answers, one model against itself five times