Kenneth W. Bingham

case study · AI reliability · measured on public test sets

Asking the same AI twice does not make it right. Asking a different one helps.

A common way to make an AI answer more reliable is to ask the same model several times and take the answer it gives most. I built a reliability engine and tested that idea against another one on two public question sets I did not write. Asking the same model five times cost five times as much and barely moved its wrong answers. Checking it against a model from a different family cut them from 10.0% to 2.7%.

The problem

A wrong answer from an AI looks exactly like a right one. In regulated or customer-facing work, that is the whole risk: nobody can check every answer, so the system has to know when it should not answer. The usual fixes, asking the model again or having a second call check the first, cost more tokens. Whether they actually catch the wrong answers is rarely measured.

What was tested

Two public sets, chosen because they are built to catch a model out:

Three model families ran locally on one laptop: Llama (llama3.1:8b), Qwen (qwen2.5-coder:7b) and DeepSeek (deepseek-coder-v2 16B lite). Every answer was marked by fixed rules, never by a model: 3,256 model calls in all.

Results: documents

DesignRightWrongNo answerTokens per correct answer
One call to Llama82.7%10.0%7.3%350
Llama asked five times, majority answer83.3%9.7%7.0%1,731
Two families, Llama and Qwen, must agree79.0%2.7%18.3%845
Two families plus a word-matching check76.7%7.0%16.3%871

Results: misconceptions

DesignRightWrongNo answerTokens per correct answer
One call to Llama49.5%50.5%0%241
Llama asked five times, majority answer50.0%50.0%0%1,192
Three families must all agree22.0%13.5%64.5%1,732
Two of three families agree38.5%42.5%19.0%990

"No answer" means the system declined and handed the question to a person. It is never counted as right on a question that has an answer.

What it shows

How the engine works

The design principles, in the order the engine applies them:

A question goes to two model families; each answer must quote the document; if they agree the answer is given, otherwise a person decides. Questionand its document Model family 1answer + quote Model family 2answer + quote Quote in source?do they agree? Answerwith its source A person decidessplit or unsupported
The document path. Questions without a document go to three families instead, and agreement there is marked "consistent", never "verified".

How the test was kept fair

Did it pass?

Not yet. Three marks had to pass for the engine to count as finished; one did.

MarkResultWhy
Halve wrong answers against one call, documentsNoThe full engine went from 10.0% to 7.0%. The word-matching check was the cause; the two-family design alone reached 2.7%, but it was not the design the mark was set for.
Fewer wrong answers than five samples, at lower costYes7.0% against 9.7%, at half the tokens per correct answer.
Fewer wrong answers on misconceptions, while still answering enoughNoWrong answers fell from 50.0% to 13.5%, but it answered too few questions: 22.0% right against a floor of 34.6%.

The next step follows from the result: drop the word-matching check as a tie-breaker, make the two-family design the engine for documents, and test it on a fresh draw of questions with the marks set first again.

Why it matters to an employer

Technologies and skills

What it does not claim

10.0% to 2.7%wrong answers, one model against two families
10.0% to 9.7%wrong answers, one model against itself five times
3,256model calls, marked by rule
1 of 3pass marks met, so not yet finished