case study · node · deployed
valuesai answers your question several times over, embeds the answers, and measures whether they agree with each other. A panel that converges is something the model knows. A panel that scatters is something it is inventing, and the score says so before you act on it.
A language model gives you the same confident tone whether it is recalling something it has seen ten thousand times or filling a gap with something plausible. The prose does not change. That is the whole difficulty: the failure mode is invisible at exactly the moment it matters.
Asking the model how confident it is does not help, because it will answer that question with the same machinery it used for the first one. So the measurement has to come from somewhere the model does not control.
Ask it more than once. If it knows, the answers land in the same place. If it is improvising, each attempt improvises differently. Agreement across independent attempts is evidence you can measure without needing to know the right answer yourself.
Embed each answer, normalise to unit vectors, sum them, and divide the length of that resultant by the number of answers. Answers pointing the same way reinforce and give a resultant near 1. Answers pointing in different directions partly cancel. It asks a blunt question: are these even answers to the same thing?
Whether the answers agree on the thing the question turns on, which is often not the prose. The engine routes on the question itself:
Measured on the live service, not illustrative.
| Asked | Score | Topic | Fact | What happened |
|---|---|---|---|---|
| Boiling point of water | 91 · reliable | 99 | 92 | Every answer said 100 degrees Celsius. Nothing to disagree about. |
| Moons of Mars | 76 · reliable | n/a | n/a | Phobos and Deimos every time, with the wording varying around the agreed facts. |
| Population of a town that does not exist | 32 · fabrication risk | 96 | 33 | The interesting one. Topic collapse stayed high because every answer was clearly about the same question, while fact collapse fell through the floor: each attempt invented a different correction of the invented name. The engine abstained. |
That third row is the entire product. The model did not refuse and did not sound unsure. It produced three fluent, differently wrong answers, and the disagreement between them is what gave it away.
| Decision | Chosen | Instead of | Why, and what it costs |
|---|---|---|---|
| Confidence signal | Agreement across a panel | Asking the model how sure it is | Self-reported confidence is produced by the same machinery that produced the answer, so it fails in the same places. Agreement is measured from outside. The cost is that it takes several generations instead of one. |
| Comparing answers | Embeddings | A model judging the answers | An LLM judge is another model with the same failure mode, and it is slow. Embedding is fast, deterministic, and cheap. The cost is that it compares meaning rather than reasoning about it. |
| Request shape | A job you watch | One request that returns an answer | Measured on this host: 3.6 tokens per second, so a panel takes minutes. Hiding that behind a spinner reads as broken. Showing each answer as it lands turns the wait into the demonstration. The cost is a queue, polling, and more moving parts. |
| Panel generation | Sequential | All at once |
The engine's own entry point fires the whole panel with Promise.all. With a
single Ollama already using every core, that makes each answer slower and none of them
arrive sooner. Sequential is no slower overall and lets each answer be shown on arrival.
|