Project · Python · FastAPI · Postgres + pgvector
citeline
A retrieval augmented question answering service over the US National Primary Drinking Water Regulations. It answers with a citation to the exact regulation section, or it tells you it does not know. There is no third option, and that is enforced in code rather than requested in a prompt.
The problem
A retrieval demo that answers every question looks impressive and is nearly worthless. The interesting question is not whether a system can answer when the answer is present. It is what the system does when the answer is absent, because that is the case that occurs constantly in real use and it is the case where a language model is most confident and most wrong.
I picked federal regulations as the corpus for exactly that reason. Regulatory text is dense with numeric thresholds, deadlines, defined terms and cross references. A confidently wrong contaminant limit is not an amusing error; it is the kind of error that gets a compliance tool thrown out. So the requirement is not "answer well". It is never assert anything that is not in the retrieved text, and prove that property rather than claim it.
Why it is harder than it looks
Vector search alone quietly fails on regulations
Ask for the limit on arsenic and a dense index returns several passages about arsenic monitoring, sampling schedules and treatment technology. They are all genuinely about arsenic and semantically very close to the question. None of them contains the number. Embeddings blur exactly the tokens that matter here: numeric thresholds, units, and section identifiers.
So retrieval runs two ways at once. A dense vector search over pgvector handles paraphrase, and a Postgres full text search pins the exact terms. Their results are merged with reciprocal rank fusion, which combines by rank rather than score and therefore needs no normalisation between two retrievers whose numbers have nothing to do with each other.
Chunking on a fixed size destroys the answer
The contaminant limits live in tables. Slice those on a character count and you get a chunk holding a contaminant name whose number landed in the next chunk, and a chunk holding a number with nothing to say what it measures. Both look like fine retrieval results and both are useless. The chunker splits on paragraph boundaries first, falls back to sentence boundaries only when a single paragraph exceeds budget, and prefixes every chunk with its own section number and heading so a retrieved fragment is always attributable on its own.
A rank score is not a confidence score
This one cost me a rewrite, and it is the most useful thing on this page. My first version gated abstention on the fused retrieval score. That is wrong twice over.
First, arithmetically: with a fusion constant of 60 and two retrievers, the maximum
achievable fused score is 2/61 = 0.0328. I had set the threshold to
0.045, which no result can ever reach, so the service would have
abstained on every question ever asked.
Second, and worse, the approach could not have worked even with a reachable number.
Reciprocal rank fusion is a rank statistic. A dense retriever returns its k
nearest rows for any input whatsoever, so whatever is closest still lands at rank 1
and still scores 1/61. The top fused score for
what is the capital of France looks almost identical to the top fused score
for a question the corpus genuinely answers. It carries essentially no information
about coverage.
The gate now tests absolute cosine similarity, which does move between those two cases, plus an independent requirement that the sparse retriever matched something at all. Fusion still decides the ordering. It just no longer pretends to measure confidence.
Architecture
Two paths through one system: a batch pipeline that builds the index, and a request path that serves it.
batch · scheduled
Ingestion pipeline
eCFR API
↓ parse XML into one document per section
↓ gate · reject short, reserved, insecure
↓ SHA compare · unchanged? skip
↓ structure aware chunking
↓ gate · reject wordless, over budget
↓ embed (nomic-embed-text, 768-d)
↓ gate · reject wrong dim, zero vector
↓ upsert in one transaction
↓ record run status + counts
request · live
Query path
question
↓ embed as search_query
├─ vector search · HNSW cosine, k=24
└─ full text search · GIN, ts_rank_cd, k=24
↓ reciprocal rank fusion, dedupe by section
↓ gate 1 · similarity + lexical match
fail → abstain, model never called
↓ grounded generation, temperature 0
↓ gate 2 · model may decline
↓ gate 3 · verify every citation
fail → discard answer, abstain
The three gates
gate 1 · before the model
If no passage is semantically close enough, or no passage contains the question's terms, the language model is never invoked. This is the cheap gate and it catches the common case. Its threshold was chosen by sweeping the labelled set, not by judgement.
gate 2 · the prompt
The model is instructed to reply with a fixed marker when the supplied excerpts do not answer the question. This is the weakest of the three, because it is the only one that depends on instruction following, which is why it is not the only one.
gate 3 · after the model
Every citation the model emitted is checked against the excerpts it was actually handed. A citation to an excerpt that was never supplied, or a substantive sentence carrying none, fails the whole answer. It is discarded, not served with a warning attached.
Query it
This is the running service, not a recording. Try a question the regulations answer, then try one they do not, and watch it decline.
- In the corpus:
- Not in the corpus:
Retrieve runs the retrieval path and the gate, and returns in well under a second. Full answer additionally runs generation on the server's CPU, which takes tens of seconds. Both show the gate's decision and the numbers behind it. Rate limited to 20 requests a minute.
Evaluation
A labelled set of 28 questions: 20 the regulations answer, with the section that answers each one, and 8 they do not. Scoring asserts specific strings rather than asking a model to grade, so the result is reproducible and cannot drift with a grader's mood.
Evaluation figures are generated by eval/run_eval.py and inserted here after each run.
What is measured, and why both halves matter
Retrieval quality
- recall@k — did the correct section appear in the top k at all
- hit@1 — was it ranked first
- MRR — mean reciprocal rank of the correct section
- latency — p50 and p95, measured end to end
Answer integrity
- accuracy — answered, contained the required fact, cited the right section
- abstention recall — share of unanswerable questions correctly refused
- false answer rate — unanswerable questions that got an answer anyway. This is the hallucination rate, and it is the number the whole design exists to hold down
Run it yourself
The repository is self contained. Docker brings up Postgres with pgvector; the models run locally through Ollama, so there is no API key and nothing leaves the machine.
git clone https://github.com/kenbin64/citeline
cd citeline
docker compose up -d # postgres 16 + pgvector
ollama pull nomic-embed-text # embeddings, 768-d
ollama pull llama3.2:3b # generation
make install # venv + dependencies
make migrate # apply the schema
make ingest # fetch, gate, chunk, embed, index
make check # ruff + mypy + 31 unit tests
make eval # regenerate the numbers above
citeline search "MCL for arsenic" # retrieval only
citeline ask "MCL for arsenic" # full pipeline
make ingest is idempotent. Run it twice and the second run writes
nothing, reporting every chunk as skipped, because each document is compared by
content hash before any work is done.
Limits
Things this does not do, stated plainly so nobody has to discover them.
- No reranker. A cross encoder would likely improve ranking, but it cannot run on this CPU for the reason above. The interface for it exists; the model does not.
- The eval set is small. 28 questions written by one person. It is enough to catch a broken gate and to set a threshold, and not enough to claim a general accuracy figure. Treat the numbers as a regression baseline, not a benchmark.
- One corpus, one part. Part 141 only. Nothing about the design is specific to it, but no other body of text has been tested, so no claim is made about them.
- Gate 3 verifies grounding, not correctness. It proves every claim carries a citation to a passage that was actually supplied. It does not prove the model read that passage correctly. That is a real gap, and it is why the answer always shows its sources for a human to check.
- Abstention has a cost. A tighter gate refuses some questions the corpus genuinely answers. That trade is deliberate and the sweep table shows exactly what it costs at each threshold.