RAG in Production: 18,755 Chunks, Local Embeddings, and Cited Answers
Retrieval-augmented generation (RAG) answers a question from your own documents and cites the passages it used. A demo needs a few PDFs and an afternoon. Production needs a pipeline that parses messy files, keeps personal data out of the index, survives a stuck model server, and never shows a user a document they are not allowed to read.
We built that pipeline, sushi-rag, and ran it end to end on our own hardware. This post covers the design, the measured ingest, a 64-question evaluation against a hosted model, and the failures that changed our code. Every number below comes from the run reports, and we say which run each one is from.
sushi-rag output from the same question asked with two permission levels. Watch on YouTube.Read the transcript
00:00–00:07 · The question. Can an AI answer from your own documents, and prove where every answer came from?
00:07–00:15 · The corpus. We built sushi-rag on public tax publications, a container docs snapshot, and synthetic company docs.
00:15–00:29 · The pipeline. Each document is parsed, scrubbed of personal data, chunked, and embedded on a local Jetson Thor. 18,755 points land in a hybrid index.
00:29–00:38 · Ask a question. Now we ask a question. The answer comes back with a citation to the exact source document.
00:38–00:48 · Check permissions. Same question, different permissions. One user gets the answer. The other gets nothing internal. Zero leaks across 64 test questions.
00:48–01:00 · Compare with a hosted model. Against Claude Haiku, the local model matched or beat it on grounded answers, at about seven times the latency. The whole comparison cost 53 cents.
01:00–01:10 · Wrap up. Grounded answers, real citations, access control built in. Read the full write-up on our blog.
Test it on a corpus that looks like real work
A RAG system is only as honest as the corpus you test it on. We used three sources:
- Public IRS publications, which are long PDFs with tables, footnotes, and dense numbers.
- A pinned snapshot of the public Kubernetes documentation, which is many short pages full of command names.
- Synthetic Sushi Corp documents, which we wrote to contain fake names, emails, phone numbers, and SSNs, plus HR pages that only some users may read.
A pinned script fetches the public documents. They are not committed to the repo. The synthetic documents are the only place where personal data appears, and all of it is invented. No real or internal data went into this project.
The IRS has not endorsed this project, and nothing here is tax advice. The Kubernetes content is © The Kubernetes Authors, CC BY 4.0, modified by chunking. Neither publisher is affiliated with this demo.
How a document becomes a cited answer
sushi-rag pipeline. The ACL filter runs inside the search, before ranking, so a chunk the caller cannot read never competes for a slot.- Parse. Docling converts PDF and HTML into structured text. It parsed 1,811 of 1,812 documents. One Kubernetes page,
reference/access-authn-authz/authentication, fails in Docling. The run records the failure and moves on. - Redact. Presidio with spaCy masks emails, phone numbers, SSNs, and person names before any text reaches the index.
- Chunk. Token-aware chunking produces 18,755 chunks: 5,736 from IRS publications, 12,892 from Kubernetes, and 127 from Sushi Corp. Exact duplicate chunks are dropped, so the chunk count matches the stored points.
- Embed. bge-m3 runs through Ollama on a local NVIDIA Jetson Thor. No document text leaves our network at ingest.
- Search. Qdrant stores each chunk with a dense vector and a BM25 sparse vector. Reciprocal rank fusion (RRF) merges the two result lists, and the ACL filter limits both to the caller's groups.
- Answer. A local model,
qwen3.8:27bon the same Thor, answers from the top 5 chunks and cites them. If the chunks do not hold the answer, it refuses.
The corpus, measured
These numbers come from the latest ingest report, after the copyright-check fix described below.
| Metric | Result |
|---|---|
| Documents parsed | 1,811 of 1,812 |
| Chunks stored | 18,755 |
| Chunks by source | IRS 5,736, Kubernetes 12,892, Sushi Corp 127 |
| Chunk size in tokens | mean 196.9, p50 100, p95 608, max 804 |
| Exact duplicate chunks dropped | 50 |
| Redactions | PHONE 1,343, EMAIL 68, PERSON 50, SSN 14 |
The latest ingest added only the chunks that were missing, so its timings do not describe a full run. The last full run from an empty collection (run 10, 16,926 chunks, before the copyright fix) took 67 minutes to embed and 28 minutes to parse, redact, and chunk. Embedding is the slow step: on one local device it takes more than twice as long as everything else together.
Four failures that changed the code
The redactor treated technical words as people
An earlier run (run 8) tagged 15,636 spans as PERSON. Most were not names. Words like kubeadm and "Per Diem" came back as people, and the masks made public documents harder to search.
The redactor no longer runs name detection on public documents or on Markdown headings, and it drops implausible names before it masks a span. Synthetic Sushi Corp documents still get full redaction, and tests check that their fake names stay masked. In the latest ingest, PERSON is 50.
A PII redactor tuned on prose over-fires on technical text. Measure its output per source before you trust it.
The copyright check matched a topic, not a notice
We drop chunks that carry a third-party copyright notice. The first version matched the word "copyright", so it also skipped IRS content that discusses copyright as a subject. The check now looks for notices only. IRS chunks went from 3,959 in run 10 to 5,736, and no chunk is skipped for copyright.
The embedding server stopped answering
The bge-m3 runner in Ollama on the Thor wedged several times. Requests hung, GPU use sat at 97%, and even a one-word embedding request got no answer. Two full runs stopped because of it.
We changed the ingest so that one stuck request does not end a run:
- On the first timeout, the ingest unloads the model and retries.
- If one text keeps failing, the ingest skips it and lists it in the run report.
EMBED_BATCH_SIZEandEMBED_TIMEOUT_SECONDSset the batch size and the timeout.
In run 10 the runner still stalled at times. The ingest recovered each time and skipped 0 chunks.
A crash at the end lost all the work
The first version wrote every vector to Qdrant at the end, so a crash after an hour of embeddings lost the hour. The ingest now upserts in batches as it goes and resumes from the points Qdrant already has.
We checked that with a second full pass over the same corpus. It added 0 new chunks, resumed all 16,926, and computed no embeddings. It took 25 min 21 s, mostly re-parsing. All 5 smoke checks passed, including a vacation-policy (PTO) query that returns nothing for a user without HR access.
Does it find the right documents?
We wrote a gold set of 64 questions over the public and synthetic documents. 53 have an answer in a known document. 11 should be refused, because the caller may not read the document or because no document has the answer. Retrieval is scored at document level on the 53 answerable questions.
| Mode | recall@1 | recall@5 | recall@10 | MRR@10 |
|---|---|---|---|---|
| Dense (bge-m3) | 67.9% | 92.5% | 98.1% | 0.787 |
| Sparse (BM25) | 66.0% | 90.6% | 96.2% | 0.779 |
| Hybrid (RRF) | 67.9% | 100.0% | 100.0% | 0.821 |
Hybrid search wins where it counts for RAG: every expected document is in the top 5. Each mode alone has a weak source. Dense search found 85.7% of the Sushi Corp documents in the top 5, and sparse search found 88.9% of the Kubernetes ones. Hybrid reached 100% on all three sources.
The first eval run scored recall@5 of 96.2% for dense, 86.8% for sparse, and 98.1% for hybrid. Between runs we added each document's title to its sparse vector, stopped name detection on headings, and re-ingested after the copyright fix. Not every number improved. Hybrid recall@1 went from 73.6% to 67.9%, and MRR@10 from 0.840 to 0.821, so the right document reaches the top 5 more often but is first less often.
For how to build a gold set and read these metrics, see how to evaluate RAG retrieval quality.
Is the access filter tight?
In all 64 queries, across dense, sparse, and hybrid retrieval and in the final answers, no result contained a chunk outside the caller's groups: 0 leaks out of 64. The 64 questions include questions that target HR documents from a user without HR access.
That is a test result, not a proof. It shows the filter holds on these queries. A permission model with more groups needs its own test set.
Local model against a hosted one
To compare answer quality, we gave the same ACL-filtered top-5 contexts and the same prompt to our local model, qwen3.8:27b on the Thor, and to Anthropic's claude-haiku-5-5. Only the exported contexts left our network for this test. Claude Sonnet (claude-sonnet-5-5) judged whether each answer is supported by its cited sources.
| Metric | Local qwen3.8:27b | claude-haiku-5-5 |
|---|---|---|
| Answered, of answerable questions | 98.1% | 94.3% |
| Answers that cite an expected document | 92.3% | 92.0% |
| Citation precision (cited documents that are expected) | 72.6% | 72.5% |
| Correct refusals, of 11 should-refuse questions | 90.9% | 90.9% |
| Supported by cited sources, Sonnet judge | 52 of 53 (98.1%) | 46 of 51 (90.2%) |
| Generation latency p50 / p95 | 12.01 s / 16.48 s | 1.69 s / 2.67 s |
On quality, the local model matches the hosted one on these questions. Citations and refusals are within a fraction of a point, and the local model answered and was judged supported slightly more often. On speed, the hosted model is about 7 times faster at p50. On the Thor, retrieval is not the bottleneck: hybrid search takes 9 ms at p50, query embedding 0.52 s, and a full answer 12.55 s, almost all of it generation.
The whole hosted baseline, answers and judging, cost $0.53 over 187 API calls, against a cap of $4.
Read the judge numbers with care
- The judge is a model. A local judge grading its own model's answers rated 100% as supported. We do not report that as a quality number. The Sonnet judge is stricter, but it is still a model, not a person.
- The judge shares a vendor with one contestant. Sonnet graded Haiku, a model from the same vendor. We have not measured whether that helps or hurts Haiku.
- Some replies needed a second pass. 9 judge replies were not valid JSON at first and were re-judged with a larger output limit. 0 remain unparsed.
- The gold set is small. With 53 answerable questions, one question is about 1.9 points. A gap of a few points is within noise.
- The latency is not a fair hardware race. The local numbers come from one edge device that also runs embeddings. The hosted numbers come from a production API.
Try it yourself
make try builds a small demo collection, sushi_rag_demo, from public and synthetic documents only. In our run it ingested 73 documents into 730 chunks.
What we would add before real users
- A larger gold set, with human review of a sample of judged answers.
- Better ranking at the top, since hybrid recall@1 is 67.9% even though recall@5 is 100%.
- Higher citation precision. About a quarter of cited documents are not the expected ones.
- Faster generation on the local device, or a smaller answer model where 12 seconds is too slow.
- No re-parsing of unchanged documents, which is most of the 25-minute re-run.
- A fix for the one Kubernetes page that Docling cannot parse.
Keep the data home and measure the rest
A private RAG system can run on one local device and still find the right document, respect permissions, and cite its sources about as well as a hosted model on the same contexts. What it costs is answer speed. Measure retrieval, leaks, and answers separately, because each one fails in its own way.
To decide whether RAG fits your product, read RAG vs. fine-tuning: how to actually choose. For help building a private RAG system, explore our AI/ML consulting services or book a discovery call.