{"id":77,"date":"2026-07-22T17:35:20","date_gmt":"2026-07-22T17:35:20","guid":{"rendered":"https:\/\/emporiant.com\/developer\/?p=77"},"modified":"2026-07-22T17:35:21","modified_gmt":"2026-07-22T17:35:21","slug":"grounding-local-models-without-retraining-the-rag-pipeline-behind-iridium-one","status":"publish","type":"post","link":"https:\/\/emporiant.com\/developer\/technical\/grounding-local-models-without-retraining-the-rag-pipeline-behind-iridium-one\/","title":{"rendered":"Grounding Local Models Without Retraining: The RAG Pipeline Behind IRIDIUM ONE"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Earlier in this series we covered IRIDIUM ONE&#8217;s hardware baseline and its software stack, and promised a closer look at retrieval-augmented generation (RAG). This is that post: how the appliance grounds a local model in a client&#8217;s own documents without ever fine-tuning or retraining anything.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Why Retraining Was Never on the Table<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Fine-tuning a model per client sounds appealing in theory, but it fails the moment you think about it operationally. Every client update to their own documents would require a retraining cycle. Every model upgrade would mean redoing that work per client from scratch. And for an on-prem appliance running on modest hardware, the compute cost of retraining simply isn&#8217;t available on the same box that&#8217;s meant to run day-to-day inference.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">RAG sidesteps all of this. The base model never changes. What changes is what gets fed into its context window at request time, pulled fresh from the client&#8217;s own document store on every query.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">The Pipeline, Step by Step<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">At a high level, the RAG layer sitting on top of Ollama does three things for every incoming query:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Retrieve.<\/strong> The client&#8217;s documents are chunked and embedded ahead of time, then stored locally. An incoming query gets embedded the same way, and the closest-matching chunks are pulled from storage.<\/li>\n\n\n\n<li><strong>Augment.<\/strong> Those retrieved chunks are assembled into the prompt alongside the user&#8217;s actual question, giving the model concrete source material to work from instead of relying purely on what it learned during pretraining.<\/li>\n\n\n\n<li><strong>Generate.<\/strong> Ollama produces the response, now grounded in the client&#8217;s own material rather than in whatever the base model happened to have seen during training.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Because this all happens locally, using SQLite for the metadata and a local vector store for the embeddings, no document content or query ever needs to leave the appliance. That&#8217;s the same DSGVO-driven design constraint that shapes every other part of the ONE stack.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Why This Matters More for On-Prem Than for Cloud<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Cloud AI products can lean on much larger, more expensive retrieval infrastructure because they&#8217;re amortizing that cost across many customers. An on-prem appliance doesn&#8217;t have that luxury; the retrieval layer has to be efficient enough to run comfortably on the same Ryzen 7 8845HS handling inference, without a client noticing a meaningful slowdown compared to a cloud alternative.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That constraint is why the RAG implementation favors simpler, well-understood retrieval methods over more exotic ones. A retrieval approach that&#8217;s marginally more accurate but meaningfully heavier isn&#8217;t a good trade when it&#8217;s competing for the same limited on-device compute as the LLM itself.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Earlier in this series we covered IRIDIUM ONE&#8217;s hardware baseline and its software stack, and promised a closer look at retrieval-augmented generation (RAG). This is that post: how the appliance grounds a local model in a client&#8217;s own documents without ever fine-tuning or retraining anything. Why Retraining Was Never on the Table Fine-tuning a model [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"service_tag":[],"class_list":["post-77","post","type-post","status-publish","format-standard","hentry","category-technical"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/posts\/77","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/comments?post=77"}],"version-history":[{"count":1,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/posts\/77\/revisions"}],"predecessor-version":[{"id":78,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/posts\/77\/revisions\/78"}],"wp:attachment":[{"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/media?parent=77"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/categories?post=77"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/tags?post=77"},{"taxonomy":"service_tag","embeddable":true,"href":"https:\/\/emporiant.com\/developer\/wp-json\/wp\/v2\/service_tag?post=77"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}