Earlier in this series we covered IRIDIUM ONE’s hardware baseline and its software stack, and promised a closer look at retrieval-augmented generation (RAG). This is that post: how the appliance grounds a local model in a client’s own documents without ever fine-tuning or retraining anything.

Why Retraining Was Never on the Table

Fine-tuning a model per client sounds appealing in theory, but it fails the moment you think about it operationally. Every client update to their own documents would require a retraining cycle. Every model upgrade would mean redoing that work per client from scratch. And for an on-prem appliance running on modest hardware, the compute cost of retraining simply isn’t available on the same box that’s meant to run day-to-day inference.

RAG sidesteps all of this. The base model never changes. What changes is what gets fed into its context window at request time, pulled fresh from the client’s own document store on every query.

The Pipeline, Step by Step

At a high level, the RAG layer sitting on top of Ollama does three things for every incoming query:

• Retrieve. The client’s documents are chunked and embedded ahead of time, then stored locally. An incoming query gets embedded the same way, and the closest-matching chunks are pulled from storage.
• Augment. Those retrieved chunks are assembled into the prompt alongside the user’s actual question, giving the model concrete source material to work from instead of relying purely on what it learned during pretraining.
• Generate. Ollama produces the response, now grounded in the client’s own material rather than in whatever the base model happened to have seen during training.

Because this all happens locally, using SQLite for the metadata and a local vector store for the embeddings, no document content or query ever needs to leave the appliance. That’s the same DSGVO-driven design constraint that shapes every other part of the ONE stack.

Why This Matters More for On-Prem Than for Cloud

Cloud AI products can lean on much larger, more expensive retrieval infrastructure because they’re amortizing that cost across many customers. An on-prem appliance doesn’t have that luxury; the retrieval layer has to be efficient enough to run comfortably on the same Ryzen 7 8845HS handling inference, without a client noticing a meaningful slowdown compared to a cloud alternative.

That constraint is why the RAG implementation favors simpler, well-understood retrieval methods over more exotic ones. A retrieval approach that’s marginally more accurate but meaningfully heavier isn’t a good trade when it’s competing for the same limited on-device compute as the LLM itself.