# Pointing an LLM at your own documents (RAG, honestly) > Everyone wants 'a ChatGPT that knows our internal docs'. The mechanism is called RAG, it's simpler than the hype, and most of the work is the boring retrieval half nobody demos. An ops-eye view. Source: https://rfriedmann.de/blog/your-own-documents/ Published: 2026-06-21 · Track: learn · Level: Ops The most common AI request to land on an infrastructure team is some version of: "can we have a ChatGPT that knows our stuff?" Our runbooks, our wiki, our tickets, our contracts. It's a reasonable ask, and the good news is it's a solved pattern. The honest news is that the demo is the easy 20%, and the part that makes it actually useful, or quietly useless, is the boring retrieval plumbing nobody puts on a slide. The pattern has a name, **RAG**, and it's worth understanding properly before you greenlight a project around it, because the failure modes live in details that glossy vendor decks skip. ## Why not just train it on our data? The question everyone asks first. You *can* fine-tune a model on your corpus, and it's almost always the wrong first move. [Fine-tuning](/blog/pretraining-finetuning-rlhf/) teaches a model *style and behaviour*, not reliable *facts*: stuff a thousand documents into the weights and the model gets a vague, blurry sense of them, not a quotable memory, and it'll still [confidently make things up](/blog/why-models-make-things-up/) about the details. It's also a rebuild every time a document changes, and changed documents are the norm, not the exception. RAG sidesteps all of that. The model stays fixed; your data lives in a store you can update any time. Change a runbook, re-index that one file, done. The model reads the current text at question time instead of trying to remember a snapshot. For "answer from our documents", retrieval beats memorisation almost every time. ## How it actually works Two halves. A slow one you do up front, and a fast one that runs per question.
RAG: index once, then retrieve-and-answer per question
[diagram omitted — see the page for the chart]
Up front: split documents into chunks, turn each into an embedding, store them. Per question: embed the question the same way, pull the closest chunks, hand them to the model with the question. The model answers from text you just gave it, not from memory.
The one piece of genuine cleverness is the **embedding**. So matching a question to the right chunk isn't keyword search, it's *meaning* search: embed the question, find the stored chunks whose vectors sit nearest, hand those over. That's the magic in the demo. And it's also where it quietly goes wrong. ## The boring half is the whole game Here's the part I'd put in bold on the project kickoff slide: **the quality of a RAG system is almost entirely the quality of its retrieval, not its model.** If the right chunk doesn't make it into the prompt, no model on earth answers correctly, it either says it doesn't know, or worse, [fills the gap with something plausible](/blog/why-models-make-things-up/). The flashy generation step is only as good as the unglamorous fetch feeding it. Where I've watched these projects struggle, it's almost always retrieval, and almost always one of these: - **Chunking.** Split documents too small and you sever the context a sentence needed; too large and you bury the relevant line in noise the model has to wade through. There's no universal right size, and getting it wrong quietly tanks answer quality with nothing in the logs to show for it. - **The corpus is a mess.** Stale runbooks, three contradictory versions of the same policy, a wiki half of which is wrong. RAG will faithfully retrieve and cite your out-of-date document with total confidence. Garbage in, *confident* garbage out. It surfaces your documentation debt mercilessly. - **Permissions.** The instant your assistant can read "all company documents", it can leak the salary spreadsheet or the security review to whoever asks nicely. Retrieval has to respect who's allowed to see what, per user, per query, and that's real access-control engineering, not a checkbox. - **Tables, PDFs, images, diagrams.** Half your real knowledge isn't clean prose. Text extraction from a scanned PDF or a spreadsheet is its own ugly project, and what the embedder never sees, retrieval can never find. None of these are AI problems. They're data, plumbing and access-control problems, the exact things an ops team is already good at, which is rather the point. The model is the easy, mostly-solved bit you rent from an API. The data pipeline around it is the project, and it's the part that decides whether the thing earns its keep. ## What it's genuinely good for, and what it isn't

RAG shines

RAG struggles

The sweet spot is narrow and real: *look up the relevant passage and answer from it, with a citation you can click to check.* That last part is the quiet superpower, a RAG answer can show its sources, which turns the model from an oracle you have to trust into a research assistant you can verify, exactly the posture that [keeps an LLM in the right seat in your infrastructure](/blog/llms-in-your-infrastructure/). ## The honest recommendation If the requirement is "answer questions over our documents", RAG is the right pattern, not fine-tuning, not waiting for a model big enough to memorise everything. But scope it honestly: budget for the retrieval and data work as the main project, treat the LLM as a rented commodity, and demand citations so every answer is checkable. Build it that way and it's genuinely useful. Build it as "point a chatbot at the wiki and ship", and you've built a confident liar with a search bar. Same as everywhere else with this technology, the value is entirely in the discipline around it, and most of that discipline is plumbing you already know how to do.