Retrieval-augmented generation: what RAG is and how it actually works

A chatbot can get a fact wrong for several reasons: the relevant information may be missing, retrieval may fail, or the model may misread evidence already in its prompt. A model used without tools has its learned parameters and the text you supply; it does not automatically have your current codebase or yesterday's meeting notes. Retrieval-augmented generation, or RAG, adds a search step. The system finds relevant passages from a corpus, puts them into the prompt, and asks the model to answer using them. That can help, but it does not guarantee a correct answer.

This article is the long version of that one sentence. I want you to come away with a working mental model: what plain inference is doing, what RAG adds, where the seams are, and how a stack like hydra-llm bakes the whole pipeline into a few commands you can run on your own laptop. No prior machine-learning background needed. Where I have to use a technical term, I will define it before using it.

Plain inference, in the simplest possible terms, is what happens when you type a message into a chat interface and the model writes back. The model is a function. The function takes a sequence of tokens (a token is a sub-word piece, roughly four characters of English on average) and outputs a probability distribution over the next token. The runtime samples one token from that distribution, appends it to the input, and runs the function again. Repeat until the model emits a stop token or hits a length limit. That is the entire trick at the bottom.

The reason this produces sensible answers at all is that the function's parameters, billions of numbers tuned during training, encode an extraordinary amount of statistical regularity about how text is written. Train a model on a large slice of the public web, books, code, and synthetic data, and it learns enough about language and the world that the next-token distribution very often points at the right answer. Its learned information lives in those parameters; the prompt can also supply new information. There is no database the model is consulting. There is no fact-checker. The output is a confident continuation of the input, and confident is not the same as correct.

Without relevant information in either training or the prompt, a model cannot reliably recover a private fact or a later event. It can also make mistakes about facts that were in its training data. You can supply a new document directly in a prompt without RAG. Retrieval becomes useful when choosing the right material from a larger or changing collection would otherwise be manual work.

RAG changes the input. Before calling the model, retrieve relevant passages and provide them as context. The 2020 RAG paper by Lewis et al. studied combining a generator with retrieved material. A useful system still needs both good retrieval and a model able to interpret the evidence. Fluency alone is not enough.

From the model's point of view, very little changes. It still receives a prompt and emits the next token, and the next, and the next. If retrieval succeeds, the prompt now contains evidence that may support an answer. Retrieval has to find the right passages, and most of the difficulty sits there.

"The right passages" is where every interesting decision in a RAG system lives. You cannot just paste your entire knowledge base into the prompt; the model has a limited context window (the maximum number of tokens it can read at once) and even when the window is large, more context costs more time, more memory, and more attention budget that the model has to spread thin. So the retrieval step has to be selective: out of however many millions of tokens of source material, surface the few thousand that matter for this specific question.

The trick that makes selective retrieval feasible at scale is called vector search, and the building block underneath it is called an embedding. An embedding is a fixed-size list of numbers (a vector) that represents the meaning of a piece of text. A small embedding might be 384 numbers long; a large one, several thousand. The numbers themselves are not interpretable. What matters is that a well-suited embedder tends to place related passages near one another. That is learned behaviour, not a guarantee: negation, exact identifiers, and domain-specific language can still cause poor matches.

You get embeddings from a special kind of language model called an embedder. It looks like a chat model from the outside (you give it text, it gives you something back), but instead of emitting the next token it emits one fixed-size vector summarising the whole input. Embedders are usually small (anywhere from 30 megabytes for a tiny one to a few gigabytes for a large one), much smaller than chat models, because their job is narrower. Some are tuned for prose. Some are tuned for code. The best modern embedders are tuned for both and produce vectors of around 768 to 4096 dimensions.

Once you have an embedder, the indexing step looks like this. Walk through your corpus. Split each document into chunks, which are short, mostly self-contained passages of a few hundred words each. Run every chunk through the embedder. Store the resulting vector, alongside the original text and a pointer back to the source file, in a database that knows how to search by vector similarity. That database is called a vector store; LanceDB and Qdrant provide databases; FAISS is a vector-search library that can be part of a storage system. The result is an index in which every chunk of your knowledge has a coordinate in meaning-space.

Querying the index is the mirror image. Take the user's question. Run it through the same embedder. Now you have a query vector in the same meaning-space as your chunks. Ask the vector store for the chunks whose vectors are closest to it, by some distance measure (cosine similarity is the usual one). The top few chunks come back. Those are your candidates for "passages that probably answer this question." It is, fundamentally, a meaning-aware search engine.

Once retrieval has done its job, prompt assembly is the boring but load-bearing middle step. You take the chunks the vector store returned and you build the actual message you send to the model. There is no single right format, but most systems do something like this: a brief system instruction telling the model how to behave, a delimited block of retrieved context with each chunk labelled by its source, and finally the user's question.

You are an assistant. Answer the user's question using the context below.
If the context does not contain the answer, say you do not know.

<context>
--- src/auth/middleware.go:42-78 ---
func authenticate(r *http.Request) ...
--- README.md:54-72 ---
Auth flow: every request must carry an X-Api-Key header ...
</context>

User question: where do we handle auth tokens?

The chunks are labelled with their sources so a reader can check the answer. The instruction asks the model to rely on that context and admit when it is insufficient, though an instruction alone cannot guarantee compliance. The question goes last as a template choice. Liu et al.'s 2023 Lost in the Middle study found sensitivity to evidence position in the long-context tasks it tested. It did not establish one best question position for every model. Test the template with your own inputs.

The model then runs ordinary inference. Same function as before. Same next-token loop. The only thing that has changed is that the input now contains the relevant pieces of your corpus, so the most likely next tokens are the ones that draw on that text rather than on the model's frozen training memory.

Think of two stages. First retrieve evidence; then generate an answer from the question and that evidence. This is not necessarily two network round trips. Retrieval may involve an embedding call, a database lookup, keyword search, and reranking, or it may all run locally. Ordinary inference also uses the supplied prompt, not just information learned during training.

Retrieval adds work and can improve answers about your data when it finds the right evidence. It can also return irrelevant or outdated passages and make an answer worse. A useful interface shows the retrieved sources so the reader can check the answer. A document written last night can be used either by supplying it directly or by indexing and retrieving it; retraining the model is not required.

It is worth being honest about what RAG does not fix. RAG does not give the model new reasoning skills; if the underlying model is bad at multi-step reasoning, retrieving good context will not save it. RAG does not eliminate hallucination; the model can misquote, conflate two chunks, or outright ignore the context even with a carefully designed prompt. RAG does not magically protect privacy; if you index a folder full of secrets and then expose the chat to someone, you have just exposed your secrets. RAG is a way to feed better text to a competent model. The model still has to be competent, and the system around it still has to be designed honestly.

There is one more subtlety worth understanding before we get to the local stack. Pure vector search is not the only way to retrieve. There is a much older technique called keyword search (also called lexical search or BM25, after a particular ranking formula) that simply matches the literal words in the query against the literal words in the documents. Vector search is good at meaning ("auth token" finds a chunk that talks about "API key" because the embedder knows they are related). Keyword search is good at exact matches ("X-Api-Key" finds the chunk that contains that exact string, even if no embedder ever heard of it).

The two complement each other. Modern RAG systems often run both, get a list of top results from each, and combine them using a fusion algorithm. One widely used fusion algorithm is called Reciprocal Rank Fusion, and it has the pleasing property of being absurdly simple: each result gets a score of 1 / (k + rank) for each list it appears in, the scores are summed, and you sort by the total. k is a small constant, typically 60 in the paper that introduced it. That is the whole formula. The reason it works as well as it does is that it does not need the underlying scores from the two systems to be on the same scale; it only needs them to produce a ranked list. This same fusion idea applies when you index code and prose separately (a code embedder is often better at code and a prose embedder is often better at prose) and you want to query both indexes for one question.

The implementation walkthrough below describes hydra-llm's RAG work, which landed on 7 May 2026, after this article's original 3 May publication date. This section was checked and clarified on 15 October 2026. The v0.2.0 release commit is the dated source for the original pipeline.

Hydra-llm runs llama.cpp in Docker and can keep the chat model, embedder, and index on your machine. In that local configuration, prompts and retrieved passages need not go to a model provider. Setup still downloads software and weights. Choosing a remote endpoint changes where data is sent, so "local" is a configuration property, not a privacy guarantee attached to the word RAG.

The pieces hydra-llm puts in front of you are exactly the ones the previous sections introduced, with names you can type:

The choreography from the user's side is short. One command sets up the embedders. One command indexes a folder. One command starts a chat that retrieves on every turn.

# 1. Pull embedders sized to your hardware
hydra-llm rag setup

# 2. Index a folder. Walks the tree, classifies each file as code
#    or prose, chunks it line-aware (1500 chars target, 200 overlap),
#    embeds each chunk, writes everything to ./.hydra-index/
cd ~/projects/cool-app
hydra-llm index .

# 3. Chat with retrieval on every turn
hydra-llm chat llama-3.1-8b --rag .

What happens behind --rag . is exactly the two-pipeline picture from earlier. You type a message. hydra-llm sends it to the embedder container, gets a query vector back, asks LanceDB for the top matching chunks from both the code and prose tables, fuses the two lists with Reciprocal Rank Fusion, takes the top-K survivors, builds a <context> block with each chunk labelled by file path and line range, prepends that to your message, and only then sends the assembled prompt to the chat container. The chat container streams tokens back. Same model, same inference loop, smarter input.

A few details in that pipeline are worth pointing at, because they are the kind of decisions every RAG system has to make and they shape the answers you get. The chunker is line-aware: it never splits a chunk in the middle of a line, which matters for code more than it does for prose. Files are classified before chunking: code files get the code embedder, prose files get the prose embedder, as a design choice to evaluate. A specialised embedder may help, but a code/prose split is not automatically better for every corpus. The index moves with the folder: the .hydra-index/ directory sits next to your files, so copying a project to another machine takes the index along for free. Incremental updates are by file modification time and size: re-running hydra-llm index . re-embeds files whose recorded size or modification time changed. Content edits that preserve both can be missed, so this is a fast change-detection shortcut rather than a content hash.

One feature of hydra-llm worth singling out is what it calls catalog-bound bundles. A chat-catalog entry can carry a system prompt, sampling parameters, and a rag_index: path all together as one declarative unit. Once you have create'd a bundle, chatting with it is one word:

hydra-llm create llama-3.1-8b ~/personas/senior-engineer.md cool-app-bot \
    --rag-index ~/projects/cool-app

hydra-llm chat cool-app-bot   # no flags, retrieval just works

The bundle freezes the whole loop into a name: this model, with this persona, retrieving from this corpus, every turn. Once saved, the workflow is something you can use instead of repeatedly explaining it.

For anyone who wants to peek under the floorboards, this is roughly how the pieces are wired in the source. The whole RAG pipeline landed in v0.2.0, in commit 87b27e3 (release tag) and the merge that brought the feature branch into main, commit 0b4c7e1. The branch was split into four reviewable stages, and the design choices each one bakes in are worth naming, because they are the same choices any RAG system has to make.

Stage 1a: 5d6387c embedder catalog. Embedders are treated as a separate model species, not "chat models that happen to embed." They live in their own catalog at ~/.config/hydra-llm/embedders.yaml, with their own download/list/info subcommands and their own port range so they cannot collide with chat models. The decision behind that is mundane and important: an embedder is a long-lived sidecar. It must be running for indexing and for every chat turn that retrieves, but it is otherwise invisible. Modeling it as its own thing keeps that lifecycle honest.

Stage 1b: 44dd621 embedder sidecars and an embeddings client. Each embedder runs in its own llama.cpp container in --embeddings mode, exposing the OpenAI /v1/embeddings shape on a stable port. The client batches requests, normalises vectors (so cosine, dot-product, and Euclidean-distance rankings are equivalent for unit vectors), and lets the rest of the code treat embedding as "give me text, get back a numpy array."

Stage 1c: cb78cd9 walker, classifier, and line-aware chunker. Three honest little functions doing most of the heavy lifting that retrieval quality depends on. The walker respects every .gitignore in the tree, plus a hard-coded blacklist of node_modules, .venv, target, lockfiles, binaries, archives, media, and weights, and it skips files larger than 1 MB or anything that fails a UTF-8 decode. The classifier picks code or prose per file: extension first (so install.sh is correctly code, not prose just because it sits next to a README), then canonical basenames (Makefile, Dockerfile as code; README, LICENSE as prose), then a shebang sniff for extensionless scripts. The chunker has a soft target of 1500 characters with 200 characters of overlap, but it never cuts mid-line, which sounds trivial and is not: a chunk that ends in the middle of a function signature is genuinely useless to retrieval, because the embedder ends up vectoring half a thought.

Stage 1d: 2c98ade LanceDB storage and the index pipeline. This is where it all comes together. Each indexed folder grows a .hydra-index/ with two LanceDB tables (code.lance, prose.lance), a meta.yaml recording which embedder ids the index was built with, and a files.json that lists each indexed file by (path, size, mtime, chunk-ids). Two design decisions in this stage shape the whole user experience: incremental refresh diffs files by (size, mtime) so re-indexing a folder is fast on the second run, and a forced rebuild fires automatically if the recorded embedder ids no longer match what would be picked now, because mixing vectors from different embedders silently destroys retrieval and it is much better to spend a minute re-embedding than to spend a week wondering why answers got worse.

Stage 2: ef7b4cf chat with retrieval and federated query. The chat --rag path embeds each user message, runs k-nearest-neighbour search against both tables, fuses the results with Reciprocal Rank Fusion at k=60, and wraps the survivors in a <context> block before sending the assembled prompt to the chat model. --rag-all federates that retrieval across every indexed folder on the machine; --rag-tag federates across folders that share a tag. Saved chat sessions store the original (un-augmented) text, not the augmented version, so resuming a session does not poison the new turn with stale context.

Stage 3: de360fe catalog-bound bundles. A chat-catalog entry can carry a rag_index: field directly, alongside system_prompt and params. create bakes a model + a persona + a corpus path into one declarative alias, and from that point on, chat <alias> with no flags reproduces the entire workflow. The catalog file is plain YAML you can copy between machines.

Three other decisions show up across the diff and are worth pulling out, because they are the kind of thing you only learn after the second or third RAG system you build. Code and prose use different embedders by default, as an initial design choice; later releases (0e9b939, 687d7ed) made the dual-index optional and single-embedder the new default, to simplify setup. The resource savings depend on the selected models and corpus; this article does not present an accuracy or timing comparison proving an exact halving. Vectors are normalised at write time, and the cited query path uses LanceDB's default vector search rather than explicitly selecting dot-product distance. For unit vectors the ranking is equivalent. The index is co-located with its source: if you copy ~/projects/cool-app to a different machine, the .hydra-index/ goes along, and reuse also depends on compatible storage/runtime versions and valid source-path metadata, as well as the same embedders. None of these are exotic; all of them save real time once your indexed corpus is more than a toy.

The useful mental model is a text continuation engine with frozen knowledge connected to a search step. RAG leaves the engine alone and changes what enters it. Feeding the model relevant passages from a corpus you control turns a closed function with fixed knowledge into a system that can answer about your code, notes, contracts, or anything else you indexed half a minute ago. The model still has to read and write well. Retrieval has to find the right text. The prompt has to present that text honestly. When all three pieces work, the result is the experience most people imagine when they hear "AI assistant for my own data": fast, grounded, and verifiable, with no mystery cloud step in the middle.

Building this locally keeps the corpus, embedder, chat model, and index on your disk. Sensitive data stays under your control, and less sensitive work still benefits from lower variable cost and fewer external dependencies. RAG gives you a pipeline for deciding what the model may read before it answers. After running that pipeline yourself a few times, the limits of what a model knows become much less surprising.