A chat model knows the world up to its training date and nothing about your notes. Retrieval-augmented generation fixes that by fetching the relevant passages first and handing them to the model alongside the question, so answers stay current and can point at their sources. This guide builds that loop entirely on your own machine: a local chat model, a local embeddings model, and an index you can inspect. The pattern itself is defined in what RAG is and originates with the retrieval-augmented generation paper; the two models it needs are sized in our quantization guide and our embeddings guide.
The loop in five stages
Every RAG system, from a weekend script to a production cluster, runs the same five stages. First, chunk: split documents into focused passages of a few hundred tokens each, because one vector per whole page blurs every topic together. Second, embed: convert each chunk into a vector with a local embedding model and store the pairs. Third, retrieve: embed the incoming question the same way and pull the nearest chunks by cosine similarity. Fourth, generate: place those chunks into the chat model context next to the question with instructions to answer from the evidence. Fifth, cite: require the answer to name the chunks it used so every claim stays checkable. When answers go wrong, the culprit is almost always stage one or three, never the size of the chat model, which is why this guide spends its budget on chunks and retrieval rather than a bigger quant.
The parts list for a 6 GB card
The whole stack fits the card we already run. A 7B chat quant around 4.4 GB handles generation, an embeddings model under 0.7 GB handles stages two and three, and the index itself lives on disk, not in VRAM. Concretely: pull qwen2.5:7b-instruct-q4_K_M from the Ollama Qwen2.5 page and nomic-embed-text at about 274 MB from its library page, keep both resident as laid out in the RTX 2060 report, and serve them through the chat and embeddings endpoints documented in the Ollama API reference. For the index, start with a plain JSONL file and graduate to a real vector store such as Chroma or Qdrant once you pass a few thousand chunks; the single-file sqlite-vec extension is a fine middle step. Frameworks like LlamaIndex package all five stages if you prefer assembly over hand-rolling, but build the manual version first so every later abstraction stays legible.
Walkthrough part one: build the index
Take a folder of plain-text notes and index it with a short script. Split each file into passages of roughly 300 to 800 tokens with a sentence or two of overlap, prepend the file name to every passage so each vector carries its provenance, and skip boilerplate before embedding rather than after. These ranges are starting points from our own site index, not universal constants: short crisp notes want smaller chunks, long essays want larger ones, and the inspection habit in part three tells you which way to move. Embed each passage through the local endpoint:
curl http://localhost:11434/api/embed \
-d '{"model": "nomic-embed-text", "input": ["passage one text", "passage two text"]}'Store each returned vector beside its text in one JSON line per chunk: file name, chunk number, character offsets, and the vector itself. A thousand chunks of a 768-dimensional model fit comfortably in tens of megabytes of JSON, which is why no database is needed yet. Verify the index by embedding a question and scoring cosine similarity in a few lines of Python: normalize all vectors once, take dot products, and print the top three texts. Read them with your own eyes. If a human cannot see why those three won, fix the chunks before touching anything else.
Walkthrough part two: answer from evidence
Retrieval earns its keep in the prompt. Send the chat model the question plus the winning chunks inside a template that leaves no room for improvisation:
Answer the question using ONLY the passages below.
Passage [small-models-rtx2060 #3]: ...
Passage [local-agent-team #7]: ...
Question: Which models fit a 6 GB card?
If the passages do not contain the answer, say so plainly.
End every factual sentence with the passage label you used, like [small-models-rtx2060 #3].Generate through the local chat endpoint with a low temperature so the model stays close to the supplied text, following the chat format in the API reference. The labels do double duty: they force grounding during generation and they become clickable citations in the rendered answer, which readers can check against the same files you indexed. Retrieve three to five chunks per question as a starting point: one chunk starves the model, ten drown it, and the inspection loop below tells you where your collection lands. This evidence-first discipline is the same habit behind our agent workflow, where every claim ships with a reproducible artifact.
Tuning what matters, ignoring what does not
Keep twenty real questions and read the top three hits for each after every change; that list is your evaluation harness until the collection outgrows it. The dials that move results, in order, are chunk boundaries, the number of retrieved passages, overlap size, and only then the embedding model. Search quality problems almost always trace to a chunk that mixes two topics, a question whose answer spans a boundary the splitter severed, or stale files indexed months ago. Refresh the index whenever sources change, timestamp every chunk so staleness is visible, and consider a tiny reranker only after the basics are solid. Skip hybrid keyword-plus-vector search until plain vectors demonstrably fail on your data; most personal and team collections never need it. And keep the chat model fixed while tuning retrieval, because changing both ends at once makes every experiment unreadable.
Honest limits
Local RAG inherits every weakness of its index. Contradictory sources produce confident contradictions, undated notes answer with expired facts, and anything you never indexed might as well not exist, so curate the collection like a library rather than dumping a drive into it. Very long multi-hop questions that need synthesis across dozens of passages still favor large cloud models with giant context windows; route only those upward and keep everything else local.
One risk deserves its own warning: retrieved passages are untrusted input wearing a trusted uniform. A note that says to ignore previous instructions becomes part of the prompt the moment it is retrieved, which is exactly the prompt injection pattern in miniature. Keep the agent that acts on RAG answers narrowly permissioned, show citations so a human sees which passage drove each claim, and never let retrieved text reach tools or credentials without review. The index stays useful precisely because it is treated as evidence to check, not orders to follow.
Private data stays private throughout, which is the entire point: nothing in this loop leaves the machine, there is no per-query bill, and the index works offline on a train. Measure the split for a week as we did and you will likely find, as in our measurements, that local handles the large majority of daily questions.
Keep going
The loop is complete: size the models with quantized models explained, serve search with embeddings on consumer GPUs, and ground answers with this walkthrough. The companion notes are small models on a 6 GB card and the local agent team, the glossary entries are RAG and embeddings, and everything in the series hangs off the local-AI topic.