Every local search box, related-posts grid and retrieval pipeline starts with the same tiny step: turning text into a list of numbers that captures its meaning. That step is an embedding, and unlike chat models it is so cheap that even a 6 GB card can keep it loaded forever. This guide shows which embedding models fit consumer GPUs, how much VRAM they really take, and three ways to run them locally. For the one-paragraph definition, see what an embedding is; for the card they will share, see what fits on an RTX 2060.
Why embeddings belong on the GPU permanently
Chat models load and unload as tasks come and go, but embeddings are called constantly: every search query, every related-post computation, every document ingested into a retrieval index. Reloading the model for each call would add seconds of startup to millisecond work, so the standard practice is to keep one embeddings model resident at all times. Fortunately the sizes make this easy. The popular nomic-embed-text model downloads at about 274 MB through its Ollama library page, mxbai-embed-large at about 669 MB via its library page, and bge-m3 at about 1.2 GB via its library page. Even the largest of the three is a rounding error next to a 4 GB chat quant, and the weights behind the first are documented on the nomic-embed-text model card with an 8K-token context window that swallows whole articles in one pass.
The arithmetic for a 6 GB card is therefore comfortable: roughly 4.4 GB for a 7B chat quant, under 0.7 GB for the embeddings model, and the remainder for KV cache and headroom. That is exactly the layout we run daily, with the embeddings model pinned while chat models come and go, as described in the RTX 2060 field report. The general sizing logic, weights plus context cache plus one resident helper model, is the same VRAM math covered in our quantization guide.
Picking a model without a leaderboard obsession
Three models cover nearly every local need. nomic-embed-text is the default: small, fast, permissively licensed, and strong enough that most builders never outgrow it. mxbai-embed-large trades about twice the VRAM for higher dimensionality and better separation on tricky topical queries. bge-m3 adds multilingual coverage across dozens of languages for collections that mix tongues, with weights documented at the BGE-M3 model card. When in doubt, start with the smallest and upgrade only when real queries fail, because chunking strategy moves retrieval quality far more than the gap between these three.
Two numbers on each model card deserve a glance. The embedding dimension, typically 768 for the small models and 1024 for the large one, sets the storage cost of your index: one million 768-dimensional float vectors need about 3 GB on disk before compression, so dimension is a storage dial as much as a quality dial. The context window, up to 8192 tokens on the modern cards linked above, sets the longest chunk you can embed in one call. Rankings for the genuinely curious live on the public MTEB leaderboard, which evaluates embedding models across retrieval, clustering and classification tasks; treat it as a tiebreaker between finalists, not as a reason to chase the top slot for a personal wiki. The sentence-transformers documentation remains the friendliest entry point to the underlying techniques.
Three ways to run embeddings locally
Option one is Ollama, which serves an embeddings endpoint beside the chat API documented in the Ollama API reference:
ollama pull nomic-embed-text
curl http://localhost:11434/api/embed \
-d '{"model": "nomic-embed-text", "input": "quantization trades precision for memory"}'The response is a JSON array of floats, one vector per input string, ready to store. Batch many short texts per call rather than looping one by one; throughput is dramatically better and the GPU stays fed.
Option two is the llama.cpp server, which exposes an embedding endpoint from any GGUF embedding model as described in its repository documentation:
./llama-server -m nomic-embed-text-v1.Q4_K_M.gguf --embedding --port 8080
curl http://localhost:8080/embedding -d '{"content": "quantization trades precision for memory"}'This path shines when Ollama does not package the exact model you want, or when you need the server on a nonstandard port inside a larger compose stack.
Option three is plain Python with sentence-transformers, which skips the server entirely and is ideal for one-off indexing scripts, following the patterns in the official documentation:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1")
vectors = model.encode(["quantization trades precision for memory"],
normalize_embeddings=True)Normalizing to unit length at encode time means cosine similarity later reduces to a dot product, a small decision that keeps every downstream comparison fast and numerically calm.
Squeezing embeddings even smaller
When every megabyte counts, the embedding model itself can be quantized. A 4-bit GGUF of nomic-embed-text cuts the resident size roughly in half again with only a small retrieval penalty, using the same quantization machinery described in our quantization guide and documented in llama.cpp. Prefer this only after the layout is proven: first keep the full-precision embed model resident and confirm the whole card budget holds, then step down if headroom is tight. For most 6 GB layouts the unquantized model already fits with room to spare, so treat embed quants as a reserve parachute, not the default packing list. Throughput is rarely the constraint either way; even laptop CPUs embed hundreds of short passages per second, and the GPU path mostly matters for keeping latency flat while chat generation runs beside it.
Keeping it resident next to a chat model
Two settings protect the layout. First, tell Ollama never to evict the embeddings model: the keep_alive parameter in the API reference accepts a duration, and setting it to a very long value for the embed model while leaving the chat default alone gives you a permanent resident plus a rotating guest. Second, verify instead of assuming: run nvidia-smi --query-gpu=memory.used --format=csv -l 1 while embedding a batch and chatting at the same time, and confirm the total sits below the card limit with margin. On our 6 GB card the pair lands near 5 GB, leaving about a gigabyte for cache growth, and the moment usage creeps past 5.8 GB we shorten chat context rather than evicting embeddings. Evicting the thing every query needs in order to widen the thing one query needs is always the wrong trade.
What actually determines search quality
Newcomers upgrade the embedding model when they should fix their chunks. A model can only compare the texts it receives, so a 2000-word page embedded as one vector blurs every topic on the page into mush, while the same page split into focused 300-to-800-token passages retrieves precisely. Overlap consecutive chunks by a sentence or two so boundaries never sever a thought, prepend each chunk with its document title so the vector carries provenance, and strip boilerplate like navbars before embedding rather than after. Evaluate by reading: keep a list of twenty real questions, inspect the top three hits for each, and only change the model when the chunks are good and the hits are still wrong. This inspection habit is also the core skill behind the local RAG walkthrough, where these vectors become the retrieval half of grounded answers.
Keep going
Embeddings are the always-on half of the local stack; the chat model is the reasoning half. Pair this guide with quantized models explained for sizing the chat side, RAG on your own machine for putting retrieval and generation together, and small models on a 6 GB card for the full resident layout. The workflow around it all is the local agent team, the glossary entries are embeddings and RAG, and the series hub is the local-AI topic.