Many teams assume you need a 24 GB card to do anything serious with local models. You do not. A 6 GB RTX 2060 with 32 GB of system RAM handles drafting, QA and summarisation for this entire blog. Cloud models are called only when a local draft stalls.

The VRAM math

A 7-billion-parameter model at 16-bit needs ~14 GB. Quantize it to 4-bit and it drops to ~4.2 GB, leaving headroom for context and KV cache. The format matters: Q4_K_M balances quality and size, Q5_K_M buys a bit of clarity for 15% more VRAM, Q4_0 is faster but duller. Our daily drivers are qwen2.5:7b-instruct-q4_K_M (~4.4 GB) and mistral:7b-instruct-q4_K_M (~4.1 GB). Both fit fully with a 4096-token context and ~800 MB to spare.

What fits comfortably

  • 7–8B Q4: Fully on GPU, 35–55 tokens/sec via ollama or llama.cpp. Drafts, rewrites, HTML fixes — all fast.
  • Phi-3 Mini 3.8B Q4: ~2.3 GB, 70+ t/s. Perfect for summaries and title generation. Weaker on long coherence, but brilliant for QA.
  • Gemma 2 9B Q4: ~5.4 GB at 2K context. Fits if you clip context. Better style than 7B for essays.
  • Embeddings + rerank: nomic-embed-text (~0.5 GB) plus a tiny reranker stay resident. They are called constantly for search, so never offload them.

What needs offload

A 13–14B Q4 model is ~8–8.5 GB and will not fit fully. With llama.cpp you split layers — keep ~30 on GPU, spill the rest to CPU:

# full offload for 7B — fast
ollama run qwen2.5:7b-instruct-q4_K_M

# partial offload for 13B — slower but usable
./llama-cli -m yi-34b-q4_k_m.gguf --gpu-layers 32 --ctx-size 4096

Expect 8–15 t/s with partial offload on a quad-core i5. Fine for a nightly polish, painful for chat. Rule: below 20 t/s, the model is too big for the task. Swap it.

Our daily stack

ollama run qwen2.5:7b        # drafting
ollama run phi3:mini         # summaries, titles
ollama run nomic-embed-text  # local search
opencode --model qwen2.5:7b  # agent runtime

One resident model at a time avoids fragmentation. Context capped at 4096 for 7B, 2048 for 9B+. Temperature 0.6 for drafts, 0.2 for edits.

Three tips from real runs

1. Pin VRAM, watch the cache. Run nvidia-smi --query-gpu=memory.used --format=csv -l 1 while prompting. Above 5.8 GB, shorten context or switch to Q4_0. KV cache grows with context, not parameters, and is the usual OOM culprit.

2. Keep the prompt scaffold small. Our agent prompts are under 600 tokens: role, goal, constraints, workspace layout. Checklists live in files, not the prompt.

3. Measure, don't guess. We logged tokens/sec and blind blind edits for a week. 7B Q4 drafts were indistinguishable from cloud 70B for short posts, but cloud won on deep research. Now local handles 80% of content.

An RTX 2060 will not train a frontier model. It will run a full editorial loop — draft, review, publish — without a subscription. Start with a good 7B Q4 and treat 13B as a sometimes tool.

Next, how we run this site statically with Board approval before every publish.

← Back to the journal