Own Your AI is a nine-part Top Tool Stack series on running your own local AI instead of renting it from the giants. This is Part 6 of 9. See the whole series →
You have picked a tool. Now, which model do you run inside it? This is the question that trips people up, because the honest answer is “the biggest one your hardware can comfortably fit, matched to the job”. Let’s make that concrete, with no marketing and no promising you a free GPT-5.
First, the one concept that decides everything: VRAM
A model’s size is measured in billions of parameters (the “7B”, “70B” in the name). Roughly, the model needs to fit into your graphics card’s memory (VRAM), or your Mac’s shared memory, to run fast. Quantisation (that compression trick from Part 3) shrinks the requirement. As a rule of thumb for 4-bit quantised models: a 7-8B model wants about 8GB, a 20B model about 12-16GB, a 70B model about 48GB. Go over your budget and the model still runs, just slowly, by spilling onto the processor.
Pick by hardware
Laptop, 8GB, no dedicated graphics. Stick to 3B-8B models. Google’s Gemma (small versions) and Llama 3.1 8B are excellent, friendly all-rounders. Qwen‘s smaller models are strong at reasoning and coding. Expect a capable everyday assistant, not a genius.
Gaming PC or Mac, 16GB with an 8-12GB graphics card. The sweet spot. Run 8B models very comfortably, or stretch to a quantised ~20B like GPT-OSS-20B for a noticeable quality bump. This tier handles the large majority of real work well.
Enthusiast, 24GB+ VRAM (or a 64GB+ Apple Silicon Mac). Now 70B-class models like Llama 3.1 70B are on the table, and these genuinely start to feel “serious”. A high-memory M-series Mac is a quietly brilliant machine for this because of its shared memory.
Home rig, multiple GPUs. The territory of the very large open models (100B and up, like GPT-OSS-120B or Qwen’s biggest). This is where you approach frontier-adjacent quality at home, and also where the electricity bill starts to matter, which is exactly what Part 9 is about.
Match the model to the job
Bigger is not always the right call. A small, fast model is often the better choice for simple, repetitive tasks, and it uses a fraction of the energy. A sensible split:
- Everyday chat, summarising, drafting: a good 8B model (Gemma, Llama 3.1) is plenty.
- Coding help: Qwen’s coder models and GPT-OSS punch above their size.
- Reasoning and analysis: reach for the largest model you can run; this is where size shows.
- Questions about your own documents (RAG): the retrieval setup matters more than raw model size; a mid-size model plus good document search beats a huge model guessing.
A neat trick, if your tool supports it (Odysseus has a “Cookbook” that does exactly this): let the software scan your hardware and suggest models that will actually run, rather than downloading a 200B checkpoint you can never load.
The honest ceiling
Here is the part the enthusiast forums sometimes gloss over. Even a strong 70B model running at home still trails the frontier cloud systems on the hardest reasoning, the longest documents and the most obscure knowledge. The gap has narrowed dramatically, open models now handle a large share of everyday queries perfectly well, but it has not closed. Run local for the bulk of your work, keep a cloud model for the genuinely hard 10% if you need it, and you are getting the best of both worlds without pretending the trade-off does not exist.
Next week, the unglamorous but essential bit: doing all of this safely.
The AI tool stack actually worth paying for
One email a week. The tools, models and moves that matter, minus the hype and the horseshit filter set to maximum. Free.