6 Local AI Models You Can Run Today: Sizes, Context, and Licensing

TLDR: Local AI models are open-weight LLMs you download and run on your own hardware, no API calls, no per-token billing. The practical shortlist in September 2026 is Qwen, Llama, Gemma, Mistral, Phi, and DeepSeek families, each spanning sizes from roughly 1B to 70B parameters. What separates a good choice from a bad one is not benchmark scores, it is whether the model's size fits your RAM, its license fits your product, and its context window fits your documents. This guide gives you the six families, the numbers that matter, and the test that prevents a wasted weekend.

Local AI models are open-weight language models you download and run yourself. You pick the file, you pick the hardware, and you own the latency and the privacy posture. That control is the whole point: a chat app for a regulated employer, a coding assistant on an air-gapped machine, or a data pipeline that cannot send customer text to a third party all fail with a hosted API and work with a local model.

This is the model-selection spoke of our local AI cluster. If you are earlier in the journey, start with the hub: what running an LLM locally actually involves. If you already know your hardware and want the software stack, the best local LLM guide covers runtimes like Ollama and llama.cpp in depth.

What Does "Local" Actually Require?

A local model needs three things: the weights (a file you download, typically several gigabytes), a runtime that loads and executes them (Ollama and llama.cpp are the two most used), and enough memory to hold both the weights and the working context. The weights are quantized, meaning compressed to a fraction of their original precision, and quantization is why a "70B" model can fit in 43GB rather than 140GB: Ollama lists llama3.1:70b at 43GB with a 128K context window (Ollama model library, accessed 2026-09-04).

The rule of thumb that saves people the most pain: your model file size plus a few gigabytes of overhead must fit in available RAM (or VRAM if you want GPU speed). An 8B model at standard quantization is roughly 5GB, which is why 8B-class models are the default recommendation for laptops, and 70B-class models assume a workstation with a large GPU or a Mac with unified memory.

Which Model Families Should Be on Your Shortlist?

Six open-weight families cover nearly every local use case in September 2026. Sizes and licensing here are from each project's own model pages, accessed 2026-09-04.

1. Qwen (Alibaba). The Qwen3 line on Ollama spans roughly 0.6B to 30B-plus in commonly run sizes, with the default tag at 5.2GB and a 40K context window (Ollama model library, accessed 2026-09-04). A 0.6B variant at 523MB is the smallest model in this guide that is still useful for classification. Check the model page for the license of the specific size you ship, licenses vary within the family.

2. Llama (Meta). Llama 3.1 ships in 8B, 70B, and 405B parameter sizes with a 128K context window (Ollama model library, accessed 2026-09-04). The 8B at 4.9GB is the classic laptop pick, and the 70B at 43GB assumes a workstation-class machine. The license is the Llama Community License: fine for most products, but it carries conditions at very large scale, so read it before you ship.

3. Gemma (Google). Gemma 3 is available in 270M, 1B, 4B, 12B, and 27B parameter sizes. The 270M and 1B have a 32K context window and text-only input; the 4B and above have a 128K context window and support both text and image input. The default tag is 3.3GB (Ollama model library, accessed 2026-09-04). The 4B is the sweet spot when an 8B feels heavy but a 1B feels thin. The license is Google's custom Gemma terms, not a standard open license.

4. Mistral (Mistral AI). The original Mistral 7B is distributed with the Apache license and runs at 4.4GB with a 32K context window (Ollama model library, accessed 2026-09-04), which made local inference practical on consumer hardware in the first place. Apache licensing is the quiet advantage here: no conditions at scale, no reading required before you ship a product on it.

5. Phi (Microsoft). Phi-4 is a 14B model that runs at 9.1GB with a 16K context window (Ollama model library, accessed 2026-09-04), and the Phi line is known for strong reasoning and math at its size, a product of its synthetic-heavy training data. The classic tradeoff: brilliant on textbook-style tasks, thinner on open-ended web knowledge, and the 16K window is the tightest in this guide, so long documents need chunking.

6. DeepSeek. DeepSeek-R1's weights are licensed under MIT, support commercial use, and allow modifications and derivative works. The distilled versions span Qwen-based sizes from 1.5B at 1.1GB up to the current default 8B at 5.2GB (based on Qwen3), plus a Llama-based 70B, all with a 128K context window (Ollama model library, accessed 2026-09-04). The full R1 is a 671B-parameter model, which is why the distills are the local story. Read the model page for the license of the distill you pick, the base-model licenses differ.

How Do You Choose Between Sizes?

Match the size to the job, not to the leaderboard. The decision framework has three inputs: available RAM (sets the ceiling), expected concurrency (each simultaneous request holds its own context copy), and the task's tolerance for error. An 8B model summarizes and extracts reliably. Coding and multi-step reasoning improve noticeably at 30B to 70B. Anything below 4B is for classification and routing, not generation.

The cheapest way to be wrong is to buy hardware for a 70B model when your task only needs 8B. The second cheapest is the reverse: shipping a 3B model into a summarization product and discovering it hallucinates entity names. Test on your own data before committing, described below.

What Does the Working Setup Look Like?

Ollama is the fastest path from zero to a running model, and its CLI is three commands. This example pulls and runs the 8B Llama 3.1, the standard laptop-class pick (Ollama documentation, accessed 2026-09-04).

ollama pull llama3.1:8b   # downloads ~4.9GB once
ollama run llama3.1:8b   # opens an interactive chat session
ollama list           # shows what you have and how big it is

For programmatic use, Ollama exposes a local HTTP API. Here is a minimal Python client with the error handling that matters in a real loop, connection failures and malformed responses, rather than the happy path only.

import json, urllib.request, urllib.error

OLLAMA_URL = "http://localhost:11434/api/generate"
MODEL = "llama3.1:8b"

def generate(prompt: str, retries: int = 1) -> str:
    payload = json.dumps({
        "model": MODEL,
        "prompt": prompt,
        "stream": False,
    }).encode()
    req = urllib.request.Request(
        OLLAMA_URL, data=payload,
        headers={"Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            data = json.loads(resp.read())
            return data["response"]
    except urllib.error.URLError as exc:
        if retries > 0:
            return generate(prompt, retries - 1)
        raise RuntimeError(f"Ollama unreachable, is it running? {exc}")
    except (KeyError, json.JSONDecodeError) as exc:
        raise RuntimeError(f"Unexpected response shape: {exc}")

if __name__ == "__main__":
    print(generate("Summarize the CAP theorem in two sentences."))

Every number in that script is verifiable: the model tag and file sizes come from the Ollama model library page, and the API shape follows the Ollama documentation. For the deeper alternatives in runtimes, the llama.cpp project on GitHub is the reference implementation most other tools build on, and Hugging Face's model hosting is where the weights themselves live.

What Is the Failure Mode That Ruins Local Deployments?

Context overflow, and it fails silently. A model with a 128K context window does not gracefully degrade when you feed it 200K tokens: depending on the runtime it truncates the input, errors mid-request, or slows to a crawl while memory swaps. The symptom users report is "it got worse when I added more documents," which looks like a model-quality problem and is actually a length problem.

Detection is a token count before the call. If your pipeline ingests whole documents, count tokens client-side and chunk anything over roughly 80 percent of the window, because the prompt and the generated reply share the budget. Log the count with every request so the failure is attributable after the fact.

Where Do Local Models Stop Being the Right Tool?

A local model knows only what is in its weights, and weights are frozen at training time. Anything that depends on current information, prices, news, library versions, competitor moves, needs retrieval, and retrieval means search. This is the line between the local cluster and our API work: the same pipeline that runs a local model can call the You.com Web Search API for the fresh half of the job, keeping inference private while sourcing live context. Our LLM web search guide covers that wiring, and if privacy is the driver rather than cost, our self-hosted search alternatives guide explores keeping the search side local too.

Next action: pick the size that fits your RAM from the shortlist above, run the three Ollama commands, and evaluate it on ten of your own documents before you commit anything. An afternoon of testing beats a quarter of regret.