Skip to main content

Run AI Models Locally with Ollama

Intermediate

The frontier models live in the cloud, but a whole class of open-weight models — Llama, Mistral, Qwen, DeepSeek, Gemma — you can download and run on your own laptop or server. Ollama is the simplest way to do it: one install, one command, and a model is running locally. This page gets you from zero to a running model, calling it from code, and knowing the trade-offs so you don't expect a 7B local model to behave like a frontier one.

What you'll learn
  • Know WHY you'd run a model locally — and the honest trade-offs vs cloud frontier models
  • Install Ollama and run your first model in two commands
  • Use the core CLI: pull, run, list, ps, rm, stop
  • Call your local model from code via Ollama's OpenAI-compatible API
  • Understand RAM needs, model sizes and quantization — so you pick a model your machine can actually run
  • Know when LM Studio (a GUI) is the better starting point

Why run a model locally (and when not to)​

Running a model on your own hardware buys you four things:

  • Privacy — your prompts and data never leave your machine. The reason teams in regulated or sensitive domains reach for local models. (See the privacy/self-host fork in Choosing a Model.)
  • Offline — no internet, no API, no outage dependency. The model is a file on your disk.
  • Cost — no per-token bill. Once it's downloaded, running it is "free" (you pay in electricity and hardware, not API spend). If token spend is your constraint at scale, local can win.
  • Control — pin an exact version, customize behavior, and integrate without rate limits or terms-of-service surprises.

The honest trade-offs:

  • Capability gap. A model you can run on a laptop (1B–14B parameters) is not in the same league as a frontier cloud model on hard reasoning, long-context, or agentic tasks. For many everyday tasks the gap is small; for the hardest ones it's large.
  • Hardware. Bigger, more capable models need more RAM/VRAM than a typical machine has. You're often choosing the largest model your hardware can run, not the best model that exists.
  • You operate it. No managed scaling, no automatic upgrades — that's the price of control.

A common, durable pattern: prototype and pick with a tiny eval (same method as cloud models — see Choosing a Model), use local for private/offline/cheap-at-scale work, and reach for a frontier API when the task genuinely needs the extra capability.

Install Ollama and run your first model​

Guided walkthrough1 of 4
  1. macOS and Windows: download the installer from ollama.com/download and run it. Linux: use the official install script (below). This installs the ollama command and a local background service.

Linux install (macOS/Windows use the downloaded installer instead):

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a small model (good first choice)

ollama run llama3.2

That single command both downloads llama3.2 (a small 1B/3B-class model that runs on modest hardware) and starts an interactive chat. To download without chatting yet, use ollama pull instead.

The core CLI​

A handful of commands cover almost everything. Run ollama --help for the full list.

# Download a model without starting a chat
ollama pull qwen3

# Start an interactive chat (downloads first if needed)
ollama run qwen3

# One-shot: pass the prompt inline, get the answer, exit
ollama run qwen3 "Summarize the CAP theorem in two sentences."

# List models you've downloaded
ollama list

# Show models currently loaded in memory
ollama ps

# Stop a running/loaded model (frees memory)
ollama stop qwen3

# Delete a downloaded model to reclaim disk
ollama rm qwen3

Inside an interactive ollama run session, type /bye to exit and /? to see in-session commands. The background service that actually serves models is started by ollama serve (the desktop app starts it for you automatically).

Call it from code (OpenAI-compatible API)​

This is the part that makes local models genuinely useful in apps. The Ollama background service exposes a local HTTP API at http://localhost:11434, on default port 11434. It has its own native endpoints (/api/generate, /api/chat) and an OpenAI-compatible layer at /v1 — so most code written for the OpenAI SDK works against your local model by changing two lines.

Raw HTTP with curl (native endpoint):

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "Why is the sky blue?" }
  ],
  "stream": false
}'

From Python using the official OpenAI SDK — just point base_url at Ollama and pass any non-empty api_key (Ollama requires the field but ignores its value):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the SDK, ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain gradient descent in two sentences."},
    ],
)

print(response.choices[0].message.content)

Because it's OpenAI-compatible, the same patterns you'd use against a hosted API carry over — including streaming and tool use / function calling, which work locally the same way they work in the cloud (subject to the model actually being good at it). Ollama's /v1 layer covers /v1/chat/completions, /v1/completions, /v1/models and /v1/embeddings, among others.

Hardware, model sizes and quantization​

The number you'll see attached to a model — 1B, 7B, 8B, 70B — is its parameter count (in billions). More parameters generally means more capable and more memory-hungry. The practical limit on what you can run is RAM (and on a GPU, VRAM).

Quantization is the key trick that makes this feasible on normal machines. The model's weights are originally stored in high precision (e.g. 16-bit), but they can be compressed to ~4-bit with a small, often barely-noticeable quality cost. Ollama ships most models quantized by default — that's why a model with billions of parameters can fit in a few GB. A rough rule of thumb (verify against the specific model's page):

  • ~1B–3B models: run comfortably on most modern laptops (a few GB of RAM).
  • ~7B–8B models: the sweet spot for capable-yet-runnable; budget several GB of free RAM.
  • ~13B–14B models: need a fairly well-specced machine.
  • ~70B+ models: need a workstation/server with a lot of RAM or a strong GPU — not typical laptop territory.

A practical recipe: start with a small model (llama3.2), confirm the workflow end-to-end, then size up to the largest model that still runs smoothly on your hardware — not the largest that exists.

Local-models vocabulary
Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 6

LM Studio: a GUI alternative​

If a command line isn't your thing, LM Studio is a desktop app (macOS, Windows, Linux) that does the same job through a graphical interface: browse and download open models, chat with them in a built-in UI, and — like Ollama — run a local OpenAI-compatible server so your code can talk to it. It's the easier on-ramp for non-developers or anyone who prefers point-and-click over a terminal; Ollama tends to win when you want a scriptable CLI and a lightweight background service. Both run the same kinds of open-weight models, so the concepts on this page transfer directly.

Check yourself

0/4
  1. What's the single biggest honest trade-off of running a model locally vs a frontier cloud model?
  2. Which command both downloads a model (if needed) AND starts chatting with it?
  3. To call your local model from OpenAI-SDK code, what do you set?
  4. Why can a model with billions of parameters fit in just a few GB of RAM?
Key takeaways
  • Local = privacy + offline + no per-token cost + control; the trade-off is a real capability gap and hardware limits.
  • Two commands get you running: install Ollama, then ollama run llama3.2.
  • Core CLI: pull (download), run (chat), list, ps, stop, rm — that's most of it.
  • Call it from code via the OpenAI-compatible endpoint: base_url http://localhost:11434/v1, any non-empty api_key.
  • Parameter count + quantization decide if your machine can run a model — start small, size up to what runs smoothly.
  • Prefer a GUI? LM Studio does the same job point-and-click and also exposes a local OpenAI-compatible server.

Sources & further reading​

Next​