Skip to main content

Building Local AI Agents

Advanced

A local AI agent is an autonomous loop that runs entirely on your own hardware: an open-weight model (served by Ollama or LM Studio) decides what to do, calls tools you give it, reads the results, and keeps going until the task is done — with nothing leaving your machine. No cloud API, no per-call bill, no internet required. The catch: a model small enough to run on a laptop is weaker at hard reasoning and long-horizon planning than a frontier model, and you own its reliability and safety. This page covers the honest case for local agents, the minimal architecture, what actually runs locally, and a realistic path to your first one.

What you'll learn
  • Know WHY you'd build an agent that runs locally — and the honest trade-offs vs a cloud-API agent
  • Understand the minimal architecture: local model + tool-calling loop + tools + a guardrail/stop condition
  • Pick a local model that can actually do tool use / agentic work
  • Know which agent frameworks run locally by pointing at a local endpoint (LangGraph, CrewAI, OpenAI Agents SDK)
  • Follow a 'start simple' path from a one-shot tool call to a guarded loop
  • Sandbox and budget-cap the agent so an autonomous loop can't do real damage

Why build a local agent (and when not to)​

An ordinary tool-use agent calls a cloud model. A local agent swaps that cloud call for a model running on your own machine. You give up some capability and inherit some operational burden; in exchange you get four things that are hard to get any other way:

  • Privacy — prompts, tool inputs, and tool outputs never leave the machine. This is the whole reason teams in regulated, sensitive, or air-gapped settings build local agents: the data physically can't go to a third party.
  • Offline — no internet, no API dependency, no provider outage. The agent is files on your disk; it runs on a plane or behind a firewall.
  • No per-call cost — an agent loop can fire dozens of model calls per task. Locally those calls are "free" (you pay in electricity and hardware, not tokens), so you can let it iterate without watching a meter.
  • Full control — pin an exact model version, customize behavior, and run without rate limits or terms-of-service surprises.

The honest trade-offs — be clear-eyed about these before you commit:

  • Capability gap. The hardest part of an agent is the reasoning: planning multi-step work, recovering from a failed tool call, knowing when to stop. A model you can run on a laptop (roughly 1B–14B parameters) is markedly weaker here than a frontier model. Simple, well-scoped loops work well locally; long-horizon, open-ended tasks are where local agents most often go off the rails.
  • You own reliability and safety. No provider is filtering, monitoring, or guard-railing for you. If the agent loops forever, calls the wrong tool, or takes a destructive action, that's on your design. (See the warning below — this is the part people underestimate.)
  • Hardware limits. Bigger, smarter models need more RAM/VRAM than most machines have. You're usually choosing the largest capable model your hardware can run, not the best model that exists.

A durable rule of thumb: start local, escalate when the task demands it. Use a local agent for private/offline/cheap-at-scale work and well-scoped loops; reach for a frontier-API agent when the task genuinely needs the extra reasoning. The architecture below is identical either way — only the endpoint changes — so you can prototype locally and swap models later.

The minimal architecture​

Strip an agent down to its core and there are four parts. Everything else is convenience on top of these.

        ┌─────────────────────────────────────────────┐
        │                                             │
        │   1. LOCAL MODEL  ──►  decides next action  │
        │      (Ollama / LM Studio, tool-capable)     │
        │                  │                          │
        │                  ▼                          │
        │   2. TOOL-CALLING LOOP                       │
        │      parse the model's tool request,         │
        │      run it, feed the result back            │
        │                  │                          │
        │                  ▼                          │
        │   3. TOOLS  ──►  search / read file /        │
        │      run code / call an API (your code)      │
        │                  │                          │
        │                  ▼                          │
        │   4. GUARDRAIL / STOP CONDITION              │
        │      max steps, budget, approval gate,       │
        │      "done" check ──► exit the loop          │
        │                                             │
        └─────────────────────────────────────────────┘
  1. A local model that supports tool calling. The model must be able to emit a structured request to call a tool (a.k.a. function calling), not just chat. Ollama exposes this through its API and through an OpenAI-compatible endpoint at http://localhost:11434/v1, so any framework that speaks the OpenAI format can drive a local model.
  2. The tool-calling loop. The heart of the agent: send the conversation to the model, see if it asked to call a tool, run that tool, append the result, and repeat. When the model answers without requesting a tool, the loop ends.
  3. Tools. Plain functions you expose to the model — search the web, read a file, run a shell command, query a database, hit an API. Each tool has a name, a description, and a typed input schema so the model knows when and how to use it.
  4. A guardrail / stop condition. Non-negotiable for autonomy. At minimum a max-step cap so the loop can't run forever, plus — for anything that writes, deletes, spends, or sends — an approval gate or a sandbox. Without this you don't have an agent, you have an infinite loop with file access.

The loop in step 2 is genuinely small. Here it is in Python pseudocode against a local Ollama endpoint:

A minimal local agent loop (Python pseudocode, points at local Ollama)

from openai import OpenAI

# Point the OpenAI client at your LOCAL Ollama endpoint — nothing leaves the machine
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

tools = [{
  "type": "function",
  "function": {
      "name": "read_file",
      "description": "Read a UTF-8 text file and return its contents",
      "parameters": {
          "type": "object",
          "properties": {"path": {"type": "string"}},
          "required": ["path"],
      },
  },
}]

def run_tool(name, args):
  if name == "read_file":
      # GUARDRAIL: only allow reads inside a sandboxed directory
      return safe_read(args["path"])
  raise ValueError(f"unknown tool: {name}")

messages = [{"role": "user", "content": "Summarize ./notes/today.md"}]

for step in range(8):                       # GUARDRAIL: hard step cap
  resp = client.chat.completions.create(
      model="llama3.1", messages=messages, tools=tools,
  )
  msg = resp.choices[0].message
  messages.append(msg)

  if not msg.tool_calls:                  # STOP: model answered, we're done
      print(msg.content)
      break

  for call in msg.tool_calls:
      result = run_tool(call.function.name, json.loads(call.function.arguments))
      messages.append({
          "role": "tool", "tool_call_id": call.id, "content": str(result),
      })
else:
  print("Stopped: hit the step cap without finishing.")

That is the whole pattern. Frameworks add memory, retries, multi-agent orchestration, tracing, and structured state on top — but every one of them is a more robust version of this loop.

Which local models suit agentic / tool-use work​

Not every open-weight model can drive an agent. The bar is reliable tool calling: the model has to emit well-formed tool requests consistently, pick the right tool, and not hallucinate arguments. Two filters when choosing:

  • It must be a tool-capable model. Ollama tags these — browse the Tools category for the current list rather than assuming. Models commonly cited for solid local tool use include the Qwen and Llama instruction-tuned families; the exact best pick moves quarter to quarter.
  • It must fit your hardware with room for context. Agent loops accumulate long message histories (every tool result is appended), so you need both the weights and a generous context window in memory. A smaller model that comfortably fits and runs fast often beats a larger one that swaps to disk and stalls mid-loop.

The decisive move is not reading benchmarks — it's running a small eval of your task against two or three candidate models. A model that tops a leaderboard can still be unreliable at the specific tools your agent needs. Measure on your own loop.

Frameworks that run locally​

You can hand-roll the loop above, and for a first agent that's a great way to learn. For anything real, a framework gives you retries, memory, multi-agent coordination, and tracing. The key fact: the popular agent frameworks are model-agnostic — they don't care whether the model is in the cloud or on localhost, as long as you point them at the right endpoint.

  • LangGraph — a low-level orchestration framework for stateful agents (durable execution, persistence, human-in-the-loop). Model-agnostic; wire it to a local model via the LangChain Ollama integration with no workarounds. Good when you need explicit control over the agent's state graph.
  • CrewAI — a higher-level framework for orchestrating one or more role-based agents ("crews"). Model-agnostic via LiteLLM; point an agent at a local model with LLM(model="ollama/llama3.1", base_url="http://localhost:11434"). Good when you want to compose multiple cooperating agents quickly.
  • OpenAI Agents SDK — a lightweight multi-agent framework. Despite the name it's provider-agnostic: via its LiteLLM integration you can point it at a local Ollama model instead of an OpenAI one. Good when you want OpenAI's agent ergonomics on a local backend.

Pick one framework and learn it well rather than sampling all three. The concepts (agents, tools, loops, state) transfer; the APIs are details.

Build your first local agent​

A realistic path goes from "no loop at all" to "guarded autonomous loop" in deliberate steps. Don't skip to step 4 — most of the failures people hit with local agents come from giving a weak model too much rope too early.

Guided walkthrough1 of 5
  1. Install Ollama (see Run models locally), then pull a model tagged for tools, e.g. ollama pull llama3.1. Confirm it serves on http://localhost:11434 and that ollama list shows it. No agent yet — just a model you can call.
Watch out
  • A local agent with tools can still take real actions — sandbox it, require approval for destructive steps, and cap its loops/budget.

Check yourself​

Check yourself

0/4
  1. What is the single most important reason teams build agents that run locally rather than against a cloud API?
  2. Which four parts make up the minimal local-agent architecture?
  3. What does it mean that LangGraph, CrewAI, and the OpenAI Agents SDK are 'model-agnostic' for local use?
  4. You're building your first local agent. What should you do BEFORE adding any tool that writes files or runs commands?
Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 6
Key takeaways
  • A local agent is the standard tool-use loop with the cloud model swapped for an open-weight model on your machine — private, offline, and free to iterate.
  • Minimal architecture = local tool-capable model + tool-calling loop + tools + a guardrail/stop condition. The loop itself is tiny.
  • Ollama's OpenAI-compatible endpoint (/v1) supports tool calling, so any OpenAI-format framework can drive a local model.
  • LangGraph, CrewAI, and the OpenAI Agents SDK are model-agnostic — point them at a local endpoint instead of the cloud.
  • Pick a tool-capable model that fits your hardware, then decide with a small eval of your own task — not a leaderboard.
  • Be honest about the capability gap and own the safety: cap loops and budget, sandbox tools, and require approval for anything destructive.
  • Start simple: one clean tool call → bounded read-only loop → guarded destructive tools → (optionally) a framework.

Sources & further reading​