Build a Local AI Agent: Ollama, a Small Model and MCP Tools

What you'll build

By the end of this guide you will have a working AI agent that runs entirely on your own laptop: a small open-weight model served by Ollama, a clean agent loop that lets the model propose and execute tool calls, a set of tools wired in through the Model Context Protocol (MCP), short- and long-term memory, and the guardrails that keep the whole thing from looping forever or shelling out something it shouldn't. You will also have a routing rule for the moments when a small local model is the wrong tool and the job belongs in the cloud.

The audience here is deliberately two-sided. If you are a Chennai or Bengaluru indie hacker on a 16GB MacBook who wants to stop burning credits on every experiment, this gets you an agent that costs nothing per token. If you are on a London or Manchester team with data-residency obligations, this keeps customer data on a machine you control rather than shipping it to a third-party endpoint. The architecture is the same; only the routing rule at the end differs.

One framing note before we start. Models churn fast, so everywhere a specific model is named, treat the name as a snapshot. As of June 2026 the picks below are sound; six months from now you should re-run the evaluation in the last section rather than trust a name you read in an article.

Prerequisites

You need a machine that can hold a small model in memory and decode it at a usable speed. In practice that means one of two setups: an Apple Silicon MacBook with at least 16GB of unified memory, or a Linux or Windows box with a discrete GPU carrying 8GB or more of VRAM. Either comfortably runs a 3B to 8B model quantised to GGUF Q4_K_M. A 16GB machine leaves enough headroom for the model, your tools and the operating system at the same time; below that you should drop to a 3B model or trim the context window.

Install Ollama from ollama.com and confirm it is serving. As of June 2026, Ollama v0.14.0 and later expose two HTTP surfaces on localhost:11434: its own /api/chat endpoint with native tool-calling, and an Anthropic-compatible Messages API at /v1/messages. The second one matters later, because it lets you point tools that already speak Anthropic's wire format at a local model with nothing more than an environment variable.

You will also want Python 3.11 or newer and the ollama client library. That is the entire dependency list for the core agent; MCP tooling adds one more package when we get to it.

# Confirm Ollama is up
ollama --version

# Pull a small, reliable tool-use model (GGUF Q4_K_M by default)
ollama pull qwen2.5:7b-instruct

# Smoke test it
ollama run qwen2.5:7b-instruct "Reply with the single word: ready"

# Python client for the agent loop
pip install ollama

Choose a small tool-use model

Not every small model follows tool calls well. A model can be a fine chatbot and still mangle structured tool arguments, which is the only thing that matters for an agent. The job here is to pick a model that reliably emits a well-formed call when it should, and plain text when it shouldn't.

As of June 2026, the most reliable small tool-follower we have measured is Qwen2.5 7B Instruct. It rarely hallucinates tool names and keeps its arguments inside the schema. Llama 3.1 8B Instruct and Mistral 7B Instruct v0.3 are close behind and worth keeping as fallbacks. If your agent mostly reads and edits code, Qwen3-coder is the stronger pick. The sizing table below assumes GGUF Q4_K_M, which fits 3B to 8B models on an 8GB VRAM laptop GPU while keeping roughly 92% of full-precision quality — the right trade-off for almost every local agent.

Model	Params	GGUF Q4_K_M size	Min RAM / VRAM	Tool-use quality
Qwen2.5 7B Instruct	7B	~4.7 GB	8 GB	Excellent — very reliable tool-following
Llama 3.1 8B Instruct	8B	~4.9 GB	8 GB	Strong general-purpose tool caller
Mistral 7B Instruct v0.3	7B	~4.4 GB	8 GB	Solid; good on short, well-scoped tools
Qwen3-coder	~7B class	~4.7 GB	8 GB	Best when the agent edits code
Qwen2.5 3B Instruct	3B	~2.0 GB	≤ 8 GB / low-RAM	Good for simple two- to three-tool agents

The practical reading for a 16GB MacBook: start at Qwen2.5 7B Instruct. If you are sharing memory with a heavy IDE, a browser and a database, step down to the 3B and shorten your prompts before you reach for anything bigger. Bigger is not the win at this scale — a 7B model that follows tools cleanly beats a 13B model that occasionally invents a function name.

The agent loop explained

An agent is not magic; it is a loop. The model never executes anything itself. It only ever produces text or a request to call a tool, and your code decides what to do with that. The loop has six moving parts and it pays to hold all of them in your head at once:

1. System prompt — tells the model what it is, what it may do, and how to behave when a tool fails.
2. Tool schema — a machine-readable description of each tool: name, what it does, and the shape of its arguments.
3. Model proposes a tool call — given the conversation and the schema, the model emits a structured request to run a named tool with specific arguments.
4. Execute the tool — your code validates the arguments, runs the function, and captures the result or the error.
5. Feed the result back — the tool output goes into the conversation as a tool message so the model can read what happened.
6. Repeat — until the model returns a plain text answer with no tool call, or a guardrail stops the loop.

The code below is a complete, runnable loop using the ollama client. It defines one tool — a calculator stand-in for whatever real capability you need — describes it with a schema, and dispatches calls the model proposes. Note the dispatch table: the model can only ever invoke a function you have explicitly registered, which is your first and cheapest guardrail.

import json
import ollama

MODEL = "qwen2.5:7b-instruct"

# 1. The actual tool implementations (your real capabilities)
def get_word_count(text: str) -> dict:
    return {"words": len(text.split())}

def add_numbers(a: float, b: float) -> dict:
    return {"sum": a + b}

DISPATCH = {
    "get_word_count": get_word_count,
    "add_numbers": add_numbers,
}

# 2. Tool schemas the model sees
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_word_count",
            "description": "Count the words in a piece of text.",
            "parameters": {
                "type": "object",
                "properties": {"text": {"type": "string"}},
                "required": ["text"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "add_numbers",
            "description": "Add two numbers and return the sum.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {"type": "number"},
                    "b": {"type": "number"},
                },
                "required": ["a", "b"],
            },
        },
    },
]

SYSTEM = (
    "You are a careful local assistant. Use a tool when it helps. "
    "If a tool returns an error, explain it plainly and do not retry blindly."
)

def run_agent(user_msg: str, max_steps: int = 6) -> str:
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": user_msg},
    ]
    for step in range(max_steps):
        resp = ollama.chat(model=MODEL, messages=messages, tools=TOOLS)
        msg = resp["message"]
        calls = msg.get("tool_calls") or []
        if not calls:
            return msg.get("content", "")
        messages.append(msg)
        for call in calls:
            name = call["function"]["name"]
            args = call["function"]["arguments"]
            if isinstance(args, str):
                args = json.loads(args)
            fn = DISPATCH.get(name)
            if fn is None:
                result = {"error": f"unknown tool: {name}"}
            else:
                try:
                    result = fn(**args)
                except Exception as exc:               # validation / runtime guard
                    result = {"error": str(exc)}
            messages.append({
                "role": "tool",
                "name": name,
                "content": json.dumps(result),
            })
    return "Stopped: step budget exhausted."

if __name__ == "__main__":
    print(run_agent("How many words are in 'the quick brown fox'? Then add 2 and 3."))

That is the whole idea. Everything else in this guide — MCP, memory, guardrails — hangs off this loop without changing its shape. Because tool calls run in-process and the model is local, there is no network round-trip in the hot path: latency is dominated by model decode speed and your machine's memory bandwidth, not by HTTP. On a 16GB MacBook a 7B model at Q4_K_M will feel responsive for a handful of tools.

Wiring tools via MCP

Hard-coding tools into your loop is fine for a prototype, but it does not scale across projects. MCP fixes that. It standardises how tools are described and invoked, so the same tool server works across many clients — your local agent today, a different host tomorrow. Write a filesystem tool once as an MCP server and it is reusable everywhere instead of copy-pasted into every agent you build.

The pattern is a host that connects your model to one or more MCP servers. Each server exposes tools over a small protocol; the host fetches the tool list, presents it to the model in the same schema shape you saw above, and routes calls back to the right server. The snippet below defines a minimal MCP server with two tools using the Python SDK, alongside the Ollama-as-Anthropic setup that lets an Anthropic-format host talk to your local model. For a full walkthrough of building the server side, see our guide on how to build your first MCP server with FastMCP in 12 steps.

# server.py — a minimal MCP server exposing two tools
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("local-tools")

@mcp.tool()
def list_dir(path: str) -> list[str]:
    """List the entries in a directory on this machine."""
    import os
    return os.listdir(path)

@mcp.tool()
def read_text(path: str, max_chars: int = 4000) -> str:
    """Read a text file, truncated to max_chars."""
    with open(path, "r", encoding="utf-8") as f:
        return f.read(max_chars)

if __name__ == "__main__":
    mcp.run()  # speaks MCP over stdio by default

# ---------------------------------------------------------------
# Point an Anthropic-format host at your LOCAL Ollama model.
# As of June 2026, Ollama v0.14.0+ serves an Anthropic Messages
# API at http://localhost:11434/v1/messages.
#
#   export ANTHROPIC_BASE_URL=http://localhost:11434
#   export ANTHROPIC_AUTH_TOKEN=ollama     # any non-empty value
#
# Now a tool/harness that speaks the Anthropic wire format will
# call your local model instead of a remote endpoint — no data
# leaves the machine.

If you would rather not run a host yourself, a GUI option exists. Per LM Studio's release notes, LM Studio 0.4.0 added built-in MCP server support with permission-based access control, so you can attach the same tool server to a desktop app and approve each tool's access explicitly. Whichever host you choose, the discipline of writing good tool descriptions and tight argument schemas is what makes a small model behave — covered in depth in designing tools for AI agents: schemas, errors and retries.

Adding memory

The loop above forgets everything the moment it returns. Real agents need two kinds of memory, and they solve different problems. Short-term memory is a scratchpad: the running conversation, intermediate tool results, and a running summary of what the agent has done this session. It lives in the message list you already maintain. The trick at small context sizes is to summarise aggressively — when the message list grows past a budget, fold the oldest turns into a one-paragraph summary and drop the raw messages, so you never blow the context window a 7B model can hold.

Long-term memory is a store that survives across sessions: facts the user told you, decisions you made, documents you indexed. In practice this is a small vector store or even a plain key-value file, queried at the start of a task and written to at the end. The pattern below shows a minimal long-term store you can grow into something more capable; for the production-grade treatment of both layers, read agent memory management patterns.

import json, os, time

class LongTermMemory:
    """A tiny append-then-recall store. Swap the backend for a
    vector DB when keyword recall stops being enough."""

    def __init__(self, path="memory.jsonl"):
        self.path = path

    def remember(self, key: str, value: str) -> None:
        rec = {"t": time.time(), "key": key, "value": value}
        with open(self.path, "a", encoding="utf-8") as f:
            f.write(json.dumps(rec) + "\n")

    def recall(self, query: str, k: int = 5) -> list[str]:
        if not os.path.exists(self.path):
            return []
        hits = []
        with open(self.path, encoding="utf-8") as f:
            for line in f:
                rec = json.loads(line)
                if query.lower() in (rec["key"] + rec["value"]).lower():
                    hits.append(rec["value"])
        return hits[-k:]

# At task start: prepend recalled facts to the system context.
# At task end:   write durable facts back with remember().

The design decision that matters: short-term memory should be cheap to throw away and long-term memory should be expensive to write. Most things the agent learns mid-task are noise that belongs in the scratchpad and dies at the end of the session. Only durable, user-confirmed facts earn a place in the long-term store. Get that boundary wrong and your agent either forgets what it should remember or drowns in stale context.

Guardrails that keep it safe

A local agent with tools is a program that writes part of its own control flow at runtime. Without limits it will, sooner or later, loop forever, hang on a slow tool, or pass garbage arguments into a function. Guardrails are not optional polish; they are the difference between a demo and something you would leave running unattended. Five matter most:

• Max steps — a hard cap on tool-call iterations per task. The loop in this guide already enforces max_steps. Pick a number a correct task never needs and treat hitting it as a failure, not a stall.
• Timeouts — every tool call gets a wall-clock limit. A network or filesystem tool that hangs should be killed and reported back to the model as an error, not left to block the loop.
• Retries with backoff — transient failures get a bounded number of retries with increasing delay. Bounded is the key word: infinite retry is just a slower infinite loop.
• Argument validation — validate every tool argument against its schema before you execute. A small model will occasionally send the wrong type or a missing field; catch it, return a clear error to the model, and let it correct itself.
• Allow-lists — constrain what tools can touch. A filesystem tool sees one directory; a shell tool, if you must have one, runs from a fixed allow-list of commands.

The principle underneath all five: when something goes wrong, fail back to the model with a structured error rather than crashing the process. A good agent loop treats a malformed tool call the same way it treats any other tool result — it feeds the error back and gives the model a chance to recover. The deeper catalogue of these patterns lives in our agent design patterns guide, which is worth reading once you have the basic loop working.

Evaluating your agent

You cannot improve what you do not measure, and "it felt right in the demo" is not measurement. Before you trust a local agent with anything real, build a small golden task set — ten to thirty tasks with known-good outcomes that exercise each tool and a few multi-step combinations. Run the agent against it every time you change the model, the prompt, or a tool. Two things are worth scoring on each run:

• Outcome — did the agent reach the correct final answer or state? This is the bottom line and the easiest to automate.
• Trajectory — did it get there sensibly? Count the tool calls, check it did not loop, and confirm it used the right tools in a reasonable order. A run that stumbles to the right answer through six wasted calls is a regression waiting to happen.

This is exactly the discipline that lets you swap models without fear. When Qwen2.5 is replaced by something newer next quarter, you do not guess whether it is better — you run the golden set and read the numbers. Keep the task set in version control next to the agent so a regression shows up in review, not in production.

When to escalate from local to cloud

A small local model is the right default for a large share of agent work: privacy-sensitive tasks, high-volume routine automation, anything where the data should not leave the machine. But it is not the right tool for everything, and pretending otherwise produces a slow, frustrating agent. The honest move is an explicit routing rule that sends the hard cases to a frontier model in the cloud and keeps everything else local.

For the dual-market reader this rule reads differently at each end. A Chennai or Bengaluru indie hacker escalates mainly to save time on genuinely hard reasoning while keeping experiment cost near zero. A London or Manchester team with data-residency obligations keeps customer data strictly local and only escalates on tasks where no regulated data is involved, or where a cloud region inside the relevant jurisdiction is approved. The table makes the boundary concrete.

Signal	Stay local (small model)	Escalate to cloud (frontier model)
Context length	Fits the small model's window (a few thousand to ~32k tokens)	Long context — a Claude Code harness against local Ollama wants ≥ 64k
Reasoning depth	Single tool or short, well-scoped chains	Deep multi-step reasoning your evals show the small model failing
Data sensitivity	Privacy-sensitive or regulated data must stay on-device	No restricted data, or an approved in-jurisdiction cloud region
Volume / cost	High-volume routine work — local is free per token	Low-volume, high-value tasks where quality justifies the spend
Quality bar	Golden-set scores meet your threshold locally	Golden set shows the small model missing the bar

Encode this as a function the agent consults before each task, not as a judgement you make by feel. A routing rule you can read and test is a routing rule you can trust. The broader landscape of frontier agent frameworks you might escalate into is covered in our news piece on the agent SDK wars between OpenAI, Google and Anthropic.

Conclusion and next steps

You now have the full shape of a local agent: a small tool-use model served by Ollama, a six-part loop that lets the model drive tools without ever executing anything itself, MCP for portable tools, two layers of memory, five guardrails, an evaluation habit, and an explicit rule for when to reach for the cloud. None of it depends on a paid endpoint, and all of it runs on a 16GB MacBook or an 8GB GPU.

The next steps are concrete. Replace the toy calculator tools with one real capability your work actually needs. Write your golden task set before you add the second tool, not after. Wrap your filesystem access in an MCP server with a tight allow-list. Then, and only then, start swapping models underneath to find the smallest one that clears your quality bar — because the smallest model that passes is the one you want running locally all day. Keep the model names in this guide as a June 2026 snapshot and re-test as the field moves, which it will.