Pular para o conteúdo principal

Why a 35 KB System Prompt Breaks on a Local Model

Intermediário
What you'll learn
  • Do the context-budget arithmetic before you move a prompt, and see why a 35 KB prompt that is invisible on Claude is a quarter of a small local window
  • Know exactly what Ollama does when the prompt is too long — it drops the middle, keeps a few prefix tokens, and only tells the server log
  • Recognise the five behavioural signals of context exhaustion in a local agent before it wastes an hour
  • Restructure a monolithic prompt into single-objective units that fit, and hand state between them on disk
  • Tune the runtime knobs that matter: OLLAMA_CONTEXT_LENGTH, KV-cache quantisation, flash attention, and the parallel-request multiplier

A post that hit the Hacker News front page on September 13–14, 2026 described an experience many people are about to have: a 35 KB system prompt that had been running fine on Claude Opus was pointed at a self-hosted 27B model through Ollama, and within about three minutes the agent was re-reading files it had already read, repeating identical tool calls, and rewriting finished work. Nothing was "wrong" with the model. The prompt plus the session history had simply exceeded a 65K-token window, and the runtime had started throwing parts of the conversation away.

This page is the guide the author wished existed. It is about prompt size versus context budget, which is a different problem from prompt wording across models, covered in Porting Prompts Across Models, and from which local model to pick, covered in Run Models Locally with Ollama.

The arithmetic that nobody does first​

On Claude a 35 KB system prompt is roughly 9,000 tokens, sitting inside a 1,000,000-token window: under 1 %. You never think about it. On a local runtime the same prompt is the same 9,000 tokens, but the window is whatever you configured, and the default is small:

Ollama default context (current main docs)Machine
4K tokensunder 24 GiB VRAM
32K tokens24–48 GiB VRAM
256K tokens48 GiB VRAM or more

At 4K, a 9,000-token system prompt does not fit at all. At 32K it is 28 % of the window before the user says a word. At the 65K the HN author configured it is 14 %, which sounds fine until you add what an agent session actually carries: tool definitions (a few hundred to a few thousand tokens), every file the agent reads, every tool result, and the model's own output. A coding agent that reads three medium files and runs two commands is routinely 20–30K tokens into the conversation. With a 9K fixed cost, the usable working memory is gone in a handful of turns.

The two cloud habits that make this invisible on Claude are absent locally: the window is two orders of magnitude smaller, and there is no built-in compaction. Claude Code summarises the conversation when it approaches the limit (Context Management); a raw Ollama endpoint does not. Some local agent harnesses do their own compaction, but the runtime underneath them does not.

Measure the prompt before you migrate (any OpenAI-compatible local endpoint)

# Count what the model will actually see, from Ollama's own tokenizer.
# prompt_eval_count in the response is the tokenised prompt length.
curl -s http://localhost:11434/api/chat -d '{
"model": "qwen3:32b",
"messages": [{"role":"system","content":"'"$(cat system-prompt.md | sed 's/"/\\"/g')"'"},
             {"role":"user","content":"ping"}],
"stream": false,
"options": {"num_predict": 1}
}' | jq '{prompt_tokens: .prompt_eval_count, cached: .prompt_eval_cached_count}'

Compare that number against the num_ctx you run with, then subtract the tool schema and a realistic transcript. If the remainder is under about half the window, the prompt is too big for the setup, and no amount of wording will fix it.

What Ollama does when the prompt does not fit​

This is the mechanism that turns "slightly too long" into "mysteriously stupid", and it is documented nowhere in prose; it is in the source.

When a request's prompt is longer than the context and truncation is enabled (it is, by default, for /api/chat and /api/generate), the llama.cpp-backed runner in Ollama does the following, in llm/llama_server.go:

  1. Keeps the first num_keep tokens of the prompt. The default num_keep is 4, set "to avoid issues on context shifts".
  2. Computes a target length that leaves roughly half of the remaining context free for generation.
  3. Discards tokens from the middle: the kept prefix stays, the tail stays, the block between them is removed.
  4. Logs truncating input prompt with the before/after sizes on the server. The API response carries no flag, and a client library sees a normal completion.

The project's own test table makes the scale concrete: with a 4,096-token context and the default num_keep, an over-long prompt is cut to 2,050 tokens. Half of the window is deliberately reserved for the reply. So a 9,000-token system prompt on a 4K window arrives at the model as four tokens of its opening, then a hole, then whatever fits at the end of the conversation. The instructions are gone; the last user message survives. That is exactly the "restating objectives, repeating tool calls" behaviour the HN post describes: the model is being asked to act on a conversation whose rules were silently deleted.

Two ways out of the silent version:

  • Send "truncate": false in the request. Ollama then returns HTTP 400 with a message that the prompt is longer than the context currently available, instead of guessing. For an agent harness this is the correct behaviour: fail loudly, then compact or restart.
  • Raise num_keep to the length of your system prompt so the prefix survives a shift. This preserves the rules but still deletes conversation from the middle; it is a band-aid, not a fix.

The chat endpoint has a second, gentler layer above this: when the message list is too long for the limit, Ollama's prompt builder drops older messages first while keeping the system message and the latest turn (see server/prompt.go). You can lose intermediate tool results this way too, which is why an agent may "forget" it already read a file. Watch the server log; with OLLAMA_DEBUG=1 the trimming is visible.

:::note Not just Ollama llama.cpp's server has the same context-shift behaviour under a different flag, vLLM refuses over-long prompts by default, and LM Studio exposes the context length as a per-model setting that is easy to leave at its small default. The pattern to internalise is: know whether your runtime truncates or errors, and prefer the error. :::

The five failure signals​

The author of the HN post kept a list of what context exhaustion looked like from the outside, and it matches what people report about every small-window agent. Treat any of these as "the window is full", not as "the model is bad":

  1. Identical consecutive tool calls. The model no longer has the previous result in view, so it asks again.
  2. Re-reading a file it already read. Same cause; the read result was trimmed.
  3. Restating the objective in its own words mid-task, sometimes with drift. The system prompt has been cut, and the model is reconstructing the task from the tail of the conversation.
  4. Tool-call parsing failures that appear after a run of clean calls. Once the examples and schema in the prompt are gone, small models fall back to their default formatting.
  5. High turn count relative to files actually changed. The ratio is the cheapest metric to track in a harness; when it spikes, stop and restart with fresh context.

If you run through an OpenAI-compatible client, log prompt_eval_count every turn. When it stops growing while the conversation keeps growing, truncation has started.

Restructure: single-objective prompts and state on disk​

A 35 KB prompt is usually a whole product's worth of rules: personas, coding standards, tool etiquette, output formats, edge cases, and a long list of "never do X". On a 1M-token model that works by brute force. On a 32K model it does not, and the fix the HN author landed on is the same one that the strongest commenters argued for independently: split the prompt into single-objective units and run each in its own short session.

Guided walkthrough1 of 6
  1. Go through the 35 KB and tag every paragraph with the job it serves: planning, editing, testing, reviewing, writing the commit. Most monoliths contain five to eight distinct jobs plus a shared core (persona, repo facts, safety rules) that is under 2 KB.

A hand-off file the next single-objective unit reads first

# HANDOFF — written by unit 2 (implement), read by unit 3 (test)
Objective completed: renamed getUser -> fetchUser across src/
Files changed: src/api/user.ts, src/pages/profile.tsx, src/hooks/useUser.ts
Not done: tests still reference getUser (tests/user.test.ts)
Constraints carried forward: no new dependencies; keep TypeScript strict
Next unit should: update tests, run `npm test`, write result to HANDOFF.md

This is also the honest answer to the commenter who said a 35 KB prompt is "confusing and unfocused on any LLM". Frontier models paper over the bloat; local models expose it. Splitting by objective usually improves results on Claude too, and it is the design Claude plus local models assumes when it routes cheap steps locally.

Tune the runtime​

Once the prompt is right-sized, four knobs decide how much window you can actually afford. All are from the Ollama FAQ and docs.

KnobWhat it doesGotcha
OLLAMA_CONTEXT_LENGTH=65536 ollama serve (or num_ctx per request, or PARAMETER num_ctx in a Modelfile)Sets the windowLarger context costs memory. The docs say so bluntly; the KV cache grows linearly with it
OLLAMA_KV_CACHE_TYPE=q8_0Quantises the KV cache; about half the memory of the default f16. q4_0 is about a quarterRequires flash attention; small quality cost, larger with q4_0
OLLAMA_FLASH_ATTENTION=1Enables flash attention; prerequisite for KV quantisation and reduces memory for long contextsNot every backend/model supports it; set it and check the server log
OLLAMA_NUM_PARALLEL (default 1)Concurrent requests per modelRequired memory scales by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Two parallel slots at 64K cost the KV memory of one at 128K

A useful order of operations on a fixed memory budget: pick the largest model you can hold with a small window, then enable flash attention and q8_0 KV cache, then raise the context until the server log shows it spilling to CPU. Split context across GPU and CPU is where tokens per second collapse; several HN commenters reported dense 27B–32B models at FP8 with very long windows on dual-GPU setups, but on a single 128 GB unified-memory box the practical ceiling was the 65K the author used.

:::tip Prefix caching helps, but only for the prefix Ollama reports prompt_eval_cached_count in every response. A stable system prompt at the very start of the conversation is re-used from the KV cache across requests, so its cost is paid once per session, not per turn. Anything after the first change in the conversation is recomputed. Put the fixed material first and the volatile material last, the same rule as Claude's prompt caching (Prompt Caching Economics). :::

Migration checklist​

  • Tokenise the system prompt with the local tokenizer and write the number down.
  • Set num_ctx explicitly. Never rely on the default; it depends on the machine's VRAM.
  • Decide truncate-or-error. For agents, send "truncate": false and handle the 400.
  • Split the prompt by objective; shared core under 2K tokens; each unit 2–4K.
  • Hand off between units through a file; end the session after each unit.
  • Log prompt_eval_count per turn and alert when it plateaus.
  • Enable flash attention and q8_0 KV cache before raising the window.
  • Watch the server log for truncating input prompt during the first week.
Pressione Enter ou Espaço para virar o cartão. Use as setas esquerda e direita para navegar entre os cartões.Termo exibido.
1 / 6

Check yourself

0/5
  1. A 9,000-token system prompt is sent to Ollama with num_ctx 4096 and default settings. What reaches the model?
  2. Which signal most reliably indicates context exhaustion in a local agent?
  3. You want the runtime to fail loudly when a prompt is too long. What do you send?
  4. You set OLLAMA_NUM_PARALLEL=4 and OLLAMA_CONTEXT_LENGTH=32768. What memory does the KV cache need, relative to one request at 32K?
  5. Which change gives the most durable fix for a 35 KB prompt on a 32K-window model?

Sources & further reading​

Next​