import aimu
MODEL = "ollama:qwen3:8b"Managing Context
A conversation is self.messages — a plain list[dict], OpenAI format (see notebook 02) — and it only grows. Left alone, a long-running agent (notebook 07) eventually produces a request that no longer fits the model’s context window. aimu.context holds the tools for that: count_tokens, the estimate every budget is measured in, and aimu.context.compaction, which bundles the strategies (mask, trim, summarize) and the ready-made compactions an Agent(compaction=...) applies to every request it sends, leaving the conversation itself whole. Nothing here is a policy AIMU picks for you: it’s the same “plain data, plain functions” shape as aimu.rag (notebook 13), so what gets left out and why is always something you can print, and the request that was actually sent is on client.last_request.
What this notebook covers: - count_tokens, and its honesty caveat: an estimate, not a measurement - trim_messages, and the invariant that justifies it existing (never orphan a tool message from the assistant turn carrying its tool_calls) - mask_messages: replace old content in place with a note naming the message, instead of dropping it - keep_last counts messages, not exchanges - summarize_messages: replace old turns with a generated summary instead of dropping them - Agent(compaction=): the ready-made compaction.mask / compaction.trim / compaction.chain, applied to what every request sends while the history stays whole, the get_message tool that comes with them, and how a request that leaves something out announces it - ContextOverflowError: the portable catch-compact-retry signal for when it’s already too late
This notebook runs a local Ollama model (
qwen3:8b) in sections D, E, and F; A through C are pure functions and need no backend. Generated summary text and token usage vary by run; the numbers from the pure functions (count_tokens,trim_messages,mask_messages) do not, for a fixed input.
Setup
A - Count tokens: an honest estimate
count_tokens estimates a message list’s size. The default counter is len(text) // 4 over each message JSON-serialized, with AIMU’s own inert bookkeeping keys (timestamp, thinking, provenance) stripped first — a provider never sees those, but the structural overhead of role/tool_calls/JSON punctuation does ride along in the real request:
messages = [{"role": "user", "content": "hello there"}]
print(aimu.count_tokens(messages))10
This is an estimate, not a measurement — typically wrong by 20-30% for any specific model’s real tokenizer. The only exact count AIMU can report is after the fact, via client.last_usage following a real call (see notebook 27, section E). Pass counter= with a real tokenizer when accuracy matters more than a zero-dependency default:
aimu.count_tokens(messages, counter=lambda text: len(enc.encode(text))) # enc: any tiktoken-like encoderB - Trim to a budget: trim_messages
trim_messages drops the oldest messages until the conversation fits max_tokens (per count_tokens), and returns a new list — the input is never mutated:
budget_messages = [{"role": "system", "content": "You are a assistant."}]
for i in range(6):
budget_messages.append({"role": "user", "content": f"This is user message number {i}, with some extra padding."})
budget_messages.append({"role": "assistant", "content": f"This is assistant reply number {i}, also padded out."})
print(aimu.count_tokens(budget_messages))
trimmed = aimu.trim_messages(budget_messages, max_tokens=80, keep_last=2)
print(aimu.count_tokens(trimmed))
print([m["role"] for m in trimmed])274
78
['system', 'user', 'user', 'assistant']
Three things survive any trim: the system message (keep_system=True), the first message after it (keep_first=1, here user 0), and the last keep_last messages (here user 5 and assistant 5). Everything between is droppable, oldest first. The first message is protected by default because in an agent run it is the task: dropping it leaves the agent working without knowing what it was asked. keep_first=0 restores plain oldest-first dropping:
print([m["role"] for m in aimu.trim_messages(budget_messages, max_tokens=80, keep_first=0, keep_last=2)])['system', 'assistant', 'user', 'assistant']
With user 0 droppable, the budget is met one message earlier, so assistant 4 survives instead. Section C explains why keep_last counts messages rather than exchanges.
The invariant that justifies this over a hand-written messages[-n:] slice: trimming never orphans a tool message from the assistant message carrying the tool_calls it’s answering. Every provider rejects that shape, and it’s exactly what a naive slice produces. trim_messages treats an assistant-with-tool_calls message and every tool message answering it as one indivisible unit — dropped together, kept together, never split. On a real two-turn tool-using conversation (built the same way notebook 07 does — note the system prompt lives on the client, not the agent, so both run() calls accumulate onto one history rather than resetting; see notebook 25’s gotcha for why):
from aimu.agents import Agent
from aimu.tools import tool
@tool
def add(a: int, b: int) -> int:
"""Add two integers."""
return a + b
tool_client = aimu.client(MODEL, system="You are a helpful assistant. Use the add tool for arithmetic.")
tool_agent = Agent(tool_client, tools=[add])
tool_agent.run("What is 12 + 30? Use the add tool.")
tool_agent.run("What is 100 + 1? Use the add tool.")
print([m["role"] for m in tool_client.messages])
tool_trimmed = aimu.trim_messages(tool_client.messages, max_tokens=150, keep_last=1)
print([m["role"] for m in tool_trimmed])['system', 'user', 'assistant', 'tool', 'assistant', 'user', 'assistant', 'tool', 'assistant']
['system', 'user', 'assistant', 'user', 'assistant', 'tool', 'assistant']
Reading the trimmed list against the original: the first exchange’s user question is pinned by keep_first=1, so the first thing dropped is the assistant-with-tool_calls/tool pair answering it, removed together (the whole group, never split). The first exchange’s own final assistant reply (no tool_calls, so nothing to orphan) survived on its own. The second exchange’s assistant-with-tool_calls/tool pair was kept intact even though only its trailing assistant message was technically inside the protected keep_last=1 tail — the boundary was pushed outward to the group’s edge rather than splitting it. (Which messages are big enough to need dropping depends on how much the model reasoned before each reply — a thinking model’s reasoning text inflates some assistant messages far more than others — so a re-run can drop a different combination of groups than shown here; what never changes is that a tool message is never separated from the assistant message carrying the tool_calls it answers.)
Mask instead of dropping: mask_messages
Trimming drops a tool call together with its result, so the model loses the fact that it read anything. aimu.context.compaction.mask_messages keeps every message where it is and replaces old content with a note naming the message, cheapest content first (tool results, then images, then assistant text, then user text):
from aimu.context.compaction import MASK_ORDER, mask_messages
page = "The release is planned for March. " * 60
research = [
{"role": "system", "content": "You are a researcher."},
{"role": "user", "content": "When is the release?"},
{"role": "assistant", "content": "", "tool_calls": [
{"id": "c1", "type": "function", "function": {"name": "fetch", "arguments": {"url": "https://example.com"}}},
]},
{"role": "tool", "tool_call_id": "c1", "name": "fetch", "content": page},
{"role": "assistant", "content": "It is planned for March."},
{"role": "user", "content": "Which year?"},
]
print(aimu.count_tokens(research))
masked = mask_messages(research, max_tokens=200)
for index, m in enumerate(masked):
print(index, m["role"], "|", (m.get("content") or "")[:96])
print(aimu.count_tokens(masked), MASK_ORDER)621
0 system | You are a researcher.
1 user | When is the release?
2 assistant |
3 tool | [Message 3, the result of fetch, is not shown here, to save context. get_message(3) returns it i
4 assistant | It is planned for March.
5 user | Which year?
137 ('tool', 'image', 'assistant', 'user')
Only message 3 changed, and only its content: the call that fetched the page is still there with its URL, so the model knows it read the page, and the note names get_message(3), a tool every agent with a compaction is given (section E). No message was removed or added, so the list has the history’s own shape. A note still costs tokens, so masking slows growth without bounding it, which is why trimming stays, as the backstop behind it.
C - keep_last counts messages, not exchanges
A “turn” is ambiguous: a plain back-and-forth is 2 messages (user + assistant), but an agentic turn can be 5+ (user, assistant-with-tool_calls, one or more tool results, final assistant answer). keep_last=2 on section B’s tool-using conversation protects only the last 2 messages — here, tool and the final assistant — not the whole last exchange:
print([m["role"] for m in aimu.trim_messages(tool_client.messages, max_tokens=10_000, keep_last=2)])['system', 'user', 'assistant', 'tool', 'assistant', 'user', 'assistant', 'tool', 'assistant']
Nothing dropped here — the whole conversation already fits comfortably under 10,000 tokens, so trim_messages returns everything unchanged (a copy). Pass a larger keep_last if you want whole exchanges protected on a tool-using conversation; system messages (keep_system=True, the default) are always kept regardless of budget or keep_last.
D - Summarize instead of dropping: summarize_messages
trim_messages discards the oldest turns outright. summarize_messages replaces them with one LLM-generated summary instead, so the conversation keeps some memory of what came before rather than none:
summary_client = aimu.client(MODEL)
plain_messages = [
{"role": "user", "content": "Help me plan a weekend in Lisbon."},
{"role": "assistant", "content": "Happy to. What do you enjoy most when you travel?"},
{"role": "user", "content": "Tile art and seafood, and I'd rather walk than take taxis."},
{"role": "assistant", "content": "Noted: tiles, seafood, and a walkable route."},
{"role": "user", "content": "Which museum should I start with?"},
{"role": "assistant", "content": "The National Tile Museum."},
]
summarized = aimu.summarize_messages(summary_client, plain_messages, keep_last=2)
for m in summarized:
print(m["role"], ":", m["content"])system : Summary of earlier conversation:
The user requested help planning a Lisbon weekend, emphasizing tile art, seafood, and a preference for walking over taxis. The assistant confirmed these priorities, noting the focus on tiles, seafood, and walkable routes. Open questions include specific areas, duration, or budget details not yet discussed.
user : Help me plan a weekend in Lisbon.
user : Which museum should I start with?
assistant : The National Tile Museum.
The "Summary of earlier conversation:\n" prefix is a fixed template; only the summary itself is generated, and it will read differently on your machine. The rest is the original messages kept verbatim: the first one (keep_first=1, the default, which protects an agent’s task from being paraphrased away) and the last two (keep_last=2). The three messages between them were summarized. The summary sits before the kept first message, though it describes what came after it, because several local chat templates reject a system message anywhere but the start of the conversation.
The summary knows the trip is to Lisbon although “Lisbon” appears only in the kept first message. The summarizer is shown that message as background, labeled “kept as written; do not summarize”, because the instruction asks it to keep what a continuation will need and the task is what decides that. It restated the task anyway (“requested help planning a Lisbon weekend”); models tend to, despite the label. keep_first=0 summarizes from the first message with no background block. client is anything with a generate(prompt: str) -> str method (a plain BaseModelClient, or agent.as_model_client()), passed in rather than constructed internally so this function stays free of any provider dependency. It applies the same group-protection invariant as trim_messages when carving out the kept messages. When nothing lies between the kept first and last messages, no summarization call is made and the input is returned unchanged (a copy).
E - Automate it: Agent(compaction=)
The functions above are things you call by hand, and calling one by hand rewrites the list you assign it back to. Agent(compaction=...) (and the per-run agent.run(compaction=...) override) is different in kind: it takes a compaction — aimu.context.Compaction, any list[dict] -> list[dict] — and applies it to every request the agent sends, at the moment each request is built. The model is sent what it returns. The conversation keeps everything.
aimu.context.compaction ships them ready-made, the way aimu.tools.builtin ships tool groups: compaction.trim(max_tokens, **options) and compaction.mask(max_tokens, **options) bind a strategy’s settings, and compaction.chain(...) runs several in order. A trim keeps this example’s output easy to read:
import logging
logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(message)s")
from aimu.context import compaction
from aimu.events import ContextCompacted
compaction_seen: list = []
compaction_client = aimu.client(MODEL, system="You are terse. Reply in one short sentence.")
compaction_agent = Agent(
compaction_client,
events=compaction_seen.append,
compaction=compaction.trim(60, keep_last=2),
)
compaction_agent.run("Tell me a fact about the moon.")
compaction_agent.run("Now tell me a fact about the sun.")
compaction_agent.run("Now tell me a fact about Mars.")
compacted = [e for e in compaction_seen if isinstance(e, ContextCompacted)]
print("ContextCompacted events:", len(compacted))
for e in compacted:
print("left out:", [m["role"] for m in e.dropped], "| history:", e.before_tokens, "| sent:", e.after_tokens)
print("history:", [m["role"] for m in compaction_client.messages])
print("last request sent:", [m["role"] for m in compaction_client.last_request["messages"]])WARNING Compacted the request for agent 'agent-d457f0': 2 more message(s) left out (~140 -> ~83 tokens sent, AIMU's own estimate). The history keeps them.
ContextCompacted events: 1
left out: ['assistant', 'user'] | history: 140 | sent: 83
history: ['system', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']
last request sent: ['system', 'user', 'assistant', 'user']
The last two lines are the whole idea. The history holds all three exchanges, and the request for the Mars question sent four messages: the system prompt, the moon question (pinned by keep_first=1), the sun answer, and the Mars question itself (the last two, keep_last=2). What was left out, the moon answer and the sun question, is still in compaction_client.messages, so a later request under a looser compaction (or none) would send it again. client.last_request is the payload the provider actually received, which is where to look when you want to know what the model was working from.
For a long-running agent, the shape to start from is masking first and trimming as the backstop: compaction.chain(compaction.mask(6000), compaction.trim(6000)). The mask keeps the record of what happened (section B), and the trim only ever drops messages the mask has already reduced to notes. Put a mask first in any chain, since its notes name positions in the full history. Any run with a compaction also carries a get_message tool the agent did not list, which returns a message from its own history in full, in windows for a long one: it is what a mask’s notes name, so the model can read back what it needs.
What a request leaves out is never silent. When a request leaves out a message the previous one sent, two things fire together: a ContextCompacted event for a sink attached via events= (see notebook 27), carrying the messages left out, and an unconditional WARNING log, so a caller with no sink still learns the model is working from less than the conversation holds. The agent id, the token counts, and how many messages go all vary by run, since they depend on how long the model’s sampled replies were. Note the pinned moon question: it is sent without its answer, followed directly by a later answer. That is the price of protecting the first message in a multi-turn chat; in a single-task agent run the pinned message is the task, which is exactly what should survive.
Only a change is announced. Once a conversation is over its budget, every request is compacted; a request that leaves out exactly what the previous one did says nothing, or every request of a long conversation would warn and the warning would stop meaning anything. The agent remembers what its last request left out across runs. A request that sends something the previous one left out (because the compaction changed, say) reports it in the event’s restored list, at INFO rather than as a warning, since nothing was lost. A compaction that returns the conversation unchanged announces nothing. ContextCompacted’s before_tokens/after_tokens are AIMU’s own estimate of the history and of what was sent, not a measurement of whatever the callable itself counted.
If the compaction callable raises, the run raises — the same “failures are apparent” rule as everywhere else in AIMU; a compaction that can’t be trusted to run should stop the turn, not be silently skipped while the caller believes their context is being managed.
F - When it’s already too late: ContextOverflowError
Compaction is preventive. ContextOverflowError (from aimu.models import ContextOverflowError) is what a request that already doesn’t fit raises — the input-side counterpart of TruncatedTurnError (aimu.agents), which reports an output that ran out of room instead. Coverage differs by backend, because the failure itself looks different per backend:
| Provider | How it’s detected |
|---|---|
ollama |
native API rejects the request after trimming past the required user turn |
anthropic |
a 400 whose message names the prompt as too long, or a 413 |
openai, gemini, local OpenAI-compat servers that set it |
400 with code=context_length_exceeded |
hf, llamacpp |
in-process pre-flight token count against the model’s own known window, before the call is even attempted |
The recovery recipe — catch, then retry with a compaction:
from aimu.models import ContextOverflowError
try:
agent.run(task)
except ContextOverflowError:
agent.run(task, compaction=compaction.trim(8000))The retry sends a trimmed request and leaves the history whole. To shrink the stored history itself instead, assign the trimmed list back (agent.model_client.messages = aimu.trim_messages(...)), which is a deliberate, permanent edit that you made and that nothing needs to announce.
This cell is deliberately not executed for pasted output: triggering the real error live needs a request that overflows a specific, small context window in the exact way each backend’s own overflow detection expects (Ollama’s native path only raises when its server-side trim reaches past the required user turn), which is backend- and timing-dependent rather than something a short demo prompt reliably reproduces. Raising the model’s own context window (generate_kwargs={"context_length": N} on Ollama; see docs/how-to/set-context-length.md) is the other lever, when the model actually supports a larger one.
Clean up
del tool_client, tool_agent, summary_client, compaction_client, compaction_agent
print("Cleaned up.")