Skip to content

Read long web pages

get_web_content returns a page's first 20,000 characters, whatever the agent is looking for. An agent that reads several pages one after another fills its context window with whole pages, most of which has nothing to do with its question, and stays full: every tool result is resent on every later turn.

make_page_tools() returns three tools that keep the page out of the conversation and let in only what the question needs:

  • fetch_page(url, question) fetches the page once, stores it, and returns its outline plus the passages that best match the question, verbatim, each with the character offset it starts at.
  • read_page(handle, offset) reads any part of a stored page, without fetching it again.
  • find_in_page(handle, text) lists the lines of a stored page that contain some text.
import aimu
from aimu.agents import Agent
from aimu.tools import builtin
from aimu.tools.builtin import make_page_tools

agent = Agent(
    aimu.client("ollama:qwen3:8b"),
    "You research questions on the web.",
    tools=[builtin.web_search, *make_page_tools()],
)
agent.run("How long can ochre sea stars survive out of water?")

What enters the context

This is what fetch_page returned for a short field guide, asked question="how long can sea stars survive out of water?", with make_page_tools(top_k=2):

Page page-29b5c273cc20 (https://example.org/tide-pools): 496 characters.

Outline:
- # Field guide to tide pools (offset 1, 94 chars)
- ## Zones (offset 95, 169 chars)
- ## Residents (offset 264, 159 chars)
- ## Visiting (offset 423, 74 chars)

Passages most relevant to the question (2 of 4, in page order):

[offset 1, under "Field guide to tide pools"]
# Field guide to tide pools

Tide pools form where rock holds seawater as the tide goes out.

[offset 264, under "Residents"]
## Residents

Sea anemones close up when exposed to air to keep from drying out.

Ochre sea stars prey on mussels and can survive several hours out of water.

Read more with read_page("page-29b5c273cc20", offset=N); search with find_in_page("page-29b5c273cc20", "text").

Four things to notice:

  • Everything after the header is the page's own text. Nothing was summarized or reworded, so whatever the agent quotes, the page said.
  • The first passage matched on the word "out" alone. The default ranker is BM25, which scores shared words and knows nothing about meaning. It ignores common English words ("the", "is", "of"; the list is aimu.rag.bm25.STOPWORDS), so a passage sharing only those with the question is never shown, but any other shared word counts. Passages appear in page order, not score order, so the strong match here is the second one.
  • The outline and offsets say what was left out. The Zones section did not come back, and the outline shows where it is and how long it is.
  • The handle names this copy of the page. It is a digest of the URL and the text, so fetching the same page again returns the same handle, and a page that changed in between gets a new one rather than overwriting the copy an earlier answer came from.

Without a question, or when no passage shares a word with it (common words aside), fetch_page returns the opening of the page instead (6,000 characters by default) and says which of the two happened.

Reading past the passages

Ranking matches words, so a passage that answers the question in different words can be missed. The fetch_page description tells the model to look further before concluding a page does not say something. It has two ways to do that:

read_page("page-29b5c273cc20", offset=95, max_chars=400)   # the Zones section, from the outline
find_in_page("page-29b5c273cc20", "low tide")              # every line that mentions it
1 lines match "low tide"
- offset 460: Visit within an hour of low tide, and step only on bare rock.

Each offset is where the match itself starts, so read_page at that offset begins at the match even on a page that is one long line. find_in_page matches literally and ignores case; it does not take regular expressions.

Small models and navigation

Whether an agent uses read_page and find_in_page well depends on the model. On the BrowseComp-Plus benchmark (arXiv 2508.06600), giving gpt-4.1 a full-document tool alongside short previews raised its accuracy from 35% to 44%, while Qwen3-32B barely called the tool (10% to 12%). The ranked passages are the part that does not depend on the model choosing to look further.

Keeping pages between runs

By default pages live in memory for as long as the tools do. Pass directory= to keep them as plain files instead: <handle>.md holds the text exactly, and <handle>.json holds the URL and fetch time.

tools = make_page_tools(directory=aimu.paths.output / "pages")

Nothing is ever deleted from that directory; prune it yourself.

Ranking with embeddings

ranker= replaces BM25 with any function that takes the question and the passages and returns one score per passage, higher meaning more relevant and 0 meaning no match. An embedding model ranks paraphrases that share no words with the question, which BM25 cannot:

import math

import aimu
from aimu.tools.builtin import make_page_tools

embedder = aimu.embedding_client("ollama:nomic-embed-text")


def cosine(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b)) / (math.hypot(*a) * math.hypot(*b))


def embedding_ranker(question: str, passages: list[str]) -> list[float]:
    query = embedder.embed(question, input_type="query")
    vectors = embedder.embed(passages, input_type="document")
    return [max(0.0, cosine(query, vector)) for vector in vectors]


tools = make_page_tools(ranker=embedding_ranker)

Clamping at 0 keeps "no match" meaning what fetch_page expects. An embedding ranker almost never scores a passage at exactly 0, so with one fetch_page nearly always returns top_k passages rather than falling back to the opening.

Tuning

Argument Default What it controls
top_k 5 The most passages fetch_page returns
chunk_chars 1200 The longest passage. Passages never span two sections.
opening_chars 6000 How much fetch_page returns with no question, or when nothing matches

See also