Fetch HTML and submit web forms¶

The builtin.web tools include get_web_content (Markdown) and web_search, but sometimes an agent needs to see a page's raw markup, discover its forms, and submit them (a site search, a login, a data entry). Two additions cover this:

get_webpage_html(url) — a stateless tool returning the page's raw HTML.
make_web_tools() — a factory returning find_forms, submit_form, and submit_json, which share a requests.Session so cookies (and thus logins and CSRF tokens) persist across calls.

Server-rendered HTML only

These fetch HTML over requests; they do not execute JavaScript. Pages built client-side (SPAs) return their pre-render markup, and anti-bot-protected sites (Cloudflare, captchas) will be blocked. A headless-browser backend is a possible future addition.

Read raw HTML¶

get_webpage_html sits in builtin.web alongside get_webpage. Use get_webpage when you want clean article text; use get_webpage_html when you need the markup itself (links, attributes, form structure). Long pages are truncated to stay within the model's context window.

import aimu
from aimu.tools import builtin

agent = aimu.agents.Agent(aimu.client("ollama:qwen3:8b"), tools=builtin.web)
agent.run("Fetch the HTML of https://example.com and list every <a> href you find.")

Discover and submit forms¶

make_web_tools() returns three tools bound to a shared session. Pass them alongside get_webpage_html:

import aimu
from aimu.tools.builtin import get_webpage_html, make_web_tools

agent = aimu.agents.Agent(
    aimu.client("anthropic:claude-sonnet-4-6"),
    tools=[get_webpage_html, *make_web_tools()],
)
agent.run("On https://httpbin.org/forms/post, submit the form with custname='Ada' and comments='hi'.")

find_forms(url) fetches the page and lists each form's index, submission URL (resolved to an absolute URL), HTTP method, and fields — including type=hidden fields, so CSRF tokens surface for the agent to echo back.
submit_form(url, method="POST", data=None) submits data to a URL. method="POST" sends it as a form body; method="GET" sends it as query parameters. It returns the response status, the final URL after redirects, and the (truncated) response body.

Because these tools close over one requests.Session, cookies set during one call are sent on the next. A login flow works end to end: find_forms scrapes the login form's hidden token, submit_form posts credentials (the server sets a session cookie), and a later submit_form(url, method="GET") reads a page behind the login using that cookie.

Call a JSON API¶

submit_json(url, payload, method="POST") is the JSON counterpart to submit_form. The model supplies payload as a structured object, not a string of JSON, so it never hand-serializes: the @tool decorator advertises the parameter as a JSON Schema object and validates the model's argument at dispatch. The body goes out JSON-encoded with the matching Content-Type.

agent.run("Create an item named 'widget' with tags ['a','b'] at https://api.example.com/items.")

method accepts "POST" (default), "PUT", and "PATCH" — the verbs that carry a body. GET is submit_form's job, and DELETE is deliberately absent. Anything else is refused before a request goes out, with a message the model can act on.

One limit worth knowing: payload is a JSON object, so a top-level array (a bulk endpoint that takes [{...}, {...}]) cannot be expressed. Wrap it in an object if the API allows, or write your own @tool for that endpoint.

Error bodies come back, which is the point

Both submit tools return the response whatever its status, because a 400 or 422 body normally names the field that was rejected. That is what a corrected retry needs; an agent told only Error: 422 has nothing to work from. Transport failures (DNS, connection refused, timeout) are still reported as an error string, since there is no response to show.

Authentication, and why there is no `headers` argument¶

Neither submit tool lets the model set request headers. Credentials belong on the session you pass to the factory, so a token you configure for one host cannot be aimed at a host the model names:

import requests

session = requests.Session()
session.headers["Authorization"] = "Bearer …"
tools = make_web_tools(session=session)

Note the remaining exposure: the model still chooses the URL, and the session's credentials ride along with it. Combine that with a page the agent just read (whose text can instruct it) and an arbitrary-target write is a confused-deputy risk. The control for that is the tool-approval hook, which sees the URL before the call runs and can refuse it with a reason.

Redirects stop at the host boundary¶

Both submit tools follow a redirect only while it stays on the same host. A cross-host redirect is reported instead of followed, naming the target:

Status: 307
URL: http://site.example/start
Not followed: this redirects to http://other.example/sink on a different host. Nothing was sent
there. Submit to that URL directly if you intend to.

The reason is specific: a 307 or 308 replays the request method and body, so following one off-host would re-send your whole payload to a host the model never named and no approval policy ever inspected. (requests does strip Authorization across hosts, so the credential is not the leak — the payload is.) Refusing the hop puts the new host back in front of the approval gate, because reaching it now takes a fresh tool call that the policy sees.

Same-host redirects are followed, which is what keeps the login flow above working in one call, with a 301/302/303 downgrading to a bodyless GET exactly as requests and browsers do. A chain is capped at five hops, and hitting the cap is reported rather than followed further. The read tools are unaffected: a redirected GET carries no body and no credential, so requests' own handling stays in charge there.

Control the session¶

Pass your own session to share one across several tool sets, or to raise the timeout and the response cap:

tools = make_web_tools(session=session, timeout=30, max_content_chars=40000)

user_agent= sets the User-Agent sent on every call (it is a per-request header, so it takes precedence over the session's own). Pass user_agent=None to send none and leave a session-level User-Agent in charge.

Confirm before submitting¶

submit_form and submit_json perform writes, to any URL the model names. To require confirmation before one runs, gate it with the tool-approval hook — the policy sees the tool name and the model's arguments, so you can approve reads and prompt on submits:

def confirm_submits(name, arguments):
    if name not in ("submit_form", "submit_json"):
        return True
    return input(f"Submit to {arguments.get('url')}? [y/N] ").strip().lower() == "y"

agent = aimu.agents.Agent(client, tools=[get_webpage_html, *make_web_tools()],
                          tool_approval=confirm_submits)

Async¶

Both entry points are re-exported from aimu.aio.tools.builtin; the async agent dispatches these sync requests tools via asyncio.to_thread:

from aimu import aio
from aimu.aio.tools.builtin import get_webpage_html, make_web_tools

agent = aio.Agent(aio.client("ollama:qwen3:8b"), tools=[get_webpage_html, *make_web_tools()])
await agent.run("…")