Switch providers¶

Every AIMU client implements the same BaseModelClient interface, so swapping backends is a one-line change. ModelClient is the factory; it accepts either a Model enum member or a "provider:model_id" string.

That is worth more than the refactoring convenience: it is what makes comparing models cheap. Run the same task on three providers and the only thing that varied is the model, so the difference you see is the models' and not your harness's. Compare models is that loop end to end.

Use a model string¶

import aimu

client = aimu.client("anthropic:claude-sonnet-4-6")
client = aimu.client("ollama:qwen3.5:9b")
client = aimu.client("openai:gpt-4o-mini")
client = aimu.client("gemini:gemini-2.5-flash")

Model strings have the form "provider:model_id". Colons inside the model id (e.g. Ollama's qwen3.5:9b) are preserved.

Point at a remote server¶

Every local provider defaults to localhost (see the provider keys table). To reach one on another host or port, append @<endpoint> to the model string:

client = aimu.client("llamaserver:qwen3-8b.gguf@http://gpu-box:8080/v1")
client = aimu.client("vllm:Qwen/Qwen3-8B@http://gpu-box:8000/v1")
client = aimu.client("ollama:qwen3.5:9b@http://gpu-box:11434")

This is the string-form equivalent of a constructor kwarg (see Pass provider-specific kwargs); use whichever fits how the model is configured. Each provider takes its own SDK's spelling — base_url= for the OpenAI-compatible providers, host= for the native ollama one:

client = aimu.client("vllm:Qwen/Qwen3-8B", base_url="http://gpu-box:8000/v1")
client = aimu.client("ollama:qwen3.5:9b", host="http://gpu-box:11434")

Remote Ollama¶

The native ollama provider takes host= on all three of its clients (OllamaClient, AsyncOllamaClient, OllamaEmbeddingClient), forwarded to the ollama SDK verbatim, so a bare host, host:port, or scheme://host:port all work. Two things to know:

Pass the server root, not /v1. The native API is served from the root, so host="http://gpu-box:11434/v1" raises ValueError. If you want Ollama's OpenAI-compatible endpoint (which does want /v1), use the ollama-openai provider instead.
Give the embedding client the same host. Embedding a corpus against one server and querying against another silently mixes vectors from two different models.

client = aimu.client("ollama:qwen3.5:9b", host="gpu-box")
embedder = aimu.embedding_client("ollama:nomic-embed-text", host="gpu-box")

OllamaClient pulls the model in its constructor, so with host= set the pull runs on the remote machine.

Local discovery stays env-driven¶

host= and base_url= configure one client. They are invisible to the local-availability probes behind aimu.available_text_models(), the omitted-model default, and ambiguous bare-name resolution — those run before any client exists, so they read OLLAMA_HOST (else 127.0.0.1:11434) for Ollama and each OpenAI-compatible provider's default base_url for the local servers. Export OLLAMA_HOST when you want discovery to see a remote server too:

export OLLAMA_HOST=http://gpu-box:11434

With it set, aimu.client() with no model can auto-select from the remote server's models, and aimu.available_text_models() lists them.

Run a model not in the catalog¶

AIMU normally requires the id to name a model it ships a spec for, so it never runs with guessed capabilities. For an OpenAI-compatible server you can declare the capabilities inline instead, with ;<flags> (comma-separated from tools, thinking, vision, audio, structured). The id then resolves to an ad-hoc model with exactly those capabilities:

# a custom fine-tune served by llama-server on another host
client = aimu.client("llamaserver:my-finetune.gguf@http://gpu-box:8080/v1;tools,thinking")

# any OpenAI-compatible server, via the generic prefix (an endpoint is required)
client = aimu.client("openai-compat:my-model@http://gpu-box:9000/v1;tools")

Notes:

@<endpoint> is accepted by the local providers (ollama, llamaserver, lmstudio, vllm, hf-openai, sglang, ollama-openai, omlx) and the generic openai-compat prefix. Passing it to a cloud or in-process provider raises ValueError.
Ad-hoc ids are narrower than endpoints. The native ollama provider accepts an endpoint but stays catalog-only: its ids are registry tags whose capabilities AIMU knows, so ollama:some-unknown-tag raises rather than becoming an ad-hoc model. The ad-hoc form is for providers whose ids are user-chosen (a GGUF filename, an oMLX directory name, an LM Studio loaded-model key).
The generic openai-compat prefix always requires @<base_url> (it has no default) and always takes an ad-hoc id.
Flags apply only to ad-hoc ids, and an omitted flag defaults to False, so declare ;tools (etc.) for the capabilities your model actually has. Passing flags with a catalog id raises ValueError — the catalog already declares its capabilities.
No authentication is added; the endpoint is assumed unauthenticated (api_key is unset).

Use an enum member¶

For IDE autocomplete and type checking:

from aimu.models import ModelClient, OllamaModel, AnthropicModel

client = ModelClient(OllamaModel.QWEN_3_5_9B)
client = ModelClient(AnthropicModel.CLAUDE_SONNET_4_6)

Resolve a name, string, or enum uniformly¶

When you accept a model from a CLI flag or config and don't know which form it'll arrive in, resolve_model_enum (text) and resolve_image_model_enum (image) normalise all three to an enum member:

import aimu
from aimu.models import AnthropicModel

aimu.resolve_model_enum(AnthropicModel.CLAUDE_SONNET_4_6)   # enum member → returned unchanged
aimu.resolve_model_enum("anthropic:claude-sonnet-4-6")      # "provider:model_id" string
aimu.resolve_model_enum("CLAUDE_SONNET_4_6")                # bare enum-member name

aimu.resolve_image_model_enum("FLUX_2_KLEIN_4B")           # same, for image models
aimu.resolve_image_model_enum("gemini:nano-banana")

Pass the result straight to aimu.client(...) / aimu.image_client(...) (both also accept an enum member):

client = aimu.client(aimu.resolve_model_enum(args.model))

A bare text name is often ambiguous: the same id ships under several providers (e.g. QWEN_3_8B is an Ollama, HuggingFace, and OpenAI-compat model). resolve_model_enum resolves the tie by preferring a provider where the model is already available locally (running Ollama → cached HuggingFace → reachable local server, tool-capable first), logging the choice at WARNING. If it isn't available anywhere, it raises ValueError listing the "provider:model_id" options, so pin the provider with the string form when you need a specific one. (resolve_image_model_enum has no local-availability notion and raises on the rare ambiguity.)

Use whatever model is already available locally¶

Omit model= entirely and AIMU resolves a default: the AIMU_LANGUAGE_MODEL env var if set, otherwise the first already-available local model (running Ollama → cached HuggingFace → reachable local OpenAI-compat server, tool-capable preferred). A cloud provider is never auto-selected and weights are never downloaded.

client = aimu.client()                          # auto-resolved default (logs the pick at WARNING)

export AIMU_LANGUAGE_MODEL="ollama:qwen3.5:9b"  # pin the default deterministically

The env var takes the same full model string an explicit model= does, so a remote server can be pinned there too:

export AIMU_LANGUAGE_MODEL="ollama:qwen3.5:9b@http://gpu-box:11434"

That pins the client only. Discovery stays endpoint-blind, so export OLLAMA_HOST=http://gpu-box:11434 alongside it when available_text_models() should see the remote server's models as well.

To inspect or choose among what's available instead of taking the auto-pick:

aimu.available_text_models()            # list[Model]: locally loadable models, provider-priority order
aimu.resolve_default_text_model()       # the single auto-pick, as the full string (endpoint included)
aimu.resolve_default_text_model_enum()  # the same pick as an enum member (no endpoint, no flags)

Provider keys¶

Provider key	Extra	API key env var
`ollama`	`aimu[ollama]`	None (`OLLAMA_HOST`, else localhost:11434)
`hf`	`aimu[hf]`	None
`llamacpp`	`aimu[llamacpp]`	None (`model_path=` required)
`anthropic`	`aimu[anthropic]`	`ANTHROPIC_API_KEY`
`openai`	`aimu[openai_compat]`	`OPENAI_API_KEY`
`gemini`	`aimu[openai_compat]`	`GOOGLE_API_KEY`
`lmstudio`	`aimu[openai_compat]`	None (localhost:1234)
`ollama-openai`	`aimu[openai_compat]`	None (localhost:11434)
`hf-openai`	`aimu[openai_compat]`	None (localhost:8000)
`vllm`	`aimu[openai_compat]`	None (localhost:8000)
`llamaserver`	`aimu[openai_compat]`	None (localhost:8080)
`sglang`	`aimu[openai_compat]`	None (localhost:30000)
`omlx`	`aimu[openai_compat]`	None (localhost:8000)

See the provider matrix for full details.

Run MLX models on Apple Silicon¶

MLX is Apple's ML framework; on Apple Silicon it typically generates 20-40% faster than the llama.cpp Metal backend. Three providers reach MLX-optimized weights:

import aimu

# 1. oMLX -- a dedicated MLX server. `omlx serve --model-dir ~/models` first.
aimu.client("omlx:Qwen3.6-35B-A3B-4bit")
aimu.client("omlx:Muse-Glimmer-30B-4bit")  # needs oMLX >= 0.5.8.dev3 for its ATEM parsers

# 2. LM Studio's MLX engine (picked automatically for MLX weights).
aimu.client("lmstudio:qwen3.6-35b-a3b-4bit")

# 3. Ollama 0.19+ uses MLX automatically on Apple Silicon. Nothing to configure, and the
#    tag is unchanged -- this is the same call you'd make on a Linux box.
aimu.client("ollama:qwen3.6:35b")

hf and llamacpp are not MLX paths. HuggingFaceClient is torch/transformers and LlamaCppClient is GGML/GGUF; neither can load MLX's quantized safetensors layout. mlx-community models are hosted on the HuggingFace Hub but only mlx-lm/mlx-vlm can execute them, so AIMU reaches MLX through servers rather than an in-process client.

oMLX ids are --model-dir subdirectory names, so the catalog can only ever be a convention. For any other layout, declare the model ad hoc — flags default to False, so spell them out:

aimu.client("omlx:my-own-conversion-4bit;tools,thinking,vision")
aimu.client("omlx:Qwen3.6-35B-A3B-4bit@http://mac-studio:8000/v1")  # headless Mac on the LAN

oMLX's default port (8000) is shared with vLLM and HF Transformers Serve, so pass @<base_url> when you run more than one of them.

Check what the provider supports¶

client = aimu.client("ollama:qwen3.5:9b")
client.is_tool_using_model    # True
client.is_thinking_model      # True
client.is_vision_model        # False

Pass provider-specific kwargs¶

Extra kwargs are forwarded to the underlying client constructor:

# Ollama: keep the model warm for 5 minutes
aimu.client("ollama:qwen3.5:9b", model_keep_alive_seconds=300)

# llama.cpp: load a local GGUF file
aimu.client("llamacpp:qwen3-8b", model_path="/path/to/qwen3-8b.gguf")

# LM Studio: point at a non-default host
aimu.client("lmstudio:qwen3.5-9b", base_url="http://myserver:1234/v1")

Each provider client's constructor signature is in the API reference.

Timeouts and retries¶

Networked clients take timeout (seconds) and max_retries, forwarded straight to the provider SDK's own request timeout and bounded retry-on-transient-failure. AIMU adds no retry logic of its own.

import aimu

# Cloud and local OpenAI-compat servers: both supported
client = aimu.client("anthropic:claude-sonnet-4-6", timeout=30, max_retries=5)
client = aimu.client("openai:gpt-4o", timeout=30, max_retries=5)
client = aimu.client("vllm:Qwen/Qwen3-8B", timeout=30, max_retries=5)

These are SDK-native (the anthropic and openai SDKs implement timeout + retry), so the names and behavior match those SDKs exactly. Unset values fall back to each SDK's defaults.

Two caveats:

Ollama (native provider) supports timeout but has no request-retry; passing max_retries raises ValueError. Use the ollama-openai provider (which routes through the OpenAI SDK) if you need retries against Ollama.
In-process providers (hf:, llamacpp:) run locally with no HTTP request, so they don't accept timeout / max_retries (passing them raises TypeError).

Provider failover¶

Per-client max_retries retries the same provider. To fall back to a different provider when one is down, wrap an ordered list of clients in a FallbackClient:

import aimu
from aimu.models import FallbackClient

client = FallbackClient([
    aimu.client("anthropic:claude-sonnet-4-6", timeout=30, max_retries=2),  # preferred
    aimu.client("openai:gpt-4o", timeout=30, max_retries=2),                # fallback
    aimu.client("ollama:qwen3.5:9b"),                                       # last resort, local
])

print(client.chat("Hello"))   # tries Anthropic; on error falls over to OpenAI, then Ollama

The first client that answers wins. A client that raises hands off to the next, carrying the same conversation history, so a multi-turn chat survives a mid-conversation failover. When every client fails, FallbackExhaustedError is raised with the last error chained as its cause.

FallbackClient is itself a BaseModelClient, so it composes everywhere a plain client does:

from aimu.agents import Agent

agent = Agent(client, "You are a helpful assistant.", tools=[...])   # resilient agent

Notes:

What triggers failover. By default any Exception. Narrow it with retry_on= (e.g. FallbackClient([...], retry_on=(TimeoutError, ConnectionError))) so permanent errors surface immediately instead of being masked.
Capabilities (is_thinking_model, model, etc.) reflect the first client, so use capability-compatible clients in one set.
Streaming fails over only before the first chunk is emitted; an error mid-stream propagates rather than replaying.
Async: aio.AsyncFallbackClient is the one-for-one async twin.

Combine the two layers: max_retries/timeout give in-SDK retry against one provider, FallbackClient gives cross-provider failover on top.

Anthropic prompt caching¶

An agent resends the same large prefix to the model every turn: its system prompt and tool schemas. On Anthropic you can cache that prefix (cheaper, faster) by opting in with cache_prompt=True:

import aimu
from aimu.agents import Agent

client = aimu.client("anthropic:claude-sonnet-4-6", cache_prompt=True)
agent = Agent(client, "You are a helpful assistant with a long, detailed system prompt...", tools=[...])

agent.run("First question")    # writes the cache (system prompt + tools)
agent.run("Second question")   # reads it back — most of the prefix is cached

cache_prompt=True marks the system prompt and the tool definitions with Anthropic's cache_control ephemeral breakpoints at request time. Notes:

Anthropic only. It's a provider-specific kwarg; passing it to another provider raises TypeError.
Safe to leave on. Below Anthropic's minimum cacheable size (1024 tokens; 2048 for Haiku) the API simply doesn't cache — no error.
Observe it. When the response reports caching, client.last_usage includes cache_creation_input_tokens (first call, writing the cache) and cache_read_input_tokens (later calls, reading it).
Conversation messages aren't cached (they change every turn); the win is the stable system-prompt + tools prefix.

Failure modes¶

aimu.client("foo:bar") raises ValueError listing the available providers if the prefix is unknown, and raises with the available model ids if the prefix is valid but the id isn't:

>>> aimu.client("foo:bar")
ValueError: Unknown provider 'foo'. Available providers (with installed deps): ['anthropic', 'ollama', ...]

>>> aimu.client("anthropic:claude-nonsense")
ValueError: Provider 'anthropic' has no model id 'claude-nonsense'. Available: ['claude-fable-5', 'claude-haiku-4-5', 'claude-opus-4-6', 'claude-opus-4-7', 'claude-opus-4-8', 'claude-opus-5', 'claude-sonnet-4-6', 'claude-sonnet-5']

The extended grammar has two more guardrails. An @<base_url> on a provider that doesn't take one, and ;<flags> on a catalog id, both raise ValueError:

>>> aimu.client("anthropic:claude-sonnet-4-6@http://proxy/v1")
ValueError: Provider 'anthropic' does not accept an @base_url. Supported: ['hf-openai', 'llamaserver', 'lmstudio', 'ollama-openai', 'omlx', 'openai-compat', 'sglang', 'vllm'].

>>> aimu.client("llamaserver:qwen3-8b.gguf@http://gpu-box:8080/v1;tools")
ValueError: Capability flags are not allowed with the known model id 'qwen3-8b.gguf'; it already declares its capabilities. Use a different id to define an ad-hoc model.