aimu.rag¶
Retrieval-augmented generation primitives: plain functions over the
MemoryStore interface. See the RAG how-to.
Splitting¶
aimu.rag.split_text ¶
split_text(text: str, *, chunk_size: int = 1000, chunk_overlap: int = 200, separators: list[str] | None = None, length_function: Callable[[str], int] = len) -> list[str]
Split text into overlapping chunks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text to split. |
required |
chunk_size
|
int
|
Maximum size of each chunk, measured by |
1000
|
chunk_overlap
|
int
|
How much trailing content of one chunk repeats at the start of the
next (continuity across boundaries). Must be smaller than |
200
|
separators
|
list[str] | None
|
Separator hierarchy, largest semantic unit first. Defaults to paragraphs -> lines -> sentences -> words -> characters. |
None
|
length_function
|
Callable[[str], int]
|
Measures chunk size. Defaults to |
len
|
Returns:
| Type | Description |
|---|---|
list[str]
|
A list of chunk strings in document order. Empty / whitespace-only input returns |
list[str]
|
|
Pipeline¶
aimu.rag.ingest ¶
ingest(store: Any, documents: Union[str, list[str]], *, chunk_size: int = 1000, chunk_overlap: int = 200, separators: list[str] | None = None, length_function: Callable[[str], int] = len) -> int
Split documents into chunks and store each chunk in store.
chunk_size, chunk_overlap, separators, and length_function are forwarded
to :func:~aimu.rag.split_text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
Any
|
Any object with a |
required |
documents
|
Union[str, list[str]]
|
A single document string or a list of them. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The number of chunks stored. |
aimu.rag.retrieve ¶
Return up to n_results stored chunks relevant to query.
A thin, RAG-named pass-through to store.search(). Extra keyword arguments are
forwarded to the store (e.g. max_distance= for
:class:~aimu.memory.SemanticMemoryStore). To rerank, fetch a wider set here and pass
it through :func:~aimu.rag.rerank.
aimu.rag.format_context ¶
Join retrieved chunks into a single context string for prompt augmentation.
With numbered=True each chunk is prefixed [1], [2] ... (useful when you
want the model to cite which chunk it used).
Ranking¶
aimu.rag.bm25_scores ¶
bm25_scores(query: str, documents: list[str], *, k1: float = 1.5, b: float = 0.75, stopwords: Collection[str] = STOPWORDS) -> list[float]
Score each document's relevance to query with Okapi BM25.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
str
|
The text to rank against. |
required |
documents
|
list[str]
|
The candidates, scored as a collection (term rarity is measured across them). |
required |
k1
|
float
|
Term-frequency saturation. |
1.5
|
b
|
float
|
Length normalization, from 0 (none) to 1 (full). |
0.75
|
stopwords
|
Collection[str]
|
Query words to ignore. Defaults to :data: |
STOPWORDS
|
Returns:
| Type | Description |
|---|---|
list[float]
|
One score per document, in input order; 0.0 for a document sharing no term with the |
list[float]
|
query. An empty |
aimu.rag.reciprocal_rank_fusion ¶
reciprocal_rank_fusion(rankings: Sequence[Sequence[str]], *, k: int = 60) -> list[tuple[str, float]]
Fuse rankings (each a sequence of keys, best first) into one ranking.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rankings
|
Sequence[Sequence[str]]
|
The rankings to fuse. A key may appear in any number of them; within one ranking only its first (best) position counts. |
required |
k
|
int
|
Damps the advantage of the very top ranks. 60 is the value the original paper found robust. |
60
|
Returns:
| Type | Description |
|---|---|
list[tuple[str, float]]
|
|
list[tuple[str, float]]
|
appear, reading the rankings in the order given. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If k is negative. |
Reranking¶
aimu.rag.rerank ¶
Optional cross-encoder reranking for retrieved chunks.
A retriever (vector or keyword search) is fast but coarse. Reranking re-scores each
candidate against the query with a cross-encoder -- slower but more accurate -- so you
fetch a wide set, then keep the best few. Backed by sentence-transformers (the [hf]
extra). Plain function, not a class.
rerank ¶
rerank(query: str, documents: list[str], *, model: str = DEFAULT_RERANK_MODEL, top_n: Optional[int] = None) -> list[str]
Reorder documents by cross-encoder relevance to query, most relevant first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
str
|
The search query. |
required |
documents
|
list[str]
|
Candidate chunks (e.g. the output of :func: |
required |
model
|
str
|
A sentence-transformers |
DEFAULT_RERANK_MODEL
|
top_n
|
Optional[int]
|
Keep only the top N after reranking (default: keep all, reordered). |
None
|
Returns:
| Type | Description |
|---|---|
list[str]
|
The documents reordered by descending relevance, truncated to |
list[str]
|
An empty input returns |