Skip to content

Knowledge API

Retrieval, document ingestion, text splitting, web search and crawling.

Most of this needs an optional extra

Only the base classes, BM25 retrieval and the text splitters are pure Python. The rest pull in third-party packages:

Area Extra
Chroma, sentence-transformers embeddings, cross-encoder reranking, and the numpy types the base classes use memory
PDF text extraction documents
OCR PDF ingestion ocr
Web search and crawling search

Plain PDF text extraction lives in its own documents extra, so reading a PDF's text layer does not drag in the OCR stack and its system binaries.

RAG

RAG 

RAG(vector_database_provider: VectorDatabaseProvider)

Retrieval over any vector database provider.

A thin facade: documents go in through add_document / add_documents and come back out through retrieve_documents. The storage, embedding and reranking strategies are whatever providers you hand it, not a fixed choice.

Vector databases

A VectorDatabaseProvider stores chunks and returns the nearest matches for a query. BM25VectorDatabaseProvider is a pure-Python keyword retriever with no dependencies; EnsembleVectorDatabaseProvider fuses the scores of two providers, which is the usual way to combine keyword and semantic retrieval.

VectorDatabaseProvider 

VectorDatabaseProvider(
    embedding_provider: EmbeddingProvider,
    reranking_provider: RerankingProvider,
)

Bases: ABC

VectorSearchResult dataclass 

VectorSearchResult(
    ids: list[str],
    chunks: list[str],
    scores: list[float],
    metadata: Optional[list[dict[str, Any]]] = None,
)

VectorCollection dataclass 

VectorCollection(
    ids: list[str],
    chunks: list[str],
    embeddings: list[ndarray],
    metadata: Optional[list[dict[str, Any]]] = None,
)

BM25VectorDatabaseProvider 

BM25VectorDatabaseProvider(
    embedding_provider: EmbeddingProvider,
    reranking_provider: Optional[RerankingProvider] = None,
    tokenizer: str = "[A-Za-zÀ-ÖØ-öø-ÿ0-9_]+",
    lowercase: bool = True,
)

Bases: VectorDatabaseProvider

A sparse keyword retriever that conforms to your VectorDatabaseProvider API. Uses in-memory BM25. RerankingProvider is optional and applied post-retrieval.

EnsembleVectorDatabaseProvider 

EnsembleVectorDatabaseProvider(
    embedding_provider: EmbeddingProvider,
    dense_provider: VectorDatabaseProvider,
    sparse_provider: VectorDatabaseProvider,
    reranking_provider: Optional[RerankingProvider] = None,
    dense_weight: float = 0.5,
    sparse_weight: float = 0.5,
    dense_scores_are_similarities: bool = False,
    shortlist_multiplier: int = 4,
)

Bases: VectorDatabaseProvider

Query two providers (e.g., BM25 + Chroma), fuse scores, optionally rerank with cross-encoder.

ChromaDbVectorDatabaseProvider 

ChromaDbVectorDatabaseProvider(
    embedding_provider: EmbeddingProvider,
    reranking_provider: RerankingProvider = None,
    persistent_db_path="./retrieval_memory",
    default_collection_name="default_collection",
    persistent: bool = False,
)

Bases: VectorDatabaseProvider

Embeddings

EmbeddingProvider 

EmbeddingProvider(prefix_config: EmbeddingPrefixConfig)

Bases: ABC

EmbeddingTask

Bases: Enum

EmbeddingResult dataclass 

EmbeddingResult(embeddings: ndarray)

EmbeddingPrefixConfig dataclass 

EmbeddingPrefixConfig(
    query: str = "query: ",
    store: str = "store: ",
    cluster: str = "cluster: ",
    custom: str = "custom: ",
)

OpenAIEmbeddingProvider 

OpenAIEmbeddingProvider(
    model: str = "text-embedding-3-small",
    api_key: Optional[str] = None,
    base_url: Optional[str] = None,
    prefix_config: EmbeddingPrefixConfig = EmbeddingPrefixConfig(),
)

Bases: EmbeddingProvider

OpenAI-based EmbeddingProvider.

  • api_key: if None, uses OPENAI_API_KEY env var.
  • base_url: override the OpenAI API base URL (e.g. for proxies / self-hosted gateways).

SentenceTransformerEmbeddingProvider 

SentenceTransformerEmbeddingProvider(
    sentence_transformer_model_path: str = "all-MiniLM-L6-v2",
    trust_remote_code: bool = False,
    device: str = "cpu",
    prefix_config: EmbeddingPrefixConfig = EmbeddingPrefixConfig(),
)

Bases: EmbeddingProvider

Reranking

RerankingProvider

Bases: ABC

RerankingResult dataclass 

RerankingResult(reranked_documents: list[RerankedDocument])

RerankedDocument dataclass 

RerankedDocument(
    content: str, additional_data: dict[str, Any]
)

CrossEncoderRerankingProvider 

CrossEncoderRerankingProvider(
    rerank_model: str = "mixedbread-ai/mxbai-rerank-xsmall-v1",
    trust_remote_code: bool = False,
    device: str = "cpu",
)

Bases: RerankingProvider

Documents

A Document is a list of DocumentChunks. DocumentGenerator turns raw text into one using a text splitter; a DocumentProvider loads documents from a source such as a PDF.

Document dataclass 

Document(
    id: str,
    document_chunks: list[DocumentChunk] = list[
        DocumentChunk
    ](),
    metadata: Optional[Dict[str, Any]] = None,
)

DocumentChunk dataclass 

DocumentChunk(
    id: str,
    parent_doc_id: str,
    content: str,
    chunk_index: int,
    size_in_characters: int,
)

DocumentGenerator 

DocumentGenerator(text_splitter: TextSplitter = None)

DocumentProvider

Bases: ABC

PDFProvider 

PDFProvider(folder: str, text_splitter: TextSplitter)

Bases: DocumentProvider

A document provider that extracts text directly from PDF files without using OCR.

Initialize the PDF provider.

Parameters:

Name Type Description Default
folder str

Path to the folder containing PDF files

 required
text_splitter TextSplitter

Text splitter instance for chunking documents

 required

get_documents 

get_documents(**kwargs) -> List[Document]

Get documents from PDF files in the specified folder.

Returns:

Type Description
List[Document]

List[Document]: List of processed documents

PDFOCRProvider 

PDFOCRProvider(folder: str, text_splitter: TextSplitter)

Bases: DocumentProvider

Text splitting

TextSplitter

Bases: ABC

RecursiveCharacterTextSplitter 

RecursiveCharacterTextSplitter(
    separators,
    chunk_size,
    chunk_overlap,
    length_function=len,
    keep_separator=False,
)

Bases: TextSplitter

A class that recursively splits text by a hierarchy of separators and eventually falls back to fixed-size splitting with overlap.

SimpleTextSplitter 

SimpleTextSplitter(chunk_size, overlap=0)

Bases: TextSplitter

A class for splitting text into consecutive overlapping (or non-overlapping) chunks.

NonTextSplitter

Bases: TextSplitter

A class that does not split text at all, returning the entire text as a single chunk.

Parameters:

text : str The string to be chunked.

Text processing

TextTransformer runs a prompt template over a document through an agent. summarize_list_of_strings summarizes several documents at once, and SummarizingFunctionToolPostProcessor is a post-processor that shrinks long tool results before they return to the model.

TextTransformer dataclass 

TextTransformer(
    agent: ChatToolAgent,
    settings: ProviderSettings,
    prompt_template: MessageTemplate,
    system_prompt: str = None,
)

summarize_list_of_strings 

summarize_list_of_strings(
    agent: ChatToolAgent,
    settings: ProviderSettings,
    strings: list[str],
)

SummarizingFunctionToolPostProcessor 

SummarizingFunctionToolPostProcessor(
    summarizing_agent: ChatToolAgent,
    settings: ProviderSettings,
)

Bases: PostProcessor

Each provider returns a list of URLs for a query. Fetching and extracting the page contents is a crawler's job.

WebSearchProvider

Bases: ABC

A source of web search results.

Implementations return URLs only; fetching and extracting the page contents is a :class:~ToolAgents.knowledge.web_crawler.web_crawler.WebCrawler's job.

search_web abstractmethod 

search_web(
    search_query: str, num_results: int
) -> list[str]

Search the web and return the result URLs.

Parameters:

Name Type Description Default
search_query str

The query to search for.

 required
num_results int

Maximum number of URLs to return.

 required

Returns:

Type Description
list[str]

list[str]: URLs of the results.

DDGWebSearchProvider

Bases: WebSearchProvider

search_web 

search_web(
    search_query: str, num_results: int
) -> list[str]

Search the web, like a google search query. Returns a list of result URLs Args: search_query (str): Search query. num_results (int): Number of results to return. Returns: List[str]: List of URLs with search results.

GoogleWebSearchProvider

Bases: WebSearchProvider

search_web 

search_web(
    search_query: str, num_results: int
) -> list[str]

Search the web, like a google search query. Returns a list of result URLs Args: search_query (str): Search query. num_results (int): Number of results to return. Returns: List[str]: List of URLs with search results.

HackernewsWebSearchProvider

Bases: WebSearchProvider

search_web 

search_web(
    search_query: str, num_results: int
) -> list[str]

Search the web, like a google search query. Returns a list of result URLs Args: search_query (str): Search query. num_results (int): Number of results to return. Returns: List[str]: List of URLs with search results.

Web crawling

WebCrawler

Bases: ABC

get_website_content_from_url abstractmethod 

get_website_content_from_url(url: str) -> str

Get the website content from an url.

get_website_content_from_urls abstractmethod 

get_website_content_from_urls(urls: list[str]) -> list[str]

Get the website content from urls.

TrafilaturaWebCrawler

Bases: WebCrawler

get_website_content_from_url 

get_website_content_from_url(url: str) -> str

Get website content from a URL using Selenium and BeautifulSoup for improved content extraction and filtering.

Parameters:

Name Type Description Default
url str

URL to get website content from.

 required

Returns:

Name Type Description
str str

Extracted content including title, main text, and tables.

CamoufoxWebCrawler

Bases: WebCrawler

get_website_content_from_url 

get_website_content_from_url(url: str) -> str

Get website content from a URL using Selenium and BeautifulSoup for improved content extraction and filtering.

Parameters:

Name Type Description Default
url str

URL to get website content from.

 required

Returns:

Name Type Description
str str

Extracted content including title, main text, and tables.

HTML2Markdown 

HTML2Markdown()

convert 

convert(html: Union[str, Path]) -> str

Convert HTML to Markdown.

Parameters:

Name Type Description Default
html Union[str, Path]

HTML content as string or path to HTML file

 required

Returns:

Type Description
str

Converted Markdown text

convert_file 

convert_file(
    input_path: Union[str, Path],
    output_path: Optional[Union[str, Path]] = None,
) -> None

Convert HTML file to Markdown file.

Parameters:

Name Type Description Default
input_path Union[str, Path]

Path to input HTML file

 required
output_path Optional[Union[str, Path]]

Path to output Markdown file (optional)

 None