Skip to content

Providers API

A provider wraps one LLM API. Agents talk to providers, so switching model or vendor means swapping the provider, not rewriting the agent.

Choosing an endpoint

Every provider takes the same three core arguments:

Provider(api_key: str, model: str, base_url: str = None)

base_url points a provider at a different address for the same API shape — a gateway, a proxy, or a self-hosted server. Omit it to use the vendor's own endpoint. The value is retained as api.base_url, alongside api.model.

# OpenAI's own endpoint
api = OpenAIChatAPI(api_key="your-api-key", model="gpt-4o-mini")

# Any OpenAI-compatible server: OpenRouter, vLLM, Ollama, llama-cpp-server
api = OpenAIChatAPI(
    api_key="your-api-key",
    model="qwen/qwen3.5-9b",
    base_url="https://openrouter.ai/api/v1",
)

# Anthropic behind an internal gateway
api = AnthropicChatAPI(
    api_key="your-api-key",
    model="claude-sonnet-4-20250514",
    base_url="https://anthropic-gateway.internal/v1",
)

MistralChatAPI forwards base_url to the Mistral SDK's server_url argument, so the name is the same across every provider.

Async providers are not re-exported from ToolAgents.provider; import them from their own modules:

from ToolAgents.provider.chat_api_provider.open_ai import AsyncOpenAIChatAPI
from ToolAgents.provider.chat_api_provider.anthropic import AsyncAnthropicChatAPI
from ToolAgents.provider.chat_api_provider.groq import AsyncGroqChatAPI
from ToolAgents.provider.chat_api_provider.mistral import AsyncMistralChatAPI

Settings

Assigning an undeclared setting name does nothing

Attribute assignment only routes to set_value for a name the provider already declares. settings.max_tokens = 4096 on a provider without a max_tokens setting silently creates a dead attribute — no error, and nothing is sent. Providers declare different sets:

Provider Declared settings
OpenAI temperature, top_p, tool_choice, extra_body, response_format
Anthropic temperature, top_p, top_k, max_tokens
Groq temperature, top_p
Mistral temperature, top_p

Check settings.setting_names() first, or use add_request_setting(name, value) to add one deliberately.

to_dict() emits the API request shape ({"PROVIDER_SETTINGS": ..., "REQUEST_SETTINGS": ..., "METADATA": ...}), not a round-trippable configuration; there is no from_dict. To describe a provider in a serializable form, see the pipelines ProviderConfig.

SettingLevel

Bases: Enum

Where a setting should be placed in the API request.

LLMSetting dataclass 

LLMSetting(
    name: str,
    default_value: Any,
    neutral_value: Any,
    level: SettingLevel = PROVIDER,
    current_value: Any = None,
)

Represents a single LLM setting with default and neutral values.

Simplified from the original - just stores the values directly instead of wrapping them in dicts.

get_value 

get_value() -> Any

Get current value.

set_value 

set_value(value: Any) -> None

Set current value.

reset 

reset() -> None

Reset to default value.

neutralize 

neutralize() -> None

Set to neutral (disabled) value.

is_neutral 

is_neutral() -> bool

Check if currently at neutral value.

ProviderSettings 

ProviderSettings(
    settings: Optional[List[LLMSetting]] = None,
)

Manages LLM settings with support for different placement levels.

Settings can be at provider level (top-level params) or in extra_body.

add_setting 

add_setting(setting: LLMSetting) -> None

Add a new setting.

add_request_setting 

add_request_setting(name: str, value: Any) -> None

Add a new setting.

add_provider_setting 

add_provider_setting(name: str, value: Any) -> None

Add a new setting.

remove_setting 

remove_setting(name: str) -> None

Remove a setting by name.

get_setting 

get_setting(name: str) -> Optional[LLMSetting]

Get a setting object by name.

setting_names 

setting_names() -> List[str]

Return the names of all settings, sorted.

Useful for reporting a typo: assigning an unknown name via attribute access silently creates a dead attribute instead of raising, so callers that accept setting names from configuration should validate against this list first.

get_value 

get_value(name: str) -> Any

Get current value of a setting.

set_value 

set_value(name: str, value: Any) -> None

Set current value of a setting.

update 

update(**kwargs) -> None

Update multiple settings at once.

reset 

reset(name: str) -> None

Reset a setting to default.

reset_all 

reset_all() -> None

Reset all settings to defaults.

neutralize 

neutralize(name: str) -> None

Set a setting to neutral value.

neutralize_all 

neutralize_all() -> None

Set all settings to neutral values.

to_dict 

to_dict(
    include: Optional[List[str]] = None,
    exclude: Optional[List[str]] = None,
    include_neutral: bool = True,
    param_mapping: Optional[Dict[str, str]] = None,
) -> Dict[str, Any]

Convert settings to dict for API request.

Parameters:

Name Type Description Default
include Optional[List[str]]

Only include these setting names

 None
exclude Optional[List[str]]

Exclude these setting names

 None
include_neutral bool

Include settings at neutral value

 True
param_mapping Optional[Dict[str, str]]

Rename parameters (e.g., {'max_tokens': 'max_new_tokens'})

 None

Returns: Dict with provider-level settings at top level and extra_body nested

copy 

copy() -> ProviderSettings

Create a deep copy of settings.

Provider interfaces

ChatAPIProvider

Bases: ABC

AsyncChatAPIProvider

Bases: ABC

Chat API providers

OpenAIChatAPI 

OpenAIChatAPI(
    api_key: str,
    model: str,
    base_url: str = "https://api.openai.com/v1",
    timeout: float = None,
    max_retries: int = None,
    provider_identifier: str = "openai",
    message_converter: BaseMessageConverter = OpenAIMessageConverter(),
    response_converter: BaseResponseConverter = OpenAIResponseConverter(),
    debug_mode: bool = False,
)

Bases: ChatAPIProvider

AsyncOpenAIChatAPI 

AsyncOpenAIChatAPI(
    api_key: str,
    model: str,
    base_url: str = "https://api.openai.com/v1",
    timeout: float = None,
    max_retries: int = None,
    provider_identifier: str = "openai",
    message_converter: BaseMessageConverter = OpenAIMessageConverter(),
    response_converter: BaseResponseConverter = OpenAIResponseConverter(),
    debug_mode: bool = False,
)

Bases: AsyncChatAPIProvider

AnthropicChatAPI 

AnthropicChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: ChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override, for a gateway, proxy, or self-hosted server speaking this API.

 None

AsyncAnthropicChatAPI 

AsyncAnthropicChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: AsyncChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override, for a gateway, proxy, or self-hosted server speaking this API.

 None

GroqChatAPI 

GroqChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: ChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override, for a gateway, proxy, or self-hosted server speaking this API.

 None

AsyncGroqChatAPI 

AsyncGroqChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: AsyncChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override, for a gateway, proxy, or self-hosted server speaking this API.

 None

MistralChatAPI 

MistralChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: ChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override. The Mistral SDK names this server_url; it is accepted here as base_url so every provider reads the same.

 None

AsyncMistralChatAPI 

AsyncMistralChatAPI(
    api_key: str,
    model: str,
    base_url: str = None,
    timeout: float = None,
    max_retries: int = None,
)

Bases: AsyncChatAPIProvider

Create the provider.

Parameters:

Name Type Description Default
api_key str

API key for the endpoint.

 required
model str

Model identifier.

 required
base_url str

Optional endpoint override. The Mistral SDK names this server_url; it is accepted here as base_url so every provider reads the same.

 None

Streaming

StreamingChatMessage

Bases: BaseModel

Represents a streaming chat API response.

Completion providers

For traditional completion-based models rather than chat APIs.

CompletionProvider 

CompletionProvider(
    completion_endpoint: CompletionEndpoint,
    tokenizer: LLMTokenizer = None,
    message_converter: BaseMessageConverter = MistralMessageConverterLlamaCpp(),
    tool_call_handler: LLMToolCallHandler = TemplateToolCallHandler(),
)

Bases: ChatAPIProvider