Providers Overview

ai-query supports multiple AI providers through a unified interface. Each provider has its own factory function that creates a LanguageModel instance.

Prompt caching

Prompt caching is automatic. There is no ai-query cache configuration and no application-level response cache: every request still runs the model and produces a new response.

OpenAI, DeepSeek, and supported Gemini models use their provider defaults.
Anthropic requests opt into its automatic ephemeral prompt cache.
Other providers retain their existing behavior.

Keep repeated content at the beginning of the prompt to improve the chance of a provider cache hit. Stable system instructions, tool definitions, and earlier conversation turns should come before request-specific content. Avoid timestamps, request IDs, or frequently changing tool schemas in the shared prefix. Inspect result.usage.cached_tokens, cache_write_tokens, and cache_miss_tokens to see the cache information reported by the provider. A zero means absent or unreported; ai-query does not infer cache activity that the provider does not expose.

Supported Providers

OpenAI

GPT-5, GPT-4o, and o-series reasoning models

Anthropic

Claude 3.5 Sonnet, Claude 3 Opus, and more

Google

Gemini 2.0, Gemini 1.5 Pro and Flash

OpenRouter

Access any model through one API

DeepSeek

DeepSeek Chat and Reasoner models

Groq

Groq LPU Inference

Workers AI

Cloudflare Workers AI via OpenAI-compatible endpoints

Custom Providers

Build your own provider adapters and wrappers

Meta Llama

Official Meta Llama API

xAI

Grok Models via xAI API

Cloud Providers

Bedrock

AWS Bedrock - managed foundation models

Bedrock requires an extra dependency: pip install ai-query[bedrock] or uv add ai-query[bedrock]

Quick Start

Each provider requires an API key set as an environment variable:

# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."

# Google
export GOOGLE_API_KEY="..."

# OpenRouter
export OPENROUTER_API_KEY="sk-or-v1-..."

# DeepSeek
export DEEPSEEK_API_KEY="sk-..."

# Groq
export GROQ_API_KEY="gsk-..."

# Workers AI
export CLOUDFLARE_ACCOUNT_ID="..."
export CLOUDFLARE_API_TOKEN="..."

Usage

All providers follow the same pattern - import the factory function and use it with generate_text() or stream_text():

from ai_query import generate_text
from ai_query.providers import openai, anthropic, google, openrouter, deepseek, groq, workers_ai

# OpenAI
result = await generate_text(model=openai("gpt-4o"), prompt="Hello!")

# Anthropic  
result = await generate_text(model=anthropic("claude-sonnet-4-20250514"), prompt="Hello!")

# Google
result = await generate_text(model=google("gemini-2.0-flash"), prompt="Hello!")

# OpenRouter (access any model)
result = await generate_text(model=openrouter("openai/gpt-4o"), prompt="Hello!")

# DeepSeek
result = await generate_text(model=deepseek("deepseek-chat"), prompt="Hello!")

# Groq (ultra-fast inference)
result = await generate_text(model=groq("llama-3.3-70b-versatile"), prompt="Hello!")

# Workers AI
result = await generate_text(model=workers_ai("@cf/moonshotai/kimi-k2.5"), prompt="Hello!")

Switching Providers

One of the key benefits of ai-query is how easy it is to switch between providers:

from ai_query import generate_text
from ai_query.providers import openai, anthropic

# Just change the model - everything else stays the same!
async def generate_response(prompt: str, use_openai: bool = True):
    model = openai("gpt-4o") if use_openai else anthropic("claude-sonnet-4-20250514")
    
    result = await generate_text(
        model=model,
        prompt=prompt,
        tools=my_tools  # Same tools work with any provider
    )
    return result.text

Model input capabilities

Declare input modalities when conversation history can contain images or files:

model = workers_ai("@cf/moonshotai/kimi-k2.6")
model.input_modalities = ("text", "image")

Before each provider request, ai-query projects the canonical conversation history into the target model’s capabilities without mutating stored messages:

Native endpoints such as Gemini 3 keep rich tool results inline.
Vision models on chat-compatible endpoints receive tool-result images in a follow-up multimodal user message.
Text-only models receive explicit omission placeholders instead of image bytes.

Leave input_modalities unspecified to retain provider behavior when capabilities are not known. Provider catalogs and applications should set this metadata whenever it is available.

Provider Options

Each provider supports specific options through the provider_options parameter:

result = await generate_text(
    model=openai("gpt-4o"),
    prompt="Be creative!",
    provider_options={
        "openai": {
            "temperature": 0.9,
            "max_tokens": 1000
        }
    }
)

See individual provider pages for available options.

Deterministic Tests with the Faux Provider

The public faux provider runs the normal generation, streaming, tool, and agent loops without network requests or credentials. Queue one response per model step:

from ai_query import Agent, Field, step_count_is, tool
from ai_query.providers import FauxProvider, FauxResponse, faux
from ai_query.types import ToolCall

@tool(description="Read a fixture")
def read_fixture(name: str = Field(description="Fixture name")) -> str:
    return f"fixture:{name}"

model = faux(responses=[
    FauxResponse(
        tool_calls=[ToolCall(
            id="call_1",
            name="read_fixture",
            arguments={"name": "profile"},
        )],
        finish_reason="tool_calls",
    ),
    FauxResponse(text="The profile is ready.", chunks=["The profile ", "is ready."]),
])

agent = Agent(
    "test",
    model=model,
    tools={"read_fixture": read_fixture},
    stop_when=step_count_is(4),
)
result = await agent.run("Load the profile")

provider = model.provider
assert isinstance(provider, FauxProvider)
assert provider.call_count == 2
provider.assert_exhausted()

Use a response factory when the next scripted response should inspect the model, messages, tools, or provider options captured in FauxCall. Queue exhaustion is an error by design, and assert_exhausted() catches unused scripted steps.

OpenAI-Compatible Providers

OpenRouter, DeepSeek, Groq, and Workers AI use OpenAI-compatible APIs internally. This means:

They share the same request/response format
Provider options still use their own provider keys ("openrouter", "deepseek", "groq", "workers_ai")
Tool calling works identically

# All four use the same underlying format
result = await generate_text(
    model=openrouter("openai/gpt-4o"),  # or deepseek(...), groq(...), workers_ai(...)
    prompt="Hello!",
    provider_options={
        "workers_ai": {"temperature": 0.7}
    }
)

Start Here

Foundations

Agents

Deployment

Advanced

Providers Overview

Prompt caching

Supported Providers

OpenAI

Anthropic

Google

OpenRouter

DeepSeek

Groq

Workers AI

Custom Providers

Meta Llama

xAI

Cloud Providers

Bedrock

Quick Start

Usage

Switching Providers

Model input capabilities

Provider Options

Deterministic Tests with the Faux Provider

OpenAI-Compatible Providers

​Prompt caching

​Supported Providers

OpenAI

Anthropic

Google

OpenRouter

DeepSeek

Groq

Workers AI

Custom Providers

Meta Llama

xAI

​Cloud Providers

Bedrock

​Quick Start

​Usage

​Switching Providers

​Model input capabilities

​Provider Options

​Deterministic Tests with the Faux Provider

​OpenAI-Compatible Providers

Prompt caching

Supported Providers

Cloud Providers

Quick Start

Usage

Switching Providers

Model input capabilities

Provider Options

Deterministic Tests with the Faux Provider

OpenAI-Compatible Providers