Inference sources

A source is the service that runs your models. The provider agent connects to it, discovers its models, checks what they can do and translates every customer request into the source's own API. One profile serves exactly one source; run several profiles to serve several. For the setup flow, see the Provider quickstart.

Source kinds

Run numes provider source kinds to list the kinds your agent supports. Cloud sources are subject to the upstream's terms.

Addresses. Named cloud connectors always use their official address. Ollama, vLLM, llama.cpp and OpenAI-compatible sources accept any HTTP or HTTPS address, so the source can run on another machine. Don't expose an unauthenticated engine on a public interface: bind it to loopback or a private network, or put an API key in front of it.

Retries. The agent never retries an inference request against the source, which avoids duplicate work and ambiguous charges. Fallback to other providers is handled by the platform.

Usage. Sources must report exact whole-number token counts. Fractional or imprecise counts fail validation.

Local engines

Ollama

Ollama needs no key; models are discovered from the local server.

• Context length. The agent reads each model's trained context length from Ollama's metadata, and uses 16,384 tokens when it's missing. The context you publish is sent to Ollama with every request, so choose a smaller one during review if the machine lacks memory. A context above the model's trained limit can't be published.

• Tools. Automatic tool calling is supported. A named tool_choice fails explicitly; tool_choice: "required" is sent as an automatic choice.

• Structured output stays off until you turn it on for a model, because Ollama's metadata doesn't prove that the model honors JSON schemas. Publication then checks it.

• Reasoning follows the model family's metadata. Some models require a reasoning level and can't turn thinking off.

Ollama doesn't report cache reads or writes separately, so Ollama offers can't have cache prices.

vLLM

Start the server with a stable model name and the features you intend to sell:

vllm serve /models/example-instruct \  --served-model-name example-instruct \  --host 127.0.0.1 --port 8000 \  --enable-prefix-caching --enable-prompt-tokens-details

Then set up the profile:

numes provider setup vllm --profile local-vllm \  --provider-name "Example Labs" \  --model example-instruct \  --currency USD --prompt-price 1 --completion-price 2

• vLLM's model list doesn't show which features the server was started with. Without more metadata the agent assumes only chat, tools and reasoning, and checks each before publishing. To advertise cache prices or other capabilities, describe the model in a catalog overlay.

• Server flags and the published offer must agree. Restarting vLLM with a tool parser doesn't add tools to a published offer. Refresh the source in the terminal UI or desktop app and publish again after changing flags.

• The context is capped by vLLM's max_model_len, and customer tools and system prompts count towards it. A small --max-model-len can make ordinary coding-agent requests fail routing. If the published context later exceeds the server's limit, the model shows Model not served until you raise the limit or publish a smaller context.

• Thinking is turned off unless the customer requests reasoning; an explicit reasoning effort is forwarded unchanged.

• The agent sets a cache_salt so cached prompts aren't shared between customers.

llama.cpp

Start llama-server with its Jinja template engine:

llama-server -m /models/example-instruct.gguf \  --alias example-instruct --host 127.0.0.1 --port 8080 \  --jinja

• --jinja applies the model's chat template, which tool calling depends on.

• The agent turns off llama.cpp's prompt reuse, so llama.cpp offers have no cache prices.

• Reasoning can be turned on or off; there are no effort levels.

• If a reverse proxy adds bearer authentication, choose --auth-mode api_key and use a secret reference for the key.

OpenAI-compatible servers

Use openai-compatible for any server that implements the OpenAI Chat Completions API, including LM Studio. The address is required:

numes provider setup openai-compatible --profile lab \  --source-url http://127.0.0.1:1234/v1 \  --provider-name "Example Labs" --model example-model \  --currency USD --prompt-price 0.5 --completion-price 1.5

• Without model metadata, only tools and reasoning are assumed, then checked before publishing. A supported_parameters list from the server takes precedence, and reported context and output limits are imported.

• Reasoning uses only the controls the server reports. Reported effort levels become selectable in coding clients.

• Vision models can keep image input with token-only billing. The server must report prompt and completion tokens; image usage included in prompt tokens uses your input-token price.

• For LM Studio, the agent also reads LM Studio's native model list to learn the loaded context, vision and tool support.

• New compatible endpoints are declared as collecting data with unknown retention, because the agent can't know what the server does. Correct this during setup with --collects-data and --retention.

• A maximum output length you saved stays in place when the source's reported limit changes. To raise it, edit Maximum output length in the desktop app, or deselect and reselect the model in the terminal UI and review its terms again.

Cloud connectors

OpenRouter

• Discovery reads OpenRouter's live model list, including context, output limits, modalities, supported parameters and exact USD list prices. Every model with a list price gets a reference price.

• Each request requires OpenRouter to pick an upstream that supports its parameters; unsupported controls fail instead of being ignored.

• The agent adds OpenRouter's automatic cache control to requests that have none. Set [backend].prompt_cache_policy = "explicit" to forward only the customer's own cache directives. Automatic caching can incur upstream cache-write charges, and your cache prices must still be discounts.

OpenAI, Anthropic and Google Gemini

• OpenAI uses the Responses API with response storage turned off. Strict function schemas, parallel tool control, reasoning effort and reasoning continuation are supported.

• Anthropic uses the Messages API. Thinking blocks survive tool-call round trips, and cache_control is supported, with automatic cache control as for OpenRouter. Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1 don't accept a forced tool choice: named choices fail, and a required choice is sent as automatic.

• Google Gemini uses GenerateContent. Thought signatures survive tool-call round trips, and required and named tool choices are enforced.

Reasoning controls are advertised only for models whose reasoning behavior is known. Other models still serve ordinary requests.

Kimi Code

K3 models support low, high and max reasoning effort, and thinking can't be turned off. k3 and k3-256k are separate models. Kimi's usage limits surface as rate limits. Reported cache hits are charged at your cache-read price; when Kimi omits cache usage, input is charged at the prompt price.

OpenCode Zen

OpenCode Zen serves models over different APIs. The agent keeps it as one source and picks a dialect per model:

Setup shows API dialect and Capability catalog before connecting. Pick one dialect with --dialect when every model you sell uses it, or use a catalog overlay with per-model dialects.

PPQ.ai

PPQ.ai is chat only. Tools, structured output, reasoning and image input are supported where the model declares them. Private models, automatic routers and hosted tools are excluded. Models with a variable price get no reference price, so you set their prices yourself.

The agent never tops up your PPQ balance. When it runs out, requests fail with payment_required until you fund the account.

DeepInfra

The connector discovers chat, embedding, image, speech and transcription models from DeepInfra's standard service tier; Flex and Priority variants aren't selectable. Native model IDs are case-sensitive. Tool support is read per model from DeepInfra's inventory. Images are returned inline as PNG or JPEG, without references or edits, and transcription returns text.

For models that advertise reasoning, selectable effort levels follow OpenCode's DeepInfra fallbacks. Most get low, medium and high; DeepSeek V4 also gets max, Grok 3 Mini gets low and high, and North Mini Code gets none and high. DeepSeek Chat, Reasoner, R1 and V3, MiniMax, Kimi, K2P, Qwen, Big Pickle and GLM releases other than 5.2 have no selectable efforts. These levels are compatibility assumptions because DeepInfra doesn't enumerate them. Reasoning budgets aren't supported. Refresh model discovery and republish offers after upgrading to update the advertised levels.

Novita AI

The connector discovers chat and text-embedding models. Chat supports streamed reasoning, automatic function calls and structured output where the model supports them; named or required tool choices, parallel tool control and reasoning budgets aren't available. Embeddings accept text batches with float or base64 output.

Venice

The connector discovers chat, embedding, speech and transcription models with their fixed USD list prices. Offline, beta and deprecated models are excluded. Chat can offer image input, automatic function calls, JSON-schema output and model-specific reasoning effort; forced or parallel function calls and JSON-object mode aren't available. The agent turns off Venice's added system prompt and web search. Speech requires a discovered voice, and transcription accepts WAV or MP3.

Encrypted chat. To serve Venice's encrypted text models, create a separate Venice profile and set encrypted_transport = "venice-ecies-v1" under [backend] in its profile file. That profile discovers only text models that advertise encryption and attestation, and publishes them with :e2ee IDs. Function tools and media aren't supported. Read the Venice security boundary before you publish.

Tinfoil

Tinfoil serves end-to-end encrypted text chat. Select Tinfoil in setup, or run:

numes provider setup tinfoil --profile tinfoil \  --api-key-env PROVIDER_AGENT_TINFOIL_API_KEY

• Models are published with :e2ee IDs and Tinfoil's exact list prices as references. The terminal UI and desktop app show a lock after each encrypted model's name.

• Tool calling and reasoning are turned on where Tinfoil reports them. Tool choice, parallel tool calls and JSON output start off; turn one on during review only if the model supports it.

• The agent relays ciphertext to a verified Tinfoil enclave and can't read customer content.

Encrypted requests are billed differently from plaintext ones. Read Encrypted inference before pricing the source.

Models without reviewed facts

For every cloud connector, live discovery decides which models are available. A model the agent has no reviewed facts for is offered as text-only chat without reasoning, image input or hosted tools. Embedding, image, speech and transcription models, and OpenCode Zen models with an unknown dialect, aren't selectable until a catalog overlay describes them.

Limits and capabilities your source doesn't report

Some sources report no context length or maximum output for a model. Setup then fills them from public model catalogues for the model's public name, where your deployment enables the reference catalogue, and labels them from catalogue. Without a catalogue value, the context falls back to a default of 16,384 tokens and the output limit stays undeclared, labelled default. Check these limits in Review; a value your source reports always wins. While choosing a public name, choosing another name updates the suggested limits until you edit a limit yourself. A model without a suggested name gets its limits once you name it. A suggestion never lowers the context below a maximum output your source reports or you set. A maximum output larger than the context length can't be published: Review and the desktop app's Edit terms say which model is affected and what to change before anything is submitted.

Capabilities a catalogue lists but your source doesn't declare, such as JSON schema output or image input, are only hints: Review says the catalogue lists it beside the switch, and nothing is turned on for you. A capability you turn on is checked before publishing like any other.

Resuming setup keeps the distinction between suggestions and limits you edited. Refreshing the catalogue doesn't overwrite your saved limits.

Capability checks

For Ollama, vLLM, llama.cpp and OpenAI-compatible sources, the agent sends short synthetic requests through the same adapter that serves customers before you publish. These must pass:

• chat with complete usage and a finish event;

• JSON object and JSON schema output, when declared;

• automatic and named tool calls, including returning the tool result and receiving a final answer;

• turning reasoning on, and off when the model's reasoning can be switched;

• image input, when declared.

These are also tried, and a failure shows as a note without blocking publication:

• tool_choice: "required", a named choice beside a competing tool, and tool_choice: "none";

• parallel tool calls;

• each declared reasoning effort.

A failed required check blocks publication. Your selection is kept, and the error names the check and a fix. Repair the source, or turn the capability off for that model, and try again. The agent never removes a capability silently.

Cloud connectors don't send paid checks during discovery; their capabilities come from reviewed catalog data or the upstream's metadata. After you publish, the platform verifies every self-hosted endpoint, and those requests use upstream capacity. Hosted sources follow their own rules.

Catalog overlays

A catalog overlay is a JSON file you maintain that describes exact models: their tasks, modalities, limits, supported parameters, dialect and the usage the source reports. Point a profile at it with --catalog-path PATH during setup or [backend].catalog_path in the profile.

Entries are keyed <kind>:<served-model-id>. The model must still be live at the source; an overlay never makes a model available.

{  "vllm:example-instruct": {    "family": "example",    "usageMeter": "provider-native-v1",    "quantization": "fp16",    "contextLength": "32768",    "maximumOutputLength": "8192",    "tasks": ["chat"],    "inputModalities": ["text"],    "outputModalities": ["text"],    "supportedParameters": ["max_tokens", "temperature", "top_p", "tools", "tool_choice"],    "hostedTools": [],    "usageEvidence": [      { "meter": "prompt_tokens", "unit": "token", "provenance": "provider", "required": true },      { "meter": "completion_tokens", "unit": "token", "provenance": "provider", "required": true },      { "meter": "requests", "unit": "request", "provenance": "provider", "required": true },      { "meter": "cache_read_tokens", "unit": "token", "provenance": "provider", "required": true },      { "meter": "cache_write_tokens", "unit": "token", "provenance": "provider", "required": true }    ],    "dialect": "compatible"  }}

• family may be empty when unknown. Declare quantization accurately. Use the usageMeter value shown in the example.

• Declare a feature only after confirming that the exact model, template, engine flags and engine version support it. Publication and platform verification test every declared capability.

• Include cache meters only when the server reports cache usage exactly. A required counter that's missing fails the response.

• For Anthropic and Gemini models, an entry may set reasoningPolicy to anthropic-budget, anthropic-hybrid, anthropic-adaptive, anthropic-mandatory, gemini-budget-pro, gemini-budget-flash, gemini-level-pro or gemini-level-flash. Choose only a policy the model supports.

Unknown fields, duplicate entries and unsupported values are rejected when the file loads.

Keys and secret references

The agent never accepts an API key as a command-line argument. Supply it in one of three ways:

With env://, the variable must be set wherever the agent runs, including under a service manager.

Changing a source's kind, address or key with source configure, or editing the connection in the terminal UI, clears the model selection and reference prices so they're discovered again. See Profiles and configuration for the reference formats.

Internal synthetic sources

Synthetic Mock (mock) and Synthetic E2EE Mock (mock-e2ee) are reserved for internal testing. They are unavailable in Desktop and terminal setup. Direct API publication also requires internal authorization for the exact endpoint and source; account roles and administrator status do not grant it.

Existing authorized internal endpoints and profiles remain usable. Synthetic sources provide test responses, not model inference.

Next steps