Model fallbacks

​

How it works

• A primary model: the provider and model that trigger the chain when a request fails.
• One to five fallbacks: an ordered list of direct provider models or saved model configurations.
• Triggers: the upstream HTTP status codes that move the request to the next model. For example, use 429 for rate limits, or 500, 502, 503, and 504 for provider errors.

1. Calls the primary model selected by the request’s provider-prefixed model ID.
2. If the request fails with a configured trigger status or a transport error, loads the matching fallback chain.
3. Calls each fallback in order until one succeeds, returns a status that does not trigger another fallback, or the chain is exhausted.
4. Returns the final response in the API format used by the client.

​

Create a fallback chain

1. Go to LLM Gateway and select the Model Fallbacks tab.
2. Click Create fallback chain.
3. Select the Workspace where the chain applies.
4. Select the primary provider and model. Requests to this provider-prefixed model ID use the chain when the primary attempt fails.
5. Under Fallbacks, add one to five backup models in the order the gateway should try them. Choose a provider and model directly, select an existing model configuration, or create a custom model configuration.
6. Under Configure fallback triggers (advanced), review the HTTP status codes that should trigger the next fallback. Add or remove status codes as needed.
7. Click Create chain.

​

Make a call

curl https://gateway.smith.langchain.com/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"anthropic/claude-opus-5","messages":[{"role":"user","content":"Hello!"}]}'

curl https://<data_plane_host>/gateway/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"anthropic/claude-opus-5","messages":[{"role":"user","content":"Hello!"}]}'

​

Set fallbacks for a prompt

1. Create a model configuration pointing to the provider and model you want the prompt to use.
2. Create a fallback chain in the same workspace. Select your saved configuration as the primary model, rather than selecting its underlying provider and model directly. Add the backup models, configure the triggers, and save the chain.
3. Create a prompt in the Playground. Open Model Configuration, select LangSmith Gateway as the Provider, and enter custom/<my_config_name> in the Model field. Replace <my_config_name> with the saved configuration’s name, without angle brackets. You can type the value even if it is not listed. Click Apply.
4. Save the prompt, then pull it with its model. In Python, set include_model=True when calling client.pull_prompt so the saved Gateway model configuration is included.

​

Choose fallback candidates

• Direct provider model: select a supported gateway provider and model. This option uses the workspace’s secret for that provider, or Gateway Credits for eligible hosted models.
• Model configuration: select a saved model configuration. Use this option for a custom OpenAI-compatible or Anthropic endpoint, a custom model name, or configuration-specific parameters.

​

See also

• API formats: review supported request formats and translation behavior.
• Spend policies: apply cost limits alongside fallback routing.