LiteLLM
LiteLLM routes through the gateway per call via api_base, or fleet-wide by pointing a model at the gateway in the LiteLLM proxy config.yaml.
import os
import litellm
res = litellm.completion(
model="openai/gpt-4.1",
api_base="http://127.0.0.1:8787/w/my-service/openai/v1",
api_key=os.environ["OPENAI_API_KEY"],
messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
# LiteLLM proxy? Point the model at the gateway in config.yaml instead:
# model_list:
# - model_name: gpt-4.1
# litellm_params:
# model: openai/gpt-4.1
# api_base: http://127.0.0.1:8787/w/my-service/openai/v1
# api_key: os.environ/OPENAI_API_KEYWorks per-call via api_base, or fleet-wide via the LiteLLM proxy config.yaml.
Local vs. connected
The base URL above is the local proxy default (http://127.0.0.1:8787). Connected or managed use swaps in the cloud gateway URL and adds x-cave-api-key (gateway auth) plus x-cave-upstream-key (your per-request provider key) as request headers on the same client.