Serving inference

Operations guide: configure, deploy and observe the gateway. What the capability does, product side: product guide; calling it: developer guide.

Goal: serve the OpenAI-compatible surface (/v1/chat/completions, /v1/models, /v1/embeddings since 1.5.0) through the same data plane as the API - same identity, same rate limits, same policies, same trace -, routed by policy to backends, under a per-tenant token budget.

Command - an ai: section. It needs no contract: the gateway declares its operations itself (ai.chatCompletions, ai.listModels, ai.embeddings), served on every host.

ai:
  security:                          # the identities accepted, one suffices - as `security` in a contract
    - oidc: { scopes: [inference] }
    - api_key: {}                    # a key bound to ai.chatCompletions (see API keys)
  sensitive_header: X-Sensitive      # "true" pins the request to sensitive_backend
  default_backend: eu-provider
  sensitive_backend: on-prem         # internal; unreachable, or no internal backend at all: refused, never an external one
  backends:
    eu-provider:
      url: https://api.provider.example/v1
      api_key: "file:/etc/flexway/secrets/provider-key"   # a reference, read at each call
      default_model: mistral-small-3.2-24b                # the gateway chooses the model
      external: true
    on-prem:
      url: http://vllm.inference.svc:8000/v1
      default_model: mistral-7b-instruct
  budget:
    store: "${COUNTERS_URL}"         # redis://..., shared by every replica
    monthly_tokens: 40000000         # per tenant, 0 for none
    daily_tokens: 2000000
  overrides:                         # the first match wins
    - { match: { tenant: acme, agent: batch-summaries }, backend: on-prem }

Four possible identities, each an alternative of ai.security:

Alternative	The caller presents	Tenant
trusted_headers	The headers a mesh ingress wrote (auth.trusted_headers), believed from a peer of server.trusted_proxies only	The tenant header; subject names the agent
oidc	A token of one of the issuers, with the declared scopes or roles	The tenant_claim claim
api_key	Authorization: ApiKey <key>, the key bound to ai.chatCompletions	The key's
mtls	A declared client certificate, bound to the operation	The certificate's

Behind a mesh ingress that already authenticated the caller (the header names are the deployment's, never the product's):

server:
  trusted_proxies: [10.42.0.0/16]    # the ingress, and only it: from anyone else, the header is a forgery
auth:
  trusted_headers: { tenant: X-Mesh-Tenant, subject: X-Mesh-Agent }
ai:
  security: [ { trusted_headers: {} } ]

Cost, soft cap and fallback

Each backend carries its price per million tokens; the budget in euros adds to the token budget - the first reached decides (full example: Cost, soft cap and fallback):

ai:
  backends:
    eu-large:
      # ...
      price: { input_per_mtok: 2.00, output_per_mtok: 6.00 }   # euros per million tokens
      fallback: [eu-small, on-prem]   # in order, before any byte reaches the client
  budget:
    monthly_eur: 50                   # per tenant, over the calendar month
    soft: { at: 0.8, backend: eu-small }
  overrides:
    - { match: { tenant: acme }, monthly_eur: 500 }

Mechanism	Behaviour
Cost	Pre-booked (input estimated at body size / 4, output at max_tokens), reconciled on the real usage at the price of the backend that served; a call that fails or is refused gives its booking back, tokens and euros; carried by the audit and the flexway.inference.cost metric (euros, by tenant and backend)
Soft cap	Past at x budget: a Flexway-Budget: soft header, a metric, a log line; the call goes to soft.backend - except a sensitive request, which stays pinned
Hard cap	A pre-booking that would pass the euro (or token) budget: 429 budget_exceeded, before any call
Fallback	On connection failure, 5xx, timeout or open circuit: the next of fallback, each backend once; never on a 4xx (the provider's 429 included), nor after a stream's first byte, nor to an external backend for a sensitive request. The answer is marked Flexway-Degraded: backend-fallback

A price is required for every backend once a budget in euros exists: a cost nobody can compute cannot be budgeted. 0 is a price - an internal backend's.

Check:

Call	Response
An accepted identity, budget left	relayed as the backend wrote it; the backend gets the client's body with the policy's model, and its own key - never the caller's identity
stream: true	events arrive as they are produced; the budget is reconciled on the final usage
No identity, or an identity without a tenant	401 tenant_required
The tenant's budget spent (completion or stream)	429 budget_exceeded, before any backend call
X-Sensitive: true with no internal backend	403 sensitive_no_internal_backend - never an external backend
Backend down, no fallback left	502 backend_unavailable; during a stream, a last event {"error":{"type":"backend_unavailable"}}
A gateway refusal on /v1/* (scope, rate, unknown route)	OpenAI envelope, OpenAI's type (permission_error, rate_limit_error...) and the gateway's code
A tenant past soft.at of its euro budget	Flexway-Budget: soft; served by soft.backend
A backend down, with fallback	served by the next one, Flexway-Degraded: backend-fallback
A policy naming an undeclared backend	refused at load, the key named

Rate limits: /v1/* follows the policy chain like any route - a rate-limit policy of its own, attached to routes: { "/v1/**": [...] }, bounds it apart (example 12.17). Behind an ingress, the bucket is the tenant and agent it vouches for, never the ingress's address.

Migrating from config/policy.yaml - no longer read since 0.1.0: without FLEXWAY_CONFIG, the gateway refuses to start. The mapping, to apply once by hand:

policy.yaml	flexway.yaml
default_backend, backends.<n>.default_model, external	ai.default_backend, ai.backends.<n>.*
backends.<n>.base_url_env	ai.backends.<n>.url - the URL itself
backends.<n>.api_key_env: VAR	ai.backends.<n>.api_key: "${VAR}" or file:/...
pii_guard.sensitive_backend	ai.sensitive_backend
budget.*_tokens_default	ai.budget.*_tokens; REDIS_URL -> ai.budget.store
overrides	ai.overrides, same shape
Headers injected by the ingress	auth.trusted_headers + server.trusted_proxies + ai.security: [ { trusted_headers: {} } ]

Several backends: images, embeddings, relays and specialised models

Since 1.5.0, a backend declares where it is (kind), what it serves (capabilities), the models a caller may name (models), how the gateway authenticates (auth) and who meters the cost (metering). A configuration written before stays valid as it is: kind: openai, capabilities: [chat], api_key is auth: { scheme: bearer }, metering: local.

The three kinds:

ai:
  default_backend: platform
  sensitive_backend: local-vision
  backends:
    platform:                          # another Flexway: it meters, it caps
      kind: relay
      url: https://llm.platform.example/v1
      auth: { scheme: token_file, path: /var/run/flexway/peer-token }   # read again at each call
      identity_headers: { tenant: X-Tenant, agent: X-Agent, sensitive: X-Sensitive }
      metering: upstream
      external: false                  # the operator declares it: sensitive data may go there
      capabilities: [chat, vision, embeddings]
    provider-direct:                   # a provider called directly: the key lives here
      kind: openai
      url: https://api.provider.example/v1
      auth: { scheme: header, header: X-Api-Key, key: "${PROVIDER_KEY}" }
      external: true
      capabilities: [chat, vision, embeddings]
      models: [model-a, model-vision-b, embed-c]   # served when asked for, else default_model
      default_model: model-a
      price: { input_per_mtok: 0.15, output_per_mtok: 0.35 }
    local-vision:                      # a model of the cluster
      url: http://vision.models.svc:8080/v1
      default_model: vision-small
      auth: { scheme: none }
      capabilities: [vision]           # an image; text too
    faces:                             # a specialised model: the upstream of a gRPC contract
      kind: grpc
      upstream: models                 # declared under upstreams:, protocol: grpc
      capabilities: [vision]
  overrides:
    - { match: { agent: screen-reader }, backend: provider-direct }
    - { match: { capability: embeddings }, backend: provider-direct }   # routes only, never a budget
  vision:
    max_image_bytes: 8MiB              # one image, decoded (the default)
    allowed_origins: ["https://cdn.example"]   # otherwise inline images (data:) only

Key	Values	Effect
kind	openai (default), relay, grpc	An OpenAI-compatible API; another Flexway; the upstream of a gRPC specialised model, never a target of /v1/*
capabilities	chat (default), vision, embeddings; model (grpc)	An image only goes to a vision backend, embeddings only to an embeddings one, else 400 capability_unavailable
models	list	The model asked for is served when listed, else default_model; the audit keeps the model served. A relay without default_model forwards the model asked for
auth.scheme	none, bearer (key), header (header, key), token_file (path)	The key is a reference (${VAR}, file:), read again at each call; token_file reads a short-lived token again (workload identity)
metering	local (default), upstream	See below
external	boolean	A sensitive request never goes there; a relay is external unless external: false
identity_headers	tenant (required), agent, sensitive	The names a relay receives the call's identity under - those the other Flexway declares in auth.trusted_headers and ai.sensitive_header
tls	as an upstream's	Mutual TLS toward a relay
match.capability (override)	chat, vision, embeddings	Chooses the backend of the calls of that capability, after the tenant and agent override; never a budget

Metering - who counts the cost:

metering	Tokens	Euros	Upstream refusal	Series
local	Booked, reconciled on the usage	At the declared price	A provider's 4xx: 502 backend_unavailable	metered_by="local"
upstream	Local cap always applied, on the usage the upstream returns	At the declared price if any; without one, the upstream is authoritative	Relayed as it came, 429 budget_exceeded included	metered_by="upstream"

A consumption view sums metered_by!="upstream" only - never ="local": a series from before 1.5.0 has no such label -, a relayed call being counted local by the Flexway that pays for it. The console shows the upstream's share beside, under "Metered upstream" (Reading with the console). flexway validate warns of a relay metered locally.

Platform side - the Flexway receiving the relay believes the identity headers from this peer only:

server:
  trusted_proxies: [10.42.0.0/16]      # the products' Flexway
auth:
  trusted_headers: { tenant: X-Tenant, subject: X-Agent }
ai:
  security: [ { trusted_headers: {} } ]
  sensitive_header: X-Sensitive

Image and embeddings:

# an inline image, to a vision backend
curl -s http://localhost:8080/v1/chat/completions -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"model-vision-b","messages":[{"role":"user","content":[{"type":"text","text":"What does this screen show?"},{"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw0..."}}]}]}'
# embeddings
curl -s http://localhost:8080/v1/embeddings -H "Authorization: Bearer $TOKEN" -d '{"model":"embed-c","input":["one","two"]}'

gRPC specialised model - a method of the product's contract, metered by the ai option:

rpc Classify(Image) returns (Labels) {
  option (flexway.v1.operation) = { ai: { unit: "call", price_per_1k: 0.02, capability: "vision" } };
}

Each call books its price in the tenant's euro budget (ai.budget.monthly_eur and the overrides), before the upstream; unit: byte counts the bytes of the request and the answer as they cross. An upstream that is down costs nothing.

Check:

Call	Answer
Image to a vision backend	served; the body as written, the model admitted
Image with no vision backend chosen, or embeddings with no embeddings backend	400 capability_unavailable, before any backend
Image past max_image_bytes	413 payload_too_large
Image by URL from an undeclared origin, an input_audio part	400 invalid_request
Relay: tenant past the platform's budget	the platform's 429 budget_exceeded, relayed as it came
Relay: tenant past the local cap	429 budget_exceeded, the platform never called
Metered gRPC method past the budget	RESOURCE_EXHAUSTED, reason budget_exceeded; the upstream never called
Metered gRPC method with no grpc backend naming its upstream	refused at load

Available since 0.1.0; several backends since 1.5.0.