Troubleshooting

Operations guide: configure, deploy and observe the gateway. What the capability does, product side: product guide; calling it: developer guide.

Symptom	Likely cause	What to do
Not sure which version runs	-	The console's footer (sign-in page included) shows both; or flexway version (docker run --rm ghcr.io/flexwiz/flexway:<version> version), build in the admin /config. dev means a build that is not a release: its commit says which one
flexway validate: x-flexway-sunset ... is set on an operation that is not deprecated	A withdrawal date on an operation (or inherited from info) that is not deprecated	Add deprecated: true (in gRPC option deprecated = true;), or remove the date
Start refused: kms.key "...": kms key absent or unusable	The field-encryption key does not exist in the KMS (a typo) or cannot wrap (a Vault key not derived)	Fix kms.key or create the key - Field encryption
Kafka consumers get CORRUPT_MESSAGE after kms.key changed	Records written before name the former key	Declare it in kms.previous_keys - Field encryption
Start refused: reads the schema registry's wire format and is isolated by the tenant attribute alone	A consumption of a channel in the registry's wire format filtered on the tenant attribute, without a binding: an ordinary Kafka producer's messages have no attribute	Bind the channel to the tenant (x-flexway-bind: tenant, address {tenant}.<topic>), or declare it public (x-flexway-public: true)
POST on an x-flexway-publish door: 504 upstream_timeout	The broker did not acknowledge within the budget: unknown outcome, the message may have been written	Send again with the same Idempotency-Key; check the broker's health, widen timeout
503 overloaded on a publication	The broker's max_in_flight buffer is full: it acknowledges slower than callers publish	Read flexway_async_publish_duration_seconds; raise max_in_flight or the broker's capacity
502 upstream_error on a publication	The broker refused: ACL, missing topic (the gateway never creates one), message rejected; AMQP: no queue takes the message	Create the tenant's topic, grant the gateway's principal write access, bind a queue to the exchange
Start refused: is consumed and isolates no tenant	A consumption on a channel neither bound to the tenant nor filtered on the tenant header	x-flexway-bind: tenant on a channel parameter, or the tenant header in every message's headers
A WebSocket subscriber receives a message twice	At least once: no acknowledgement within ack_timeout, or a cut before it	Acknowledge each id ({"ack": "<id>"}); deduplicate on source + id
flexway validate fails on "operation without upstream"	An operation was added to the contract without x-flexway-upstream, and there is no default upstream	Add the extension, or declare a default upstream
flexway import krakend exits with 3	Elements of the imported configuration have no equivalent; the rest is written	Read the "No equivalent" section of coverage.md: each line says why and, when there is one, the equivalent to write by hand
flexway import krakend: the converted configuration does not load - nothing written	The result would not start: a contract operation no endpoint covers, an aggregation, sd: dns, a validator without audience, a policy the contract names and nothing defines	The loader's message names the operation or the key; set x-flexway-upstream in the contract or fix it, then run again
flexway import krakend: is not empty: the import never overwrites	--out already holds files	An empty or absent directory
An endpoint of the imported configuration is missing after the import	The contract does not declare it: it is listed in coverage.md, never invented	Declare it in the contract, run the import again
The gateway refuses to start	Partial configuration - this is deliberate	Read the message: it names the offending operation or upstream
404 on a route you believe is exposed	It is not in the contract. "Served = declared"	Add it to the contract, not to the configuration
A stream arrives all at once at the end	Something in the path is buffering - not the gateway	Check the edge and any proxy between the client and it
A stream stops midway, log read budget exhausted	The upstream went silent longer than server.timeouts.read	Have the upstream emit a heartbeat (: keepalive), or raise read
504 upstream_timeout	The upstream did not answer within the operation's budget	Read flexway_upstream_duration_seconds; raise x-flexway-timeout on that operation, not the global default
Flexway-Cache: miss on every request	The upstream forbids keeping it (no-store, no-cache, private, Set-Cookie), answers without Content-Length, past server.cache.max_entry_bytes, or with a status other than 200, 203, 204, 301, 404	Read the upstream's response headers; flexway_cache_lookups_total per operation
No Flexway-Cache at all on a cached operation	A bypass: an identified caller on a public cache, a when condition that does not hold, a method other than GET/HEAD	result="bypass" in flexway_cache_lookups_total; a per-caller cache is declared scope: subject on the operation
Start refused: response cache declaration(s) refused	A cache on a stream, a WebSocket, gRPC, inference, a write, or a public one on an authenticated operation	The message names the operation and the door (x-flexway-cache or the policy)
502 response_too_large	An operation with a body policy received an answer past server.body.max_bytes	Raise server.body.max_bytes (reloadable), or page on the upstream's side
502 upstream_error on a transformed operation, log an upstream answer could not be transformed	The upstream answered something other than readable JSON, compressed despite Accept-Encoding: identity, or without a field projected with on_missing: fail	The log names the cause and the field; fix the upstream, or switch to on_missing: omit
Start refused: body policy attachment(s) refused	A body policy reaches a stream, a non-JSON response, gRPC or inference; or it drops a required field, keeps an undeclared one, masks with a refused type	The message names the operation and the policy; attach it more narrowly, or !name
flexway_policy_nonconformant_total rises	A served body no longer holds its status's schema (sampled) - the upstream changed, or the status is not declared	The departs from its schema log gives the field and the keyword; the response is served all the same
Load refused "body is read by the conditions of transform-body, redact and map-status only"	Another policy reads body in its condition or a template	Read the body in a body policy, or do without it
403 (or another 4xx) with the code of an external-call policy	The external service refused: deny.when holds on its answer	Expected; the reason is the service's, never copied into the problem
504, 503 or 502 without the upstream being called, log external call failed_closed	The external service did not answer in time, is unreachable, has its circuit open or gave an answer deny.when / set_headers cannot read - and the policy is fail-closed	The log names the policy and the cause; flexway_policy_external_calls_total{outcome="failed_closed"}; fix the service, or fail-open if the decision may be missing
A header expected from an external-call is missing, log external call at WARN	fail-open absorbed a failure of the service	flexway_policy_external_calls_total{outcome="failed_open"}; the log gives the cause
504 upstream_timeout "budget was spent before its upstream could be called"	A request-phase external-call consumed the operation's whole budget	Shorten its timeout, or raise the operation's x-flexway-timeout
Start refused: external-call attachment(s) refused	A timeout past an operation's budget, send.body on a stream, the response phase on a WebSocket, gRPC or inference	The message names the operation and the policy
503 upstream_unavailable with Retry-After	The upstream refuses connections: it is down	Check the upstream; the gateway already retried once if the method allowed it
502 upstream_error	The upstream answered something that is not usable HTTP	Look upstream, not at the gateway
401 on valid tokens right after a start	The issuer's keys are not loaded yet	The log says issuer not reachable yet; check the gateway reaches the discovery URL
401 invalid_token on a token that works elsewhere	Audience, issuer or algorithm: the reason is in the log (token refused), never in the response	Compare the token's aud and iss with auth.issuers
Inference: 401 tenant_required despite the ingress's headers	The peer is not in server.trusted_proxies: its headers are not believed	Declare the ingress's range, and only it
Inference: 429 budget_exceeded	The tenant's budget (day or month) is spent - a request pre-books max_tokens, 512 without it	Raise ai.budget or the tenant's override; the counter is in ai.budget.store
Inference: 429 rate_limited for every tenant behind the ingress	A rate-limit policy too tight on /v1/*	A policy of its own on routes: { "/v1/**": [...] } (example 12.17)
Inference: Flexway-Budget: soft on every answer of a tenant	It passed soft.at of its monthly euro budget	Expected; raise the tenant's monthly_eur (override) or wait for next month
Inference: Flexway-Degraded: backend-fallback	The chosen backend is down (connection, 5xx, timeout, open circuit); the next of fallback served	The flexway.inference.fallback metric gives the backend left and the cause
Refused at load "price is required: the budget is counted in euros"	A euro budget exists and a backend has no price	Declare price on every backend, 0 for an internal one
Inference: 403 sensitive_no_internal_backend	A sensitive request and no internal ai.sensitive_backend	Declare an internal backend; never an external one for that flow
Inference: 400 capability_unavailable	The backend the policy chooses does not serve what the request needs - an image with no vision, embeddings with no embeddings	Declare the capability on the backend (capabilities) or a match.capability override toward a backend that serves it
Warning "is a relay metered locally" (flexway validate, start)	A kind: relay without metering: upstream: the upstream Flexway counts the cost too	Declare metering: upstream; the local cap still applies
Inference: 502 backend_unavailable	Backend unreachable, or its key not found (${VAR} unset, file missing)	The log (inference backend failed) gives the cause; the key is read at each call
Start refused: FLEXWAY_CONFIG is not set	No flexway.yaml named; the old config/policy.yaml is no longer read since 0.9.0	Point FLEXWAY_CONFIG at a flexway.yaml; convert the old file (Serving inference)
helm install: Flexway serves what a contract declares, and none was given	Neither openapi nor config in the values	The command it prints (Moving to Kubernetes)
403 insufficient_scope	The token lacks the scope or role the contract requires	Read scope= in WWW-Authenticate; for a role, check roles_claim
A2A: 200 with JSON-RPC error -32009 (VERSION_NOT_SUPPORTED)	No A2A-Version header - version 0.3 for the specification	Send A2A-Version: 1.0 (or ?A2A-Version=1.0); only 1.0 is served
A2A: 403 PUSH_TARGET_NOT_ALLOWED	A push target outside agents.a2a.push.allowed_hosts, over http, written as an address, with user@, or no list declared	Declare the webhook host's name; flexway_a2a_push_refused_total gives the reason
A2A: 403 TENANT_MISMATCH	The request's tenant (body, path prefix, parameter) is not the identity's, or the identity holds none	Send the identity's tenant, or none: the gateway forwards its own
A2A: 429 TOO_MANY_STREAMS	The caller already holds max_streams open streams to this agent on this replica	Close a stream, or raise the agent's max_streams
An A2A stream ends at a fixed time	max_task_duration reached (flexway_a2a_calls_total{outcome="duration_exceeded"})	Raise it on the agent, or follow the task with GetTask
Start refused on an A2A card	gRPC binding, a scheme with no mechanism, an upstream without token_exchange or not protocol: a2a, declared signatures, signer (planned for 1.9.0), no verify: none, streaming without max_streams or max_task_duration	The message names the card, the agent and the rule
401 WWW-Authenticate: ApiKey on a key believed good	Unknown key (removed, mistyped), expired, or without the fwk_ prefix	The log (API key refused) gives the cause and, for an expired key, its id; create a new one
401 "does not accept API keys"	The operation has no apiKey alternative in the contract	Add { key: [] } to its security, if intended
403 "not bound to this operation" (key or certificate)	The entry does not name this operation	Add the operationId to the entry's operations; reload
Refused at load "no contract declares" on a key or a certificate	The entry names an operation that does not exist (renamed?)	Fix the operationId
401 "requires a client certificate" despite a certificate	Undeclared certificate, or x-forwarded-client-cert from a peer outside server.trusted_proxies	Check the URI SAN or subject against auth.mtls.clients; declare the edge's range
TLS handshake refused on :8443	A client certificate from another authority than client_ca_file	Add the authority to the file: it is re-read without a restart
Refused at load "google.api.http ... not supported at this version"	A variable spanning segments, a wildcard or a verb in a REST rule	One variable per segment (/v1/shelves/{shelf}/books/{book}), or the method in gRPC only
Refused at load "is a gRPC method and relays to upstream ... which does not declare protocol: grpc"	The method falls on an HTTP upstream (the default)	Name its upstream in flexway.v1.operation.upstream, declared protocol: grpc
gRPC client: UNIMPLEMENTED	The method is not in the served set - a stale set	Rebuild the .binpb: it is re-read without a restart
REST: upstream_status	The gRPC upstream answered an error: its answer, not a gateway failure	Read the problem's grpc.status and grpc.details
400 "Query parameter ... is not declared"	Closed by default: the contract does not declare that parameter	Declare it in the contract, rather than going around the gateway
429 rate_limited	The caller's bucket is empty	Read Retry-After; raise the class's rps/burst, or change the operation's class
All anonymous callers seem to share one bucket	The gateway sees the edge's address - or, behind a cloud load balancer without the proxy protocol, the load balancer's - not the client's	Declare the edge in server.trusted_proxies; read x-forwarded-for in the edge's access log, and turn the proxy protocol on at the load balancer and the edge (guide Setting quotas)
422 invalid_request	The request breaks the contract's schema	The errors extension names the field and the rule
503 with Flexway-Degraded: circuit-open	The upstream is failing; the breaker answers for it	Fix the upstream; one trial call goes through after cooldown
503 with Flexway-Degraded: upstream-saturated	max_in_flight reached on this upstream	Raise the cap, or find why the upstream slows down
503 overloaded with Flexway-Degraded: shedding, flexway_shed_total{reason="in_flight"} rising	The replica is at its in-flight limit (server.in_flight, derived from its memory limit unless written): it refuses the excess instead of being OOM-killed	Add replicas, or give each more CPU and memory; the limit in force is in /config (server.in_flight) and in the start log
A header does not reach the upstream	It is not declared in the contract - it is dropped, not refused	Declare it as an in: header parameter
403 "This origin is not allowed"	The browser's origin is in no cors policy	Add it to allow_origins
Quota counters disagree across replicas	No shared store	Declare a Redis, Dragonfly or Valkey
A change is not taken	The new version is refused: the previous one still serves	Read last_reload.cause in /config; in a cluster, check the ConfigMap is not mounted with subPath
Reload refused, "restart required"	server.shutdown, observability or server.reload changed	Restart: it is the only way to apply them
404 on a host that declares contracts	That host serves exactly its contracts, nothing of the default one	Add the contract to hosts.<host>.contracts
Agent: 401 with a valid token	The token is not issued for the surface: its audience must be the URL in agents.mcp.resources	Request the token with the resource indicator (resource=https://…/mcp)
An expected tool is missing from tools/list	The agent's token lacks the scope or role the operation requires - the list is filtered	Give the agent the scope, rather than widening the operation
A tool answers 401 from the upstream	The upstream verifies a JWT itself, and the agent's token is never relayed to it	Declare upstreams.<name>.token_exchange and the issuer's client (The token the upstream receives, Token exchanged toward the upstream)
A federated server's tool does not appear	Server unreachable or not discovered yet; tool outside the declared list; a missing scope; a name already taken	/config, mcp_federation: state, tools served, not_served and why; the log says MCP server not reachable
Refused at load, "is offered twice … rename one"	A federated tool has the name of a contract tool or of another server's	agents.mcp.servers.<name>.rename
A federated tool answers forbidden	The caller lacks the declared scopes (or tool_scopes), or is not authenticated	Give the agent the scope
403 "The chain of agents … is longer than allowed" or "older than allowed"	The token names more agents (act) than max_depth, or the human authenticated more than max_age ago - an operation may tighten	The log (delegation chain refused) gives the reason and the agents; have the human authenticate again, or shorten the chain
401 "The delegation was revoked"	The issuer says the token is no longer active (introspection)	Expected after a revocation; obtain a new consent
503 "The delegation cannot be verified now"	The issuer's introspection cannot be reached, or it publishes none	Check introspection_endpoint in its discovery and the client; a direct token still passes
403 on an exchanging upstream, log widened	The provider granted a scope the caller did not have	Fix the provider's exchange policy: a hop must never widen
Discovery fails with 401	The server needs a token even to list	discovery.issuer, with a client on that issuer
403 "The identity provider refused to issue a token for this upstream"	The provider refuses the exchange: a client not allowed to exchange, an unknown audience, a wrong secret	The log (token exchange refused) gives the provider's code; fix the client at the provider
503 "could not issue a token", with Retry-After	Provider unreachable, or the client secret not found (${VAR} unset, file missing)	The log (token exchange unavailable) says which; the secret is read at each exchange, no reload needed
Refused, "operationId … is declared by both"	Two contracts give two operations one name	Rename one: the id keys policies, metrics and /config
Start refused, "is protected - the gateway alone writes it"	A policy writes a header only the gateway sets (identity, tracing, tenant, X-Forwarded-*)	Pick another header name
The upstream receives X-User-Roles: [seller admin]	Gateway older than 1.6.0: a list rendered the way Go prints it	Upgrade: a list arrives as seller,admin; {{ auth.claims.roles.join(' ') }} for another delimiter (Enriching a flow)
Start refused, undeclared reference to 'x' or found no matching overload in a value	A function the language does not have, or join with a delimiter that is not text	Only CEL's functions and join exist; fix the expression
Refused, "segment … has two parameters side by side" or "a '{' that is never closed"	A path segment the router cannot read	Put text between the two parameters, close the brace
Refused, "segments … rank the same and can match the same text"	Two mixed segments of one rank read the same text (a{x} and {x}a)	Make one more literal, or rename a path
404 on /reports/2026.pdf while /reports/{year}.pdf is in the contract	Gateway older than 1.6.0: the mixed segment was taken for a literal	Upgrade (Serving your first API)
First request slow after a start, on a large contract in validation: params	Gateway older than 1.6.0: the parameters validator was built on its first request, after /readyz	Upgrade: both validators are compiled before /readyz; the first request to an upstream still opens its connection
Start refused, "on_missing is required"	A value is a {{ … }} expression with nothing said about a missing value	Add on_missing: skip, empty or fail
Start refused, "two … policies at the same level"	Two cors (or rate-limit, rewrite-path) at one level	Keep one, or move the other to a more precise level
500 internal_error, log a policy value is missing	on_missing: fail and the value is missing for this caller	Add a when condition, or switch to skip
A policy does nothing	The start log says NOT applied: its type belongs to a later version	Read effective_chains in /config

---

On languages. This documentation exists in English and in French, page for page; both are kept in step. The specifications, ADRs and the roadmap are written in French - they are the engineering record. The product itself, its configuration keys, its error codes and its logs are in English.