Observing

Operations guide: configure, deploy and observe the gateway. What the capability does, product side: product guide; calling it: developer guide.

Goal: see traffic per operation, not per raw URL.

Command

curl -s localhost:9090/metrics | grep '^flexway_'     # the admin port, never the traffic one

observability:
  otlp_endpoint: tempo.observ:4317    # optional, host:port or URL (http://, https://); otherwise OTEL_EXPORTER_OTLP_ENDPOINT, otherwise nothing is pushed

Traces and metrics go to that endpoint. An endpoint that takes traces only (Tempo, Jaeger), or a collector that already scrapes /metrics, wants OTEL_METRICS_EXPORTER=none: the metrics then stay on /metrics, pushed nowhere.

Every replica names itself in the resource it pushes: service.instance.id is POD_NAME (set it from the Downward API, metadata.name) or else the hostname, which is the pod name in Kubernetes. A backend that turns OTLP into Prometheus series makes it the instance label; without it two replicas would write the same series and the counters would jump between them. OTEL_RESOURCE_ATTRIBUTES is read last and may add attributes or replace this one.

Metric	Answers
flexway_requests_total	How many, by operation, method, status_class, upstream, tenant (bounded)
flexway_request_duration_seconds	How long, streams excluded - an SSE's length would drown the latency
flexway_upstream_duration_seconds	How much of that time is the upstream's
flexway_requests_in_flight	What is in flight, per upstream - the scaling signal
flexway_retries_total	How many retries, by upstream and reason (connection, status_502, status_503)
flexway_inference_tokens_total	How many tokens each tenant uses, by backend and direction (input, output) - as the backend reports them; by agent since 1.2.0: an agent an ai override declares, _other for any other, none without one - never a subject; by capability and metered_by since 1.5.0 - a consumption sums metered_by!="upstream" only (a series from before 1.5.0 has no such label)
flexway_inference_calls_total	How many inference calls were served - completions, embeddings, metered gRPC methods -, by tenant, agent, client, backend, capability and metered_by
flexway_inference_cost_EUR_total, flexway_inference_fallback_total	How many euros, by tenant, agent and the backend that served; how many calls moved to a fallback backend, by from, to, cause and, since 1.2.0, tenant
flexway_quota_requests_total	How many requests each quota policy counted for each tenant - what it consumed of its quota, per replica; flexway_quota_exceeded_total counts the refusals, without a tenant
flexway_build_info	How many replicas, and which version each serves (version, commit) - during a rolling update
flexway_streams_open	How many streams are open, by operation and protocol (sse, websocket, grpc, http) - never in the latency
flexway_server_in_flight, flexway_server_in_flight_limit, flexway_server_shedding	The replica against its in-flight ceiling (declared or derived from its memory), and whether it protects itself (1) - shedder, or adaptive limit below its ceiling
flexway_config_last_reload_successful	1 while the replica's last reload was applied (or none since its start), 0 when it was rejected: the console says it “Watch” while it serves a configuration behind
flexway_server_draining	1 as soon as the replica drains to stop, carried by its last push: the console says it “Retired”, not silent
flexway_circuit_state, flexway_circuit_retry_at_seconds, flexway_circuit_transitions_total	Each upstream's breaker state, read at collection (never stale without traffic), its next trial, its transitions (from, to) - a cycle shorter than a collection still counts
flexway_upstream_connections, flexway_upstream_errors_total	The connections open to each upstream; the answers the gateway wrote for it, by code (upstream_timeout, upstream_unavailable, upstream_error)

Event journal - GET /events, on the admin port: the latest reloads (with a rejection's cause), breaker transitions and starts or stops of load shedding on this replica, newest first; 128 of each kind, in memory, lost at a restart. It is what the console's overview reads - never the logs.

curl -s localhost:9090/events | jq '.events[:5]'

curl -s localhost:8428/api/v1/query --data-urlencode 'query=sum(flexway_requests_total)'
# dashboard: http://localhost:3000/d/flexway-gateway; traces: Explore > Tempo, service flexway

Check: a trace crosses edge -> gateway -> upstream without a break in context, even with no collector on the gateway side. Metrics carry the operation, never the concrete path: a rising _unmatched is traffic to undeclared routes. No log contains personal data in the clear.

Reading a shutdown. On SIGTERM, readyz turns red first - long enough for the orchestrator to take the replica out -, then what is in flight finishes, streams included, within server.shutdown.drain. The log says so in one line:

{"level":"INFO","msg":"shutting down: readiness withdrawn, draining in-flight requests","readiness_delay":5000000000,"drain":20000000000}

Past drain, the process cuts what is left, says so (the drain ran out: the requests still in flight are cut) and exits cleanly, after its last OTLP push: a Kubernetes terminationGracePeriodSeconds - or a Docker Compose stop_grace_period, 10 s by default - must therefore exceed drain by a few seconds. The stop is announced (flexway_server_draining = 1) by an immediate push on SIGTERM: even a process killed before its drain ends is said “Retired” by the console.

Metrics and propagation implemented since 0.2.0; the tenant is a dimension since 0.3.0.

Access log, sampling (1.0.0) - observability.access_log: true writes one access line per request, with the path template (never the path received). observability.sampling: { max_per_second: 50 } caps the root traces per replica, never cutting a trace begun upstream.

Log level (1.1.x) - observability.log_level: debug shows what each request does (operation, template, upstream, status, duration), without a restart: the level follows the reload, and a log level changed line says so. Go back to info after the diagnosis. Every line carries trace_id when the request has a trace; a start that cannot go on ends on a FATAL line.

Simulating (1.0.0) - "what would this configuration do?" without deploying it, on the admin port; nothing is called, spent or counted:

curl -s localhost:9090/simulate -d '{"method":"GET","host":"api.example.com","path":"/orders?status=open",
  "identity":{"issuer":"customers","subject":"alice","tenant":"acme","scopes":["orders:read"]}}'

The answer gives the verdict (served, or refused and its code), the operation, the chain and the request the upstream would receive.

Auditing decisions (1.0.0) - audit: { decisions: refusals } (or all) writes one authz_decision line per decision on a declared operation, on stdout but marked log.type: audit so that your pipeline files it apart. It names the operation and its path template (never the path received), the status and the refusal's code, the mechanism, the issuer, the tenant, the agents, and the subject hashed. Each line carries seq, prev and hash: a line removed or changed shows. With audit: { key: "${AUDIT_KEY}" } these are HMACs - without the key, whoever can write the whole stream can also rewrite it.

Opaque tokens (1.0.0) - a provider that issues references rather than JWTs is declared with opaque_tokens (one issuer at most, client required): each token is introspected (RFC 7662) and held to the same rules as a JWT; the answer is kept cache (30 s by default), never past its exp - the delay before a revocation shows.