Alerting from the console

Operations guide: configure, deploy and observe the gateway. What the feature does, product side: product guide; calling it: developer guide.

The console evaluates its rules every 15 s, in both modes, and notifies through a signed webhook. Nothing is typed into the console: rules, webhooks and summaries are declared in the chart's values, read again without a restart; an invalid version is refused and said on the Alerts screen, the previous one keeps applying.

Steps

Declare rules, webhooks and summaries

1. Write the rules, webhooks and summaries in the console's values:

   values.yaml

   # The console's values
   alerts:
     rules:     # closed kinds: error_rate, p95, breaker_open, slo_burn, new_error_code
       - { name: reports-5xx, kind: error_rate, upstream: reports, threshold: 5%, for: 1m, notify: [ops] }
       - { name: orders-p95, kind: p95, operation: createOrder, threshold: 300ms, for: 2m }
     webhooks:  # https required (allowHttp lifts it); secretFile: whsec_<base64>, one secret per line
       - { name: ops, url: "https://hooks.example/flexway", secretFile: /etc/flexway-console-alerts/ops }
       - { name: chat, urlFile: /etc/flexway-console-alerts/chat-url, secretFile: /etc/flexway-console-alerts/chat, allowPrivate: true }
     summaries:
       - { name: weekly, every: weekly, at: "07:00", weekday: monday, webhook: chat }   # UTC
     existingSecret: console-alerts    # secrets and URLs carrying a token, mounted at /etc/flexway-console-alerts
   slos:
     - { operation: createOrder, availability: 99.9%, latency: { p95: 300ms, target: 99% }, window: 30d }
     - { name: orders-api, upstream: orders-svc, availability: 99.5% }   # a group: see SLO objectives on groups of operations

2. Put the secrets and the URLs carrying a token in the Secret named by existingSecret.

3. Deploy the values: the console reads the file again without a restart.

Tune the built-in rules

Tune a built-in rule rather than silence it - a few cases:

alerts:
  builtins:
    # Tighten: a small publication traffic fires too early; 5 % refused, from 100 publications.
    - { name: broker-backpressure, threshold: 5%, minRequests: 100 }
    # Delay: a broker failing over in less than 3 min is not an incident.
    - { name: broker-unavailable, for: 3m }
    # Sooner: an open breaker already fires at once; a degraded gateway, after 30 s.
    - { name: gateway-degraded, for: 30s }

With the native Kafka proxy (since 1.6.0), the kafka-proxy-refusals rule counts closed causes and accepts excludeCauses (Native Kafka clients: a virtual cluster per tenant):

alerts:
  builtins:
    - { name: kafka-proxy-refusals, threshold: 10%, excludeCauses: [quota] }   # do not fire on a quota reached

With the MQTT device listener (since 1.7.0), five more rules - device-connections-refused, sparkplug-node-offline, mqtt-listener-saturation, device-throttled and sparkplug-sequence-gaps, the last two disabled by default; the threshold of the two Sparkplug rules is a count, not a ratio (Console: the Devices screen).

Verify the signature on the receiving side

The body: {"type": "flexway.alert.firing", "timestamp": ..., "data": {"rule", "kind", "state", "since", "target", "value", "unit", "threshold", "subjects"}} (flexway.alert.resolved, flexway.summary.daily, flexway.summary.weekly) - never a tenant. Verify the signature (Standard Webhooks) before anything else:

import base64, hashlib, hmac, time

def verify(secret: str, headers, body: bytes) -> bool:
    key = base64.b64decode(secret.removeprefix("whsec_"))
    msg_id, ts = headers["webhook-id"], headers["webhook-timestamp"]
    if abs(time.time() - int(ts)) > 300:          # replay: 5 minutes of tolerance
        return False
    mac = hmac.new(key, f"{msg_id}.{ts}.".encode() + body, hashlib.sha256).digest()
    want = base64.b64encode(mac).decode()
    # two signatures during a rotation, separated by a space: one must match
    return any(hmac.compare_digest(sig.removeprefix("v1,"), want)
               for sig in headers["webhook-signature"].split())

Rotate a secret

1. Put the new secret on the file's second line on the console's side.
2. Deploy it on the receiving side.
3. Remove the old one - the console signs with both meanwhile.

Verify

• The Diagnostics -> Alerts screen (admins) lists each rule, its state (ok, pending, firing, resolved, unknown), its value against the threshold, its curve over 6 h and each webhook's deliveries (result, status, cause).
• A refused file is said there, with its cause; the previous version keeps applying.
• GET /api/alerts gives the built-in rules' values in force and where they come from (tuning).

Good to know

Available since 1.3.0; tuning of the built-in rules in 1.6.0.