Operations¶

Status conditions¶

See the full condition/reason vocabulary in the CRD reference. Quick read: all eight conditions (Validated, Allocated, PortReady, PortAttached, FIPAttached, RoutingReady, ConfigRendered, DeploymentReady) True means the gateway pod is deployed, Available (DeploymentReady specifically tracks status.availableReplicas >= 1, not just that the Deployment object was accepted — a Pending/unschedulable or crash-looping pod holds it False/DeploymentUnavailable) and its config is current — it does not by itself mean the tunnel is up. Tunnel state lives separately in status.tunnels[].state, written by status-agent, not the controller.

kubectl get ipsecgateway <name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Metrics and alerts¶

ipsec-exporter (torilabs/ipsec-prometheus-exporter, pinned at v1.9.2 in the chart — third-party, not built in this repo) serves Prometheus metrics on port 9903 per gateway pod, reading strongSwan's VICI socket. Metric names are prefixed strongswan_; the operator itself additionally exposes ipsec_operator_servicecidr_allocated_ratio on its own --metrics-bind-address (same registry the manager already serves, not a separate endpoint).

Enable scraping with --set monitoring.enabled=true (off by default — the monitoring.coreos.com/v1 CRDs it renders aren't guaranteed present on every target cluster). This wires:

A ServiceMonitor for the operator's own metrics.
A PodMonitor covering every gateway's ipsec-exporter sidecar across all tenant namespaces (a ServiceMonitor can't do that job — see the chart's templates/podmonitor.yaml doc).
A PrometheusRule with three alerts:

Alert	Expression	Severity	What it actually means
`IPsecTunnelDown`	`strongswan_sa_status != 0` for 5m	critical	The child SA (the dataplane tunnel, not just the parent IKE session) has not been `INSTALLED`/`REKEYED`/`REKEYING` (all map to `0`) for over 5 minutes. `1` = IKE established but child down, `2` = down, `3` = unrecognized.
`IPsecRekeyFailures`	`changes(strongswan_sa_status[15m]) > 3`	warning	A proxy, not a direct signal — the exporter has no rekey-failure counter at all (verified against upstream `v1.9.2` source, every `*Collector`). A failed rekey manifests as the child SA tearing down and re-establishing, i.e. `strongswan_sa_status` flapping; this alerts on more than one scheduled-rekey-worth of flapping in 15 minutes.
`IPsecPoolNearExhaustion`	`ipsec_operator_servicecidr_allocated_ratio > 0.8` for 30m	warning	Over 80% of the configured `serviceCIDRPool`'s `/24` subnets are allocated to a gateway. Widen the pool before it exhausts (a full pool fails new gateways' `Allocated` condition with `AllocationFailed`, requeued every minute).

A pre-built, Grafana-importable dashboard ships at docs/dashboards/ipsec-gateways.json.

Security group quota exhaustion (leaked duplicate SGs)¶

Symptom: the OpenStack project's security-group quota is unexpectedly near/at its limit, and openstack security group list shows many groups with the same name (one per IPsecGateway that hit this) and only Neutron's default egress rules — no ingress. PortReady on the affected gateway(s) is False with reason EnsureSecurityGroupFailed or EnsurePortFailed.

This was a real incident (2026-08-19): EnsureSecurityGroup's create-then-tag call used gophercloud's single-tag Add endpoint (PUT .../tags/{tag}), which puts the tag verbatim into the URL path with no encoding. Every tag this operator sets is of the form ipsec.bnerd.net/uid=<uid>, which contains a literal / — splitting the request into two path segments and 404ing against Neutron. The security group had already been created by that point; the 404 aborted the call before rule convergence, and the by-tag adopt lookup on the next reconcile could never find the untagged orphan it had just left behind, so every retry created another one. Fixed by tagging via the ReplaceAll endpoint (tag list in the JSON body, not the URL path) and making every Ensure* create path leak-proof (a failed tag application now deletes the just-created, untagged resource instead of returning with it still live). OpenStack-step failures also now requeue at a fixed ~15s interval instead of controller-runtime's ms-scale exponential backoff, which was the amplifier that turned one failed call into dozens of leaked groups in seconds.

The operator self-heals from a pre-fix leak automatically: on its next reconcile, EnsureSecurityGroup also checks for a same-name, same-description, still-untagged security group (the description records which gateway/tag it belongs to) and adopts/re-tags it instead of creating another one. This does not clean up groups already leaked before the fix was deployed — those still count against quota and need a manual openstack security group delete pass (safe to delete any untagged, default-egress-only group matching the leaked name/description pattern; the one the operator actually adopts on its next reconcile is picked automatically, so deleting the rest is pure cleanup, not a race with the operator).

Node reschedule and port reattach¶

See Networking for the full mechanics of status.nodeName and EnsurePortAttachedToServer. One thing worth knowing before it causes a false-alarm triage: Nova's interface detach is asynchronous, so a reschedule (the previously-pinned node going NotReady) can log a single failed attach attempt immediately followed by a clean retry a few seconds later — the operator waits for the old attachment to actually clear before re-attaching, but a transient PortInUse/409 in between is expected, not itself a sign of a stuck gateway.

Tunnel-down triage¶

Start from status.tunnels[] on the CR, then drop into the pod:

# 1. What does the CR itself say?
kubectl get ipsecgateway <name> -o jsonpath='{.status.tunnels}' | jq .

# 2. Live state from strongSwan directly (bypasses status-agent/K8s entirely)
kubectl exec -n <tenant-ns> deploy/<name> -c strongswan -- swanctl --list-sas
kubectl exec -n <tenant-ns> deploy/<name> -c strongswan -- swanctl --list-conns

# 3. Is the rendered config actually what you expect?
kubectl exec -n <tenant-ns> deploy/<name> -c strongswan -- cat /etc/swanctl-rendered/swanctl.conf

# 4. Is status-agent seeing the same thing, and reloading/pushing the PSK?
kubectl logs -n <tenant-ns> deploy/<name> -c status-agent --tail=100

# 5. Is the security group actually admitting the peer?
#    (per-tunnel UDP/500 + UDP/4500 ingress, restricted to peer.address/32 —
#    see guides/networking.md)

Common causes, cheapest check first:

Security group / peer address mismatch — the ingress rule is scoped to spec.tunnels[].peer.address/32; if the peer's actual source IP differs (NAT on their end, a secondary WAN), IKE never arrives.
PSK not loaded — check status-agent logs for a push shared key error; a checksum-changed-but-load-failed cycle retries every tick, so a persistent failure (not a transient one) means the Secret's key name/value is wrong. Never confirm the PSK's own value in logs — status-agent never logs it (see PSK rotation); look for auth failures on the strongSwan side instead.
Config not reloaded — compare the mounted swanctl-rendered ConfigMap's content against swanctl --list-conns's output; a stuck reload shows in status-agent logs as a reload: error (it retries every tick against the same unapplied content on failure).
Genuine peer-side issue — proposals mismatch, DPD too aggressive for an intermittently-reachable peer, or the client's own device down. At this point you're reading strongSwan's own daemon log, not this operator's. Note that status-agent is already actively retrying a non-ESTABLISHED tunnel on its own (see Initiate-if-down watchdog) — a persistent "stuck at CONNECTING" state despite repeated "initiating tunnel" log lines usually means this, not a one-off negotiation glitch.

PSK rotation¶

PSK delivery is not via swanctl.conf — an earlier design rendered an ike-<name> { file = ... } secrets section there, but file = is not valid swanctl.conf syntax for an IKE PSK secret (it's documented for private-key sections only), so that path likely never authenticated anything. The render package (internal/render/swanctl.go) emits connections only; its own test suite asserts no secrets {} block or inline secret value ever appears in rendered output.

Instead:

status-agent mounts the PSK Secret at /etc/swanctl-psk/psk (same volume as strongswan, read-only, no subPath — a Secret update propagates via kubelet's live sync without a pod restart).
On every tick, it checksums that file; on a checksum change or whenever a config reload just happened (defensive — a connections reload should never leave charon without credentials for them), it reads the tunnels' peer identities off the CR and calls VICI's load-shared (data field — verified against the strongSwan vici README, not guessed from an adjacent command name) with owners = each tunnel's peer.id.
load-shared uses a stable per-gateway credential-set ID (ipsec.bnerd.net/<gateway-name>), so a later push replaces the entry rather than accumulating duplicates.

To rotate a PSK: update the Secret's value and apply it. No CR edit, no pod restart, no swanctl --load-conns needed for the PSK specifically — the next status-agent tick (default interval — see the Deployment/Agent wiring) picks up the checksum change and pushes it. Nothing in this path — the checksum, the load-shared call, any error message — ever embeds the PSK's own content; only its sha256 digest is computed and logged.

Config-reload behavior¶

Both rendered ConfigMaps (<name>-swanctl holding swanctl.conf, <name>-nat holding nat.rules) are mounted whole, never via subPath — a subPath mount is bind-mounted once at pod creation and never observes a later ConfigMap update, which would silently defeat reload entirely. Two different mechanisms then react to a change, matching each file's consumer:

nat.rules — the init-nat container is a regular (not init) container running an apply loop: iptables-restore once at start, then poll the file's md5sum every 5 seconds and re-apply on change. This is why it's a long-running sidecar-style container rather than a true Kubernetes initContainer, which would only ever run once.
swanctl.conf — status-agent checksums it every tick; on change it shells out to swanctl --load-conns --file <path> (not raw VICI — govici v0.7.0 has no high-level load-conns/load-creds helpers, so reload goes through the strongSwan image's own swanctl CLI, which already knows how to parse its own config format). On reload failure, the last-applied checksum is deliberately left unchanged so the next tick retries against the same still-unapplied content rather than silently giving up.

--load-conns, never --load-all

This is load-bearing, not a style choice. --load-all internally diffs charon's currently-loaded shared secrets against the config file and unloads anything present in charon but absent from the file — confirmed by reading strongSwan's own source (src/swanctl/commands/load_creds.c). Since this operator's rendered swanctl.conf deliberately has no secrets section (the PSK is pushed separately over VICI — see PSK rotation), every --load-all call was unloading that key again immediately after loading it — live-observed in a pod-roll test: status-agent pushed the key, Reload's --load-all then logged "unloaded shared key", and the next initiation ran keyless again. --load-conns only ever touches connections, never credentials.

No config-hash pod annotation exists, and none of this ever restarts the pod — a Deployment update only happens for a NAD-name or image change.

Initiate-if-down watchdog¶

Beyond reload/push-on-change, status-agent actively retries a tunnel that's DOWN — only DOWN, and only once it has stayed DOWN for two consecutive ticks. On every tick, once (a) --load-conns has succeeded at least once in this process and (b) the same tick's shared-key push (if one was needed) also succeeded, it sends a VICI initiate for every CR tunnel whose current state (from the same list-sas poll that feeds status.tunnels[] — no extra VICI round trip) has read DOWN on this tick AND the previous one. This covers a stalled half-open SA, a failed initiation, and a whole-pod restart (see the scope note below) that would otherwise sit there forever with a technically-correct config and no active retry.

DOWN only, and two ticks — a review-round correction

An earlier version of this watchdog fired on ANY non-ESTABLISHED state, on the first tick it was observed. Both were wrong: charon reuses an existing CONNECTING IKE_SA and queues a duplicate CHILD_SA on a repeated initiate against it (confirmed against strongSwan's own ike_sa_manager.c/ike_sa.c source — see internal/vici/status.go's StateDown doc for the citation), which both fired redundant initiations on every ordinary bring-up/config change AND broke this operator's own SA-status parsing (it expects exactly one CHILD_SA per connection name). CONNECTING/REKEYING are charon's own active negotiation, not something stuck — leave them alone. The two-tick debounce exists so a connection this same tick's reload only just loaded, or one charon's own start_action is already bringing up, is never raced.

initiate is fired with a -1 timeout (per the vici README: "returns a result immediately") — status-agent never blocks a tick waiting for a negotiation to complete; the next tick's list-sas poll observes whatever state resulted, same as it already does for a naturally negotiating SA.
Both gates (load-conns-ever-succeeded, this-tick's-push-succeeded) exist to avoid reintroducing the credential-race this whole guide's PSK-ordering discussion is about: initiating a connection charon has never loaded, or one whose shared key may not have landed, is exactly the failure mode --load-conns and the push-before-reload ordering fix close — the watchdog must never route around them. The watchdog itself runs even if the SAME tick's status.tunnels[] patch fails afterward — an apiserver hiccup writing status must never also silence the one thing actively retrying a stalled tunnel.
An initiate failure (peer unreachable, proposal mismatch, ...) is logged and retried next tick — it is a routine, expected-to-sometimes- fail condition, not a status-agent operational error. Logging is transition-based: Info the tick a down-streak first becomes initiate-eligible (or first fails), Debug on every repeat thereafter — a persistently-down tunnel doesn't spam the default log level every 15s forever. Look for "tunnel down, will initiate..." / "initiating tunnel" / "initiate failed" log lines to see it in action.

Scope: what this does and doesn't recover from. A whole-pod restart recovers fully — status-agent's own boot sequence reconnects to VICI, pushes the PSK, --load-conns, and the watchdog resumes as part of normal ticking. A strongswan/charon-container-only restart (the pod stays Running, only that container restarts) does not recover on its own: status-agent connects to the VICI socket exactly once at boot and has no reconnect logic, so every VICI call starts erroring instead of reporting DOWN, and the watchdog never sees a state it could act on. See Known Limitations — a VICI reconnect-and-re-push mechanism is ledgered as a fast-follow, not built as of this release.