Known Limitations¶

A candid list of what bnerd-vpn-operator does not cover yet, or has not proven for every scenario, as of v0.1.0. Nothing here is guessed.

v0.1.0 release status¶

Every reconciler/render/alloc code path is covered by envtest and unit suites against a fake OpenStack client (internal/openstack/fake) and a fake VICI session. The items below are this release's known, deliberate constraints and remaining gaps, ordered roughly by how much they matter.

v0.3.0 additions: the unified `vpn.bnerd.net` model¶

The items in this section are specific to VPNGateway/VPNPeer (see The unified model); everything else on this page still applies unchanged to the frozen IPsecGateway/ClusterLink group, which v0.3.0 continues to serve in parallel.

`ha.enabled` is reserved, not implemented — but node-loss detection quick-wins ARE in `v0.3.0`¶

VPNGatewaySpec.HA exists as a schema placeholder only; Validate() rejects ha.enabled: true outright rather than silently ignoring it — the same honest-reject discipline this operator uses everywhere a field exists before its behavior does (e.g. IPsecGateway.spec.tunnels[].mode: routed, rejected the same way). Warm-standby HA (a second, passive gateway pod that takes over on failure) is not built and was not delivered in v0.5.0.

What is in v0.3.0 is a pair of node-loss detection quick-wins that shorten the outage window for the existing single-replica model, not a true HA mechanism: the gateway pod's NoExecute tolerations for node.kubernetes.io/not-ready/unreachable are set to 30s (haTolerationSeconds) instead of Kubernetes' 5-minute default, and a Node watch re-triggers reconcile for every VPNGateway pinned (status.nodeName) to a node that goes NotReady, so the reselect-and- reattach onto a healthy node starts promptly rather than waiting for the next unrelated reconcile. This shortens, but does not eliminate, the outage described in "Gateway pods are hard-pinned to a single node" below — there is still exactly one pod, and the reattach still rolls it.

Active-active HA is ruled out for this horizon, not merely deferred¶

Unlike warm-standby (deferred to v0.5.0), active-active — two simultaneously-serving gateway pods for the same client — has been evaluated and ruled out under the current architecture, not just postponed: the model is one Neutron port with one 1:1 floating IP per gateway, and IKE/WireGuard both terminate at a single endpoint identity. There is no place in that model for two pods to both legitimately answer the same public IP without a load balancer sitting in front of the IKE/WireGuard negotiation itself — a materially different architecture, not a configuration option. Revisit only alongside a broader redesign, not as an incremental v0.3.x/v0.4.x/v0.5.x feature.

WireGuard gateways have no Prometheus exporter¶

See WireGuard gateways: no Prometheus exporter. WireGuard has no VICI-style RPC the way strongSwan does, and no exporter container exists in the wireguard container set as of v0.3.0. Monitor via status.peers[] instead.

A WireGuard hostname-endpoint site peer widens the gateway's listen port estate-wide¶

A WireGuard site peer whose endpoint is a hostname (not a literal IPv4 address) cannot be pinned to a /32 security-group rule, so its UDP listen port is opened estate-wide (0.0.0.0/0) — the same posture as a road-warrior peer, and safe because WireGuard authenticates by public key (unauthenticated packets are dropped in-kernel, so the open port is not an open door). The gateway's SecurityGroupScoped condition goes False with reason HostnameEndpointOpensPort, naming the affected peer(s); pin an IPv4 endpoint to keep the port scoped (True/AllSourcesPinned). Not set for ipsec gateways, which always pin per peer. See WireGuard gateways: security-group semantics.

Identity-issued road-warrior peers are specified but not built¶

A future integration point — road-warrior VPNPeers provisioned/revoked by an identity source (SSO/Keycloak) rather than a hand-applied CR per device — is spec'd in the design but not implemented in v0.3.0. Every road-warrior peer today is a manually-applied VPNPeer CR.

New-model ipsec gateways fail loud, not silently, when reverse exposures lack `internalSourceCIDRs` — a behavior change from the legacy path¶

This is a deliberate behavior change worth calling out explicitly if you're used to the legacy IPsecGateway path's behavior. The legacy group degrades silently when spec.reverseExposures[] is set but the operator has no internalSourceCIDRs configured: RoutingReady still goes True (routing genuinely is programmed), its message names the gap, and the reconciler logs a matching warning — but nothing stops the reconcile or surfaces a hard failure (see "Reverse-exposure security-group ingress requires internalSourceCIDRs" below).

The unified renderer refuses this outright: rendering any reverse exposure with InternalSourceCIDRs empty is a fail-loud error — ConfigRendered=False/ReverseExposuresRequireInternalSourceCIDRs — because post-decap iptables in the gateway pod's own netns is the only place reverse-exposure traffic can be constrained to fabric-originated sources at all (a Neutron Security Group never sees it), so rendering the rule without a source constraint would mean any tunnel peer sharing the gateway's serviceCIDR traffic-selector could reach another peer's reverse-exposure target. Set the operator's internalSourceCIDRs (IPSEC_INTERNAL_SOURCE_CIDRS) before onboarding the first VPNGateway/VPNPeer pair that uses a reverse exposure.

`internalSourceCIDRs` overlapping a peer's own subnets reopens that peer's reverse path (deployment constraint)¶

If the operator's internalSourceCIDRs is configured broadly enough to overlap a specific peer's own SourceCIDRs (its remoteSubnets/ subnets), that overlap effectively re-admits that peer's own addresses as a valid source for a reverse-exposure flow — reopening a path the per-peer authorization model intends to gate. This is an operator-level configuration discipline concern, not an attacker-controllable input: internalSourceCIDRs is a boot-time operator config value, never CR-supplied, so the risk is a careless CIDR choice by whoever configures the operator, not something a VPNPeer author can trigger. Keep internalSourceCIDRs scoped to genuine fabric-side address space that never overlaps any tenant's routed peer subnets. A render-time disjointness check (rejecting an internalSourceCIDRs entry that overlaps any peer's SourceCIDRs) is a candidate v0.3.x fast-follow, not built as of this release.

Established flows survive grant revocation until the conntrack entry expires¶

Revoking a peer's grant (removing it from authorization.exposures/ .reverseExposures, or deleting the block entirely) updates the rendered NAT/authz ruleset on the next reconcile — but the renderer only withdraws the rule; it does not proactively flush the kernel's existing conntrack entries for that flow. A connection that was already ESTABLISHED before the grant was revoked keeps flowing until its own conntrack entry naturally expires (idle timeout or explicit close), even though the iptables rule that would newly-admit it is now gone. A conntrack -D flush on grant change is ledgered as a fast-follow, not built as of v0.3.0 — treat a grant revocation as "no new connections after the next reconcile", not "existing connections are immediately cut".

Gateway pods are hard-pinned to a single node (no spreading, no HA)¶

The OpenStack port attachment doesn't follow the pod on its own — it has to be hot-attached to the exact Nova instance backing a specific Kubernetes node, and the pod has to be hard-pinned (spec.nodeName, not a nodeSelector/affinity hint) to that same node. See Networking for the full mechanics. Consequences:

Every gateway is bound to exactly one node at a time. There is no way to spread replicas across nodes for HA — see "No high availability" below, which this reinforces rather than duplicates: even a hypothetical multi-replica gateway would still need every replica pinned to nodes with their own dedicated Nova-attached ports, not a shared one.
Each node that ends up hosting one or more gateway pods accumulates one Nova interface attachment per gateway — this is capacity-relevant: Nova/Neutron enforce a per-instance interface limit, and nothing in this operator tracks or caps how many gateways can stack onto the same node before hitting it. Dedicating a node pool to gateway workloads (see "Gateway pods have no scheduling or resource controls yet" below) also bounds this exposure.
Sticky selection (kept across reconciles as long as the node stays Ready) minimizes churn, but a node that goes NotReady triggers a reselect-and-reattach onto a different node, which rolls the pod deliberately — there is no cordon/drain-aware pre-migration; the gateway is down for the duration of the reattach + pod reschedule + IKE renegotiation.

NAD/Multus attachment mechanics: host-device-by-hwaddr, verified end-to-end¶

The chart's default nadTemplate is the host-device-by-hwaddr CNI config verified end-to-end against a real OpenStack/Neutron network on our reference deployment — see Networking for the mechanics and the two cluster-level prerequisites (Multus + containernetworking/plugins installed, Cilium's cni-exclusive=false) neither the reconciler nor the chart can satisfy from inside the cluster. The reconciler still treats the whole CNI config as an opaque, string-substituted template, so a different SDN/attachment mechanism can be substituted by overriding nadTemplate — but the shipped default is now a verified config, not an unvalidated example.

`status.tunnels[].rekeyAt` semantics are unverified¶

RekeyAt is computed by adding the CHILD_SA's rekey-time VICI field (as seconds) to "now", on the assumption that rekey-time reports seconds-remaining-until-rekey (per the govici README). This has not been checked against a live charon daemon and must be verified during e2e before this field is treated as authoritative for anything operationally significant (alerting, capacity planning). A related, lower-severity gap: mid-rekey there is a dual-CHILD_SA window where BytesIn/BytesOut/RekeyAt can read as zero/nil while State stays correct — cosmetic, not a correctness bug.

NetworkPolicy egress is unrestricted (deliberate deviation, pending follow-up)¶

Every gateway's owned NetworkPolicy is ingress-only — egress on the gateway pod's NIC1 is not restricted at all. Correctly admitting DNS, the kube-apiserver (status-agent's k8s client), and the tunnel's own OpenStack-routed dataplane egress without getting any one of those wrong (which fails a tunnel silently rather than loudly) needs the real egress set characterized against a live gateway first. Separately, the operator-namespace ingress rule on that same NetworkPolicy is deliberately any-port, not scoped down further — headroom for operator-side tooling against a single-purpose pod, a considered choice rather than an oversight. See Networking for the full picture, including what NIC2 (the tunnel-carrying interface, outside NetworkPolicy's reach entirely) means for both of these.

No high availability¶

Every IPsecGateway renders exactly one Deployment replica. There is no active/passive or active/active failover model, no leader election between gateway pods, and no automatic re-scheduling guarantee beyond what the Kubernetes scheduler itself provides for a single-replica Deployment. A node failure taking down a gateway pod means that client's tunnel is down until the pod reschedules and strongSwan re-negotiates.

IKEv2 with pre-shared-key authentication only¶

There is no certificate-based (pubkey) authentication support, and spec.ike.version is schema-constrained to 2 — no IKEv1 path exists or is planned. A client whose device only supports IKEv1, or who requires certificate auth, cannot be onboarded onto this operator as it stands.

Single PSK per gateway¶

spec.auth.pskSecretRef is one Secret reference per IPsecGateway, shared across every tunnel the gateway terminates. There is no per-tunnel PSK override — a gateway with multiple tunnels to different peers authenticates all of them with the same pre-shared key (scoped per-peer via VICI load-shared's owners list, but the secret value itself is shared).

Multi-subnet tunnels render as one child SA, not split per subnet¶

spec.tunnels[].remoteSubnets with more than one CIDR currently renders into a single child SA with a comma-joined remote_ts, unlike the field-proven Ansible reference template (.j2), which splits each remote subnet into its own child SA. Every tunnel validated on the reference deployment to date has had exactly one subnet, so this gap hasn't mattered yet — but some third-party firewalls require narrow, per-subnet Phase 2 selectors and may reject or mis-negotiate the comma-joined form. Revisit before the first genuinely multi-subnet client.

CR string fields reaching swanctl.conf/iptables are validated by `Validate()`, not a CRD-level pattern¶

As of the v0.1.0 security review, spec.Validate() rejects malformed or injection-shaped values before they reach text/template: tunnel/exposure/ reverse-exposure names must match a DNS-label pattern, peer.id is restricted to FQDN/IP/email-style characters (no braces or whitespace), peer.address/target.address/serviceCIDR/remoteSubnets entries must parse as IPv4, target.host must be a valid RFC 1123 hostname, and every IKE and ESP proposal must exact-match the approved allowlist — see the CRD reference field table for the full list. This is enforced in Go (api/v1alpha1/ipsecgateway_types.go), not as a CRD-level +kubebuilder:validation:Pattern marker, so a value can still be rejected one reconcile after kubectl apply succeeds rather than at admission time.

Exposure-name uniqueness is enforced by the allocator, not the schema¶

spec.exposures[]/spec.reverseExposures[] name uniqueness (across both lists combined) is currently caught by the virtual-IP allocator at reconcile time (AllocationFailed), not by spec.Validate() up front or a CRD-level rule — so a duplicate name surfaces one reconcile-cycle later than it could. Functionally correct, just not fail-fast.

No live-cluster verification of PSK rotation or config reload¶

Both the PSK rotation and config-reload flows are covered by unit tests against fakes, but have not been exercised end-to-end against a real strongSwan charon process. In particular, the VICI load-shared field name (data, not shared) was corrected during development against the upstream vici README rather than left as an implementer's guess — but the overall flow (checksum → load-shared → successful auth) is still e2e-unverified.

RBAC drift risk between `config/rbac` and the chart¶

config/rbac/role.yaml (the kustomize/controller-gen-derived manifest) has drifted from the chart's templates/rbac.yaml, which is the one actually applied on a real Helm install and is kept in sync with the reconciler's +kubebuilder:rbac markers by hand. Not a functional gap for a Helm-installed operator, but a maintenance hazard if config/rbac is ever used as the source of truth without re-syncing it first.

`spec.tunnels[].mode: routed` is not implemented¶

The CRD's enum accepts routed alongside snat (the API shape is stable), but Validate() rejects any tunnel that actually sets it in v0.1.0, with an error naming the version — it is honest-reject-only, not silently treated as snat. The intended behavior (routing the tenant's real subnet through the gateway instead of SNATing it, reusing the same allowed-address-pair + router route machinery reverse exposures already have) is planned but not built. spec.reverseExposures[] is the only supported way today to reach a client's services from our side. See Networking.

Gateway pods have no scheduling or resource controls yet¶

buildDeployment sets no nodeSelector, tolerations, resources (requests/limits), livenessProbe, or readinessProbe on any container in the gateway pod's Deployment — it schedules like any ordinary, best-effort-QoS pod onto whatever node the scheduler picks, with no dedicated placement and no health-based restart/traffic-gating beyond the kubelet's default process-exit behavior. There is no CR or operator-config knob for any of this today. Until one exists, schedule a dedicated node pool for gateway pods (a NET_ADMIN/NET_RAW workload with a Multus second NIC benefits from isolation regardless) via a cluster-level mechanism outside the operator — e.g. a PodTolerationRestriction / nodeSelector webhook, or simply not co-locating other workloads on that pool.

Source-policy routing assumes NIC2 is always named `net1`¶

The init-nat prelude that fixes NIC2's reply-routing spoof-drop (see Networking) hardcodes the in-pod interface name net1 — Multus's naming convention for the first (and, today, only) network named in the pod's k8s.v1.cni.cncf.io/networks annotation. If a future change ever attaches a second NAD to the gateway pod, or a Multus configuration numbers interfaces differently, this name must be updated in lockstep (dedicatedNICName in internal/controller) — there is no dynamic interface-name discovery here.

The operator does not manage exposure-target security groups¶

Confirmed during live validation: DNAT'd traffic to a spec.exposures[] target arrives from the gateway's own Neutron port, which is not automatically a member of whatever security group the target (an internal LB VIP, a NodePort, a VM) normally trusts. This is deliberate, not an oversight — the target is a foreign-owned object the operator has no business reaching into and mutating the SG rules of. Admitting ingress from the gateway network's subnet (e.g. vpn-transit, 10.254.0.0/24) on the exposed ports is a prerequisite the requester of the exposure has to handle on the target side before onboarding; the operator does not check for it, and a missing rule fails silently (packets simply never arrive, same as any other SG drop). See Networking.

Reverse-exposure security-group ingress requires `internalSourceCIDRs`¶

Confirmed during live validation: spec.reverseExposures[] always gets its allowed-address-pair and router route from the reconciler (RoutingReady), so fabric-initiated traffic to a reverse-exposure virtual IP is routed onto the gateway's port — but by default nothing at the security group admits it there, since the per-peer IKE rules only ever open UDP 500/4500 from each tunnel's peer.address /32. Unlike the exposure-target gap above, this half is operator-rendered — it just needs the operator-wide internalSourceCIDRs chart value (IPSEC_INTERNAL_SOURCE_CIDRS) set to the fabric's own source CIDRs; see Networking for the exact rule set this renders. Left unset, RoutingReady still goes True (routing genuinely is programmed) but its message names the gap, and the reconciler logs a matching WARNING every reconcile a gateway with reverseExposures set has no CIDRs configured.

The initiate-if-down watchdog can't survive a charon-only container restart¶

The initiate-if-down watchdog covers a stuck/failed initiation and a whole-pod restart (that reruns status-agent's own boot sequence: reconnect to VICI, push the PSK, --load-conns, and only then the watchdog resumes). It does not cover strongswan/charon restarting on its own while status-agent's own container keeps running: status-agent connects to the VICI socket exactly once at boot (connectWithRetry in cmd/status-agent/main.go) and has no reconnect logic at all — a charon-only restart breaks that already-established govici.Session, and every subsequent VICI call (list-sas, load-shared, load-conns, initiate) starts erroring instead of reporting DOWN, so the watchdog never even sees a state it could act on. Only a full pod restart (which recreates the session from scratch) recovers. A VICI reconnect-and-re-push-on-reconnect mechanism is ledgered as a fast-follow, not built as of this release.

A cross-tenant gateway cluster's reverse routes are not scoped per customer¶

See Connecting service clusters: cross-tenant ClusterLinks for the shared-transit-network posture this builds on. Reverse-route automation converges every reverseExposures-bearing gateway's reverse route onto every ClusterLink's own router, estate-wide — a ClusterLink carries no reference back to which gateways it should or shouldn't serve, so this is not scoped to "only the gateways that customer project should be able to reach." On a gateway cluster linked to a single customer project this is harmless: every reverse route it installs belongs to a gateway whose traffic that customer's own router is already carrying anyway. It stops being harmless the moment a second customer project is linked to the same gateway cluster and a gateway serving that second customer also gets a reverseExposures entry: its reverse route would land on the first customer's router too (and vice versa), handing each customer's router a route into the other's service CIDR over the shared transit network. Per-customer transit-network scoping (which would close this structurally) is planned but not built. Until then: do not add a second customer's gateway with reverse exposures to a gateway cluster that already carries a cross-tenant ClusterLink, until reverse routes are scoped per link.

Owned children don't self-heal from manual tampering until the next reconcile¶

The reconciler has no Owns() watch on the per-gateway ServiceAccount, RoleBinding, or NetworkPolicy — only on ConfigMaps and Deployments. Manual edits or deletion of those three object kinds are only corrected on the next reconcile (a Host-exposure's 5-minute timer, or the manager's 10-hour resync period, whichever comes first), not immediately. Consistent with the minimal RBAC the operator is scoped to; revisit with a dedicated watch if drift between reconciles becomes operationally significant.