Networking

The three on-the-wire tiers

Direction	CR field	Extra provisioning needed	Opt-in?
Client → our services	spec.exposures[]	None beyond the port itself — traffic lands DNATed to the exposure's virtual IP, SNATed back out through the gateway's own port IP.	Default — plain snat-mode tunnel behavior.
Our side → client's services (reverse exposures)	spec.reverseExposures[]	An allowed-address-pair on the gateway's port and a router route for status.serviceCIDR, both applied by the reconciler (RoutingReady condition) — this is operator-managed, not something you provision separately. The security group's own ingress admitting the fabric source is also operator-rendered, but only once internalSourceCIDRs is configured — see Reverse-exposure reachability below.	Automatic whenever reverseExposures is non-empty — you don't set a separate flag, just add the entries (and internalSourceCIDRs for the SG half).
Routed mode (client's real subnet routed through the gateway, no SNAT) — planned, not implemented in v0.1.0	spec.tunnels[].mode: routed	Same allowed-address-pair + router route machinery as reverse exposures — needsRouting() triggers on either condition.	Opt-in per tunnel; default is snat.

An exposure or reverse exposure can target a service on any cluster reachable from the gateway cluster's fabric — same-cluster is the common case, but nothing in the tiers above is cluster-scoped. Reaching a different cluster for the first time needs a one-time, per-cluster fabric step (a vpn-transit router leg + static route) — see Connecting Service Clusters.

mode: snat vs mode: routed is meant as a NAT/routing-tier switch only — not a second IPsec encapsulation mode. Every tunnel's child SA (mode = tunnel in swanctl.conf) is always tunnel mode regardless of this field — policy-based IPsec.

mode: routed is not implemented yet

spec.tunnels[].mode accepts routed at the CRD level (the API shape is stable), but Validate() rejects any tunnel that actually sets it in v0.1.0, with an error naming the version. The needsRouting() / RoutingReady machinery in the table above already exists for reverse exposures and is designed to extend to routed-mode tunnels once the dataplane side (skipping the gateway's own SNAT for that tunnel's traffic) is built — until then, routed is honest-reject-only. Use spec.reverseExposures[] for the "our side reaches the client" direction today; there is no supported way to route a tenant's real subnet through the gateway without SNAT yet. See Known Limitations.

Traffic encryption layers / threat model

IPsec only covers client ↔ gateway. From the gateway pod onward, traffic to/from the exposure target rides the tenant overlay (the underlying OpenStack network) as plain, unencrypted east-west traffic — there is no second encryption hop once the client's payload has been decapsulated inside the gateway pod.

Policy: expose TLS services

Because east-west gateway→service traffic is plaintext at this layer, only expose services that are themselves encrypted end-to-end at the application layer — HTTPS (443), LDAPS (636), and equivalents. This is already true of every exposure validated on our reference deployment (HTTPS 443, LDAPS 636 — see the Quick Start worked example), so it costs nothing in practice, but it is a real prerequisite, not an incidental fact: flag any plaintext-protocol exposure (plain HTTP, plain LDAP, etc.) at onboarding rather than provisioning it as-is.

Fabric-level encryption for this hop — Cilium WireGuard transparent encryption per cluster, plus an encrypted inter-shoot path between clusters — is a planned roadmap item, not yet built. It slots underneath the routing design in this guide unchanged: nothing about exposures, reverse exposures, or the cross-cluster runbook needs to change when it lands, since it operates below the IP layer these features already route on.

Virtual IP conventions

Every gateway's status.serviceCIDR is a /24 carved from the operator's configured pool (100.64.0.0/16 by default, itself inside RFC 6598 CGNAT space 100.64.0.0/10). Within that /24:

Host range	Purpose
.1	Reserved for the gateway's own SNAT source address (serviceCIDRGatewayAddress) — never handed out as an exposure virtual IP; a pinned virtualIP that collides with it is a validation error.
.10–.199	Auto-assignment range for spec.exposures[] virtual IPs (forward — client reaching our services).
.200–.254	Auto-assignment range for spec.reverseExposures[] virtual IPs (reverse — us reaching the client's services).

Pinning a virtualIP outside its list's range, or one already claimed by another exposure, is rejected. The two ranges never grow into each other: exhausting one is an allocation error (AllocationFailed on the Allocated condition), not silent encroachment on the other.

NAT-T and floating-IP behavior

• The gateway's local_addrs in every tunnel is %any; local_id for every tunnel is the gateway's floating IP (status.floatingIP), not the Neutron port's fixed IP — this is what the remote peer's IKE identity check matches against.
• ESP/proto-50 ingress is deliberately not opened at the security-group level — NAT-T (UDP/4500) is assumed mandatory behind the gateway's 1:1 floating IP, so encapsulated ESP always arrives over UDP/4500, never bare proto-50. Only UDP/500 and UDP/4500 ingress are opened, each restricted to the specific tunnel's peer.address /32 — never to 0.0.0.0/0.
• A migration that pins an existing floating IP (spec.endpoint.floatingIP) changes nothing about NAT-T/local_id behavior — the pinned FIP is used exactly the same way a freshly-allocated one would be. See VM-to-Operator Migration.

Port attach and node-pinning mechanics

Confirmed end-to-end against a real OpenStack/Neutron network (our reference deployment, verified 2026-08-19). The gateway pod's dedicated Neutron port is not a floating resource that follows the pod on its own: it has to be hot-attached to the exact Nova instance backing whichever Kubernetes node the pod runs on, and the pod has to be hard-pinned to that same node, or the attachment and the pod end up on different machines with nothing to reconcile the mismatch.

1. EnsurePort creates (or adopts) the gateway's Neutron port, same as before.
2. The reconciler picks a node: if status.nodeName is already set and that node still exists and is Ready, it's kept (sticky — a gateway must not bounce between nodes on every reconcile just because a lexically-earlier candidate showed up). Otherwise it picks the first Ready, schedulable, non-control-plane node by name. The choice is persisted to status.nodeName before the attach call runs, same persist-before-programming discipline as the CIDR/vIP allocation.
3. The chosen node's spec.providerID (openstack:///<nova-server-uuid>) is parsed down to the bare server ID. A providerID that doesn't match that scheme fails the PortAttached condition loudly (ProviderIDUnparseable) rather than attaching nothing silently.
4. EnsurePortAttachedToServer converges the port's Nova attachment to that server ID: a no-op if already attached there; if attached to a different server (the gateway moved nodes), it's detached first, then attached to the new one; if unattached, it's attached directly.
5. The Deployment's pod gets spec.nodeName set to the same value — a hard pin, not a nodeSelector/affinity hint the scheduler could override.

If the pod ever lands on the wrong node anyway (e.g. manual tampering), the failure is loud, not silent: the host-device CNI plugin's ADD fails sandbox creation outright (failed to find host device: failed to find physical interface), immediately visible in kubectl describe pod events — not a pod that starts but never gets its second NIC. A node change (sticky selection falling back to a different node because the old one went NotReady) rolls the pod deliberately, the same way any other podTemplate change would; Cleanup detaches the port from whatever server still holds it before deleting the port itself, so a deleted gateway never leaves a dangling Nova attachment behind.

The host-device-by-hwaddr NAD

The chart's default nadTemplate (see Quick Start) is the exact config verified end-to-end on our reference deployment: host-device, selecting the interface by hwaddr — not by interface name (ens9 vs ens10 is not stable across nodes/kernels) — with static IPAM assigning the port's own fixed IP. On ADD, the whole interface moves into the pod's netns; on DEL it's returned cleanly to the node's root netns (verified: deleting and recreating the gateway pod, pinned to the same node, re-attaches cleanly with the same IP/MAC).

Two cluster-level prerequisites this mechanism depends on, neither of which the reconciler can fix from inside the cluster:

• Multus and containernetworking/plugins (host-device + static IPAM) installed on the target shoot — not shipped by this chart; install and pin versions per your own cluster's CNI conventions.
• Cilium's cni-exclusive set to false on the shoot (a Gardener-extension-managed field, spec.networking.providerConfig.cni.exclusive) — Cilium's default (true) actively renames any non-Cilium CNI config file it finds to <name>.cilium_bak on every reconcile, keeping Multus out of the CNI chain entirely with no error anywhere: the pod just never gets its second NIC. The operator logs a prominent, non-fatal startup WARNING if it can't confirm this key is "false" on kube-system/cilium-config (internal/preflight), but cannot set it itself — whoever provisions the shoot owns that.

Pod source-policy routing on NIC2

Live validation on our reference deployment found that the gateway pod's NIC2 (net1, the Multus/NAD-attached dedicated Neutron port) receives floating-IP traffic fine, but replies followed the pod's default route out NIC1 (Cilium) instead — and Neutron's anti-spoof filtering silently dropped every one of them, a 100% loss condition invisible from inside the pod. The fix is source-policy routing scoped to the port's own fixed IP, applied inside the pod netns before the NAT apply loop starts:

ip rule add from <portFixedIP>/32 lookup 100
ip rule add iif net1 lookup 100
ip route replace <subnetCIDR> dev net1 scope link table 100
ip route replace default via <subnetGatewayIP> dev net1 table 100

All three OpenStack-derived values are facts the operator already knows: EnsurePort resolves the port's subnet, its gateway_ip, and its cidr (the same extra Neutron call), and the reconcile order guarantees this happens before the Deployment is rendered. The connected-route line must run before the default-route line: the NAD's static IPAM assigns <portFixedIP> as a bare /32 with no connected route of its own, so without it the default route's nexthop (off-subnet from a /32's point of view) is unresolvable — this crash-looped init-nat in the field (ip route: Error: Nexthop has invalid gateway) until the connected route was added. The iif net1 lookup 100 rule exists because the from rule above only matches traffic still carrying the gateway's own source address at routing time — forwarded (DNAT'd) tunnel traffic is routed before POSTROUTING SNAT rewrites its source, so it's still the client's original IP at that point and falls through the from rule onto NIC1 instead, the same anti-spoof-drop failure mode this whole prelude exists to eliminate (confirmed during live validation). The init-nat container (which already runs with NET_ADMIN) installs all four lines idempotently as a prelude to its existing NAT-apply loop — there is no separate container or init step for this. Nothing here is user-configurable; it's applied unconditionally to every gateway pod.

Exposure target reachability — security groups

Prerequisite, not something the operator provisions: the security group protecting an exposure target — an internal LB VIP, a NodePort, a VM — must allow ingress from the gateway network's subnet (e.g. the vpn-transit network, 10.254.0.0/24) on the exposed ports. Traffic DNATed to a spec.exposures[] target arrives at that target from the gateway's own Neutron port, which is not itself a member of whatever security group the target's cluster/node/VM normally trusts — there's no implicit allow just because both sides are "inside OpenStack". For a Kubernetes Service sitting behind an internal Octavia load balancer, use that LB's source-range/security-group mechanism (e.g. loadBalancerSourceRanges or the Octavia-equivalent SG rule) to admit the gateway subnet, not a cluster-wide NetworkPolicy — NetworkPolicy governs pod-to-pod traffic inside the cluster, not traffic arriving at an LB's external-facing VIP. This is a target-side change the requester of the exposure has to make; the operator does not manage it (see Known Limitations).

Reverse-exposure reachability — security groups (operator-rendered)

This is the mirror-image direction, and unlike the exposure-target case above, the operator does render this half of the security group — it just needs to be told the fabric's source CIDRs first.

spec.reverseExposures[] gets its allowed-address-pair and router route from the reconciler automatically (RoutingReady, see the tiers table above), so a fabric-side initiator's traffic to a reverse-exposure virtual IP is routed onto the gateway's port. Confirmed during live validation: that routing alone is not enough — the gateway's own security group still has to admit that traffic at the port, and by default it doesn't, since the per-peer IKE rules (ikeSecurityGroupRules) only ever open UDP 500/4500 from each tunnel's peer.address /32, nothing else.

Set internalSourceCIDRs (chart value; IPSEC_INTERNAL_SOURCE_CIDRS on the operator, comma-separated) to the fabric's own source CIDRs — e.g. the vpn-transit subnet reverse-exposure initiators actually originate from — and the reconciler renders one additional ingress rule per unique (reverseExposures[].target.ports[] entry, internalSourceCIDRs entry) pair: TCP, port-scoped, RemoteIPPrefix set to that CIDR. A reverse exposure with no target.ports (all-protocol/all-port DNAT, same "empty means all ports" convention as spec.exposures[]) gets one all-protocol rule per CIDR instead. Identical rules from two different reverse exposures collapse to one (same dedupe approach the fake OpenStack client's own convergence check uses).

Leaving internalSourceCIDRs empty is a supported, if incomplete, configuration — RoutingReady still goes True (routing genuinely is programmed), but its message names the missing SG half, and the reconciler logs a matching WARNING on every reconcile of a gateway with reverseExposures set and no CIDRs configured. Compare this to the exposure-target case above, which the operator can never render (a foreign-owned SG on someone else's target) — internalSourceCIDRs is operator-owned infrastructure, so the operator renders the rule itself once it knows the value.

MTU / MSS

The rendered NAT script (RenderNATScript) unconditionally clamps TCP MSS to path MTU on the FORWARD chain (-j TCPMSS --clamp-mss-to-pmtu) for tunnel traffic — one rule, applied once regardless of exposure count. This is the usual IPsec-overhead workaround; there is no separate per-tunnel or per-exposure MTU field to configure.

NetworkPolicy posture

The reconciler creates one owned NetworkPolicy per gateway (<name>-nic1, toggled by the operator-wide gatewayNetworkPolicy.enabled config, default on) selecting the gateway pod's labels in the CR's own tenant namespace. It governs only NIC1 — the pod's ordinary cluster network interface. NIC2, the Multus/NAD-attached interface carrying actual tunnel traffic, is a secondary pod interface and entirely outside NetworkPolicy's reach; nothing here restricts IKE/ESP dataplane traffic.

What it does cover, on NIC1:

• Ingress: allowed only to the ipsec-exporter metrics port (9903), from any namespace (the operator doesn't track which namespace runs the cluster's Prometheus scraper) — plus, when the operator's own namespace is known (POD_NAMESPACE), a second rule admitting any port from that namespace.
• Egress: not restricted by this policy at all — PolicyTypes is Ingress-only.

Two honest caveats, both ledgered deviations from the original design intent, not oversights discovered after the fact:

Egress is currently unrestricted (deviation, pending follow-up)

A default-deny egress policy was scoped out of 0.1.0. Correctly admitting everything the gateway pod's containers legitimately need to reach — DNS, the kube-apiserver (status-agent's k8s client), and the tunnel's own OpenStack-routed dataplane egress on NIC2 — without getting any one of those wrong (which fails a tunnel silently rather than loudly) needs the real egress set characterized against a live gateway first. Egress on NIC1 is unrestricted until that follow-up lands.

Operator-namespace ingress on NIC1 is any-port

The second ingress rule (traffic from the operator's own namespace) is not port-scoped — it admits any port, deliberately, as headroom for operator-side tooling against a single-purpose pod. This was a considered design choice, not a placeholder; revisit only if the operator namespace itself becomes a lower-trust boundary than assumed.