Kubernetes egress gateway

The last column is the one people discover late. Every in-cluster option ties the address to the cloud account and region the cluster runs in, so the identity is a property of the infrastructure using it rather than of you.

The closest thing to a purpose-built answer if Cilium is already your CNI. A CiliumEgressGatewayPolicy SNATs matching pod traffic to an address on a node you choose. The comparison page carries the policy YAML.

What is worth knowing before you commit to it is how much has to be true first. Cilium’s own documentation requires both kube-proxy replacement and BPF masquerading to be enabled, and identity allocation to be in crd mode — the kvstore mode is not compatible. A gateway also cannot be selected across a Cluster Mesh: it must live in the same cluster as the pods it serves. And it is documented as incompatible with CiliumEndpointSlice.

Check the edition first, because most write-ups do not. Calico Open Source does not support egress gateways — the feature belongs to Calico Enterprise and Calico Cloud. If your cluster runs Calico OSS today, this option is a purchase, not a configuration.

Where it is available it is the strongest of the in-cluster options operationally. The gateway is a pod rather than a node role, so it schedules like anything else and per-namespace identities are straightforward; source addresses come from a pool you define. It is also the only option here that both owns an address and fails over: gateways are health-checked with ICMP and HTTP probes and traffic is rerouted when one stops answering. Istio’s gateway is a deployment and so is replicable, but it owns no address, so its redundancy is about routing rather than identity.

If you already run Istio the gateway is a deployment you already have, and routing traffic through it is a ServiceEntry and a Gateway away. But Istio routes traffic and does not own an address: the gateway pods still egress via their node, so you need one of the other options underneath it.

Azure’s own project, and the option missing from most comparisons. Pods annotate themselves onto a StaticGatewayConfiguration, and their traffic is carried to dedicated gateway nodes that hold an Azure public IP prefix. Because pods needing different egress addresses share the same gateway nodes and schedule normally, it avoids the usual pattern of dedicating an expensive node per identity.

apiVersion: egressgateway.kubernetes.azure.com/v1alpha1
kind: StaticGatewayConfiguration
metadata:
  name: sgw1
spec:
  gatewayVmssProfile:
    vmssResourceGroup: myResourceGroup   # required
    vmssName: gatewayVMSS                # required
    publicIpPrefixSize: 31               # required
  publicIpPrefixId: /subscriptions/.../publicipprefixes/<name>   # optional
  excludeCIDRs:                          # optional
    - <pod cidr>
    - <service cidr>

metadata:
  annotations:
    kubernetes.azure.com/static-gateway-configuration: sgw1

One annotation is the whole join, which is the nicest thing about it: the workload does not have to know where the gateway is or what address it holds. Note excludeCIDRs — that is the same job NO_PROXY does in the proxy pattern, and forgetting it sends in-cluster traffic out through the gateway and back.

The constraints are the shape of the project: it is Azure only, on AKS or self-managed Kubernetes in Azure, and only Linux nodes (Ubuntu or Azure Linux) are supported. Private IP mode needs AKS 1.34 or newer.

Worth noticing what is under it: Azure’s answer to this problem is a WireGuard tunnel from the pod to a gateway that holds the address. That is the same shape as leasing an address over a tunnel, with the gateway inside your subscription and the address belonging to it.

The provider’s answer and the one their documentation sends you to: private subnets, a default route at a NAT gateway, a reserved address attached. It needs no CNI features and works everywhere, and it is the most expensive of the options to keep running.

On AWS a NAT gateway is about $32 a month per availability zone before data processing charges, billed per region and per VPC. Two availability zones across two regions is four gateways for one identity. Azure NAT Gateway and Google Cloud NAT are billed on the same shape. The address itself costs a few dollars; the gateway around it is the bill.

Each one makes the address a property of the infrastructure using it. That is fine right up until one of these happens:

• You move region or cloud. The address does not come with you, and every partner who allowlisted it has to be asked again — which for a bank or a payment processor is weeks, not a ticket.
• You rebuild the cluster. Same problem, smaller blast radius, and it arrives at the worst possible time because you are usually rebuilding under pressure.
• You change CNI. A Cilium egress policy and a Calico egress gateway are not portable to each other, and neither is portable off Kubernetes.
• Something outside the cluster needs the same identity. A CI runner, a laptop during an incident, a VM that has not been migrated. In-cluster gateways have nothing to offer them.

The alternative is to keep the address outside the infrastructure entirely: lease one that is registered to you and announced from our own autonomous system, and bring it to whatever needs it over a tunnel. On Kubernetes that is either a tunnel sidecar in the pod, or one small egress-proxy pod that the rest of the cluster uses — the same shape as the options above, with the address held somewhere that does not get rebuilt.

It needs no CNI feature, no NAT gateway, no reserved IP and no VPC peering, so it works the same on EKS, GKE, AKS, DigitalOcean, Hetzner and a k3s box under a desk. The manifests are on the Kubernetes setup page. The one requirement is NET_ADMIN on the tunnel container, which on a cluster with a restrictive Pod Security Standard means the baseline profile for that container alone.

If you run Cilium already, the policy is cheap to try and the two failure modes above are the things to test rather than assume: start a new pod and watch what the far side sees for the first second, and drain the gateway node and watch what happens to open connections.

If you run Calico Enterprise, use its egress gateway — it is the only in-cluster option with real failover.

If you are on Azure and staying there, Azure’s kube-egress-gateway is better value than a NAT gateway per zone.

If the address has to outlive the cluster, or be shared with something that is not in it, none of the in-cluster options do that, and that is the case this product exists for. A seven-day trial is a real address on the same network paying customers use, and no card is needed to find out whether the latency is acceptable to your partner.