Native network segmentation for virtualization workloads

This article demonstrates how you can override the cluster default network in Red Hat OpenShift and connect your workloads using a Layer 2 isolated network, with managed IPAM (IP Address Management). This type of network addresses the virtualization workloads challenges in the Kubernetes platform by providing east-west communication without NAT (Network Address Translation), and stable IPAM configuration during their entire lifecycle. For this, we will introduce the user-defined network (UDN) feature.

Motivation

From a virtualization standpoint, we envision two types of OpenShift users: the traditional virtualization user, who seeks an isolated network without NAT for east-west communication (à la OpenStack Neutron), and the Kubernetes-savvy user, who seeks a fully managed network experience, complete with IPAM. This frees the user from assigning static addresses to their workloads, or having to deploy DHCP (Dynamic Host Configuration Protocol) servers into their networks.

No matter the type of user, both of them share the same goals: they need stable IPAM configuration (IP addresses, gateways, and DNS configuration), and the networking solution must work in both bare metal and cloud platforms.

Problem

In this particular case, it is the Kubernetes platform itself that is getting in the way of achieving the user’s goals—specifically, the Kubernetes networking model. Simply put, it is too opinionated.

It provides a single network to connect all the workloads in the platform. If the user wants to restrict in any way traffic to or from their workloads, the only option available is by provisioning NetworkPolicy. This is expensive for two reasons: someone has to write and maintain those policies, and they have to be reconciled every time something changes in the network. Which happens a lot, since there’s a single network connecting everything.

Adding insult to injury, Kubernetes network policies work in the network and transport OSI layers (layers 3 and 4). Meaning there simply isn’t a Kubernetes native way to perform Layer 2 isolation.

Goals

Now that we understand why and what we’re trying to solve with these isolated Layer 2 networks, let’s enumerate the goals we have for them:

Workload/tenant isolation: The ability to group different types of applications in different isolated networks that cannot talk to each other.
Overlapping subnets: Create multiple networks in your cluster with the same pod subnet range thereby enabling the same setup to be copied.
Native Kubernetes API integration: Full support for Kubernetes network API. This means services, network policies, admin network policies can be used in UDNs.
Native integration with OpenShift network APIs: Support for OpenShift network API. Currently, this means egress IPs can be used in UDNs. There are plans to add support for OpenShift routes.
Stable IPAM configuration: Workloads require their IPs, gateway, and DNS configuration to be stable during their life cycle.
Cloud platform support: Packets must egress the cluster nodes with the IP addresses of the node it runs on, otherwise they are dropped by the cloud provider.
Access to some Kubernetes infrastructure services that are available on the cluster default network: Access to kube-dns and kube-api is still available to workloads attached to UDNs.
No NAT on east-west traffic.

Requirements

OpenShift cluster, version >= 4.18
Red Hat OpenShift Virtualization and MetalLB installed and configured (the latter is required only for the services part). Refer to the OpenShift documentation for instructions.

Use cases

There are two major types of use cases we address via the OpenShift UDN feature: namespace isolation and isolated cluster-wide network. Keep in mind that all the goals noted in the previous section apply to both use cases.

Namespace isolation

In this use case all the workloads in a namespace can access one another, but they are isolated from the rest of the cluster. This means that workloads in another UserDefinedNetwork cannot access them or be reached by them. The same applies to workloads attached to the cluster default network, which cannot reach UDN pods (or vice versa).

You can find a graphical representation of the concept in Figure 1, where communication between the workloads in the green namespace and the workloads in the blue namespace is blocked, but can freely communicate with other pods in the same namespace.

alt text — Figure 1: Connectivity diagram for namespace scoped UserDefinedNetworks.

To achieve this use case, in OpenShift 4.18, the user can provision a UserDefinedNetwork CR, which indicates the namespace’s workloads will be connected by a primary UDN, and workloads on other UDNs (or the cluster’s default network) will not be able to access them.

The namespace’s workloads will be able to egress the cluster and integrate with the Kubernetes features listed in the Goals section. For that, the user must first create the namespace, ensuring to explicitly indicate their interest in overriding the default cluster network with the UDN, by adding the k8s.ovn.org/primary-user-defined-network label.

Refer to the following YAML for a reference manifest:

---
apiVersion: v1
kind: Namespace
metadata:
  name: green
  labels:
	k8s.ovn.org/primary-user-defined-network: ""
---
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: namespace-scoped
  namespace: green
spec:
  topology: Layer2
  layer2:
    role: Primary
    subnets:
      - 192.168.0.0/16
    ipam:
      lifecycle: Persistent

Native network segmentation for virtualization workloads

Share:

Motivation

Problem

Goals

Requirements

Use cases

Namespace isolation

East-west connectivity

Egress to the internet

Seamless live-migration

NetworkPolicy integration

Interconnected namespaces isolation

Services

Conclusion

Products

Build

Quicklinks

Communicate

RED HAT DEVELOPER

Red Hat legal and privacy links

Red Hat legal and privacy links

Report a website issue