Configuring Layer 7 Ingress using AKO for vSphere with Tanzu on NSX-T Networking

Overview

NSX ALB (previously known as Avi) offers rich capabilities for L4-L7 load balancing across different clouds and for different workloads. However if you run vSphere with Tanzu (TKGs) on top of NSX-T networking, NSX-T will deploy standard NSX load balancers to offer L4 load balancing for guest Tanzu clusters cluster-api and subsequent loadbalancer services deployed on top of those guest clusters, this includes Ingress services as well. This adds a challenge for customers who would like to utilise NSX Advanced Load Balancer (Avi) for their guest Tanzu clusters running on vSphere supervisor clusters (vsphere with Tanzu).

In this blog post I am going to discuss how customers running vSphere with Tanzu on NSX-T networking can run and make use of NSX ALB (Avi) for providing Layer 7 Ingress using AKO (Avi Kubernetes Operator) to their deployed workloads. This means, that NSX standard load balancer will still provide L4 load balancing for Tanzu clusters, only Avi AKO will be providing layer 7 load balancing (i.e. Ingress).

What is Ingress and Why?!

By default, when you spin up a pod no external access is allowed to containers running inside that pod, to allow external access to pods you need to expose those pods using a Service.

There are 3 service types available in Kubernetes:

• ClusterIP
• NodePort
• Loadbalancer (L4 only)

The first type which is the default is ClusterIP and this allows pod access from within the kubernetes cluster, while NodePort exposes pods ports on the hosting nodes interface and maps container ports to a local host port, users can then access containers using nodeport ip and port combination.

Last type which is Loadbalancer which by default offers standard Layer 4 TCP load balancing across workload pods.

Why Ingress?

Ingress allows users to define complex http/https rules routing criteria (Layer 7 load balancing) based on which the incoming traffic will be routed to a specific container running our application.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. (https://kubernetes.io/docs/concepts/services-networking/ingress/) It is important to highlight that Ingress does not expose any container access, so a backend service (one of the mentioned above) needs to be configured before configuring Ingress.

Lab Inventory

For software versions I used the following:

• VMware ESXi 8.0GA
• vCenter server version 8.0GA
• NSX-T 4.0.0.1
• TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
• VyOS 1.4 used as lab backbone router and DHCP server.
• Ubuntu 20.04 LTS as bootstrap machine.
• Ubuntu 20.04.2 LTS as DNS and internet gateway.
• Windows 10 pro as RDP Jumpbox.
• NSX ALB controller 22.1.2
• AKO version 1.8.2

For virtual hosts and appliances sizing I used the following specs:

• 3 x virtualised ESXi hosts each with 12 vCPUs, 2 x NICs and 128 GB RAM.
• vCenter server appliance with 2 vCPU and 24 GB RAM.

Reference Architecture

All the logical segments and T1 routers under the supervisor namespace are created by WCP as part of enabling workload management on a vCenter cluster, while the Avi Service Engine domain logical segments and T1 need to be created and configured by the user. Make sure that Avi Se management segment has access to Avi controller in your lab setup and also make sure that Avi service engines can reach Tanzu cluster nodes, either by connecting Avi-Tier1 to same T0 gateway that is used for T1 created for Tanzu workload cluster or another T0 but ensure that route advertisement is working properly.

Below is a summary of the networks I used and shown above in the reference architecture:

10.244.0.0/28 is subnet from the defined pod CIDR (10.244.0.0/16) which was defined while enabling workload management (WCP) and used to assign IPs to supervisor cluster nodes.

10.244.0.80/28 is subnet from the defined pod CIDR (10.244.0.0/16) which was defined while enabling workload management (WCP) and used to assign IPs to workload cluster Sumi-Gaeshi.

172.10.110.0/24 is an Ingress pool which was also defined while enabling workload management on cluster (by WCP) and used by NSX-T to assign IPs to ingress and load balancer services.

172.10.110.248/29 (gateway interface in Avi-Tier1 logical segment is 172.10.110.254/29) is a subset from the above pool which I will use for Avi VIP and Se data network, I used this subnet since it is already routed by our T0 and thus can be accessed from external network without extra routing.

172.10.120.0/24 is an IP address pool created by NSX-T (also defined while enabling workload management) and is used by NSX-T as Egress IPs (to assign IPs for NAT rules).

10.67.100.0/24 (gateway interface in Avi-Tier1 logical segment is 10.67.100.254/24) is a subnet manually created and will be used to connect Avi service engine management interfaces. This subnet need to be able to reach Avi controller IP, so either advertise it through Avi-Tier1 to T0 and then to external network or add a SNAT rule to Avi-Tier1. I used the latter option and I source natted 10.67.100.0/24 to one of egress IPs from egress pool 172.10.120.0/24.

Deployment Workflow

• A vCenter Cluster with vSphere with Tanzu enabled (workload management) with NSX networking.
• Create a guest Tanzu Cluster on vSphere with Tanzu
• Create a dedicated T1 logical router for Avi service engine segments on NSX.
• Create Avi management and data network logical segments on NSX.
• Adjust NSX DFW and NAT rules.
• Add NSX-T instance as cloud provider in NSX ALB (Avi).
• Deploy AKO in one of Tanzu guest clusters.
• Create a layer 7 Ingress using AKO and verify service engine and virtual service creation in Avi.

vSphere with Tanzu enabled Cluster

The focus of this blog post on how to use Avi AKO on Tanzu guest clusters which are already deployed on top of NSX networking and utilising NSX standard load balancers, so I will not cover the steps of enabling workload management in this post, however you can review some of my previous blog posts if you want to learn more about this topic.

• Enable Workload Management with NSX-T Networking.

To create a guest Tanzu cluster, login to your supervisor cluster using the command:

kubectl vsphere login --server=https://<supervisor cluster IP> --insecure-skip-tls-verify -u administrator@vsphere.local

I used the following YAML to deploy my guest cluster

apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
  name: sumi-gaeshi
  namespace: homelab
spec:
  topology:
    controlPlane:
      replicas: 1
      vmClass: best-effort-small
      storageClass: k8s-storage
      volumes:
        - name: etcd
          mountPath: /var/lib/etcd
          capacity:
            storage: 20Gi
      tkr:
        reference:
          name: v1.22.9---vmware.1-tkg.1.cc71bc8
    nodePools:
    - name: worker-pool01
      replicas: 2
      vmClass: best-effort-xlarge
      storageClass: k8s-storage
      volumes:
        - name: containerd
          mountPath: /var/lib/containerd
          capacity:
            storage: 20Gi
      tkr:
        reference:
          name: v1.22.9---vmware.1-tkg.1.cc71bc8
  settings:
    storage:
      defaultClass: k8s-storage
    network:
      cni:
        name: antrea
      services:
        cidrBlocks: ["198.53.100.0/16"]
      pods:
        cidrBlocks: ["192.0.5.0/16"]
      serviceDomain: cluster.local

Make sure that you have a namespace created under Workload Management before applying (kubectl apply -f filename.yml) the above file (in my case it is called homelab). Wait till your guest cluster is deployed then login to it using:

kubectl-vsphere login --server=https://172.10.110.2 --insecure-skip-tls-verify -u administrator@vsphere.local --tanzu-kubernetes-cluster-name sumi-gaeshi --tanzu-kubernetes-cluster-namespace homelab

verify all nodes are in Ready state

NSX-T Configuration

Step 1: Create dedicated  T1 gateway for Avi service engines

Login to your NSX-T instance and deploy a T1 and connected your T1 to T0 gateway, make sure that your T0 gateway redistributes connected T1 subnets to external network so that Avi service engines can reach Avi controller and vice versa. I am not going to cover how to created a T0 and T1 logical routers in this blog post for two reasons:

• If you do not know how to configure a T0 and T1 gateways on NSX-T then probably this post is not for you.
• VMware documentation covers this topic quite good, adding a T0 gateway and adding a T1 gateway.

After you create Avi-Tier1 T1 gateway, make sure to configure route advertisement (connected segments are a must, NAT is only needed if you NAT Avi segments subnets)

 

Step 2: Create Avi service engine management and data network logical segments

Next step is to create the two logical segments shown above in the reference architecture under Avi Service Engine domain, logical segment Avi-mgmt is an overlay logical segment which will be used to connect SE management interface and logical segment Avi-vip is also an overlay segment which will be used to connect SE data network interfaces.

• Creating logical segments can be referenced in VMware NSX documentation. For both segments I am using Overlay Transport zone.

So under Segments configuration in my NSX segments l highlighted the two segments configured for Avi SE management and data networks

If you check network topology from NSX, this is how my setup looks like

Step 3: Modify DFW to allow SE data network to Tanzu cluster nodes

When a guest Tanzu workload cluster is deployed, WCP process creates a T1 gateway and connects workload cluster nodes to it and adds specific DFW rules to allow only needed traffic to workload cluster nodes. This results in a Drop ANY ANY rule at the bottom of DFW rules which will cause Avi service engines not to be able to connect to workload cluster nodes, to resolve that we need to add an ALLOW DFW rule before the DFW rules policy created by WCP, to allow communication to workload cluster node network (10.244.0.80/28).

NSX ALB (Avi) configuration 

Step 1: Add NSX and vCenter user credentials 

In order to be able to use NSX ALB AKO to provide layer 7 ingress to Tanzu guest clusters, we need to integrate NSX-T instance as cloud provider in Avi. Remember, that for layer 4 Tanzu clusters load balancing services this will still be handled by NSX standard load balancers, but for layer 7 load balancers (ingress) this will be handled by Avi.

To add NSX-T as cloud in Avi, we need first to add NSX-T and vCenter credentials (on which NSX-T is connected) to Avi in order to authenticate to both clouds. To configure user credentials from Avi UI navigate to Administration and click on User Credentials and then CREATE

Add credentials of your NSX-T Manager instance

Create another credential to your vCenter server instance as well.

Step 2: Add NSX-T Cloud to Avi as cloud provider

Once you have created both NSX-T and vCenter credentials, you need to add NSX-T cloud to Avi, from Avi UI navigate to Infrastructure > Clouds and then click on CREATE NSX-T Cloud

Fill in your NSX-T parameters and under Management Network specify T1 and Logical Segment we created for Avi SE management earlier:

In the above screenshot I specified my overlay transport zone and which T1 and logical segment Avi should attach SE management interfaces to. 

Scroll further down, define which data network Avi should use along with adding your vCenter server instance.

We will need to create an IPAM profile but this will be done after saving the cloud configuration so we can use the SE data network as useable network for VIP placement. I did not define a DNS profile since I will be manually creating my ingress host names in my DNS server, if you would like to have Avi create Ingress DNS hostnames for you in your DNS then you need to configure a DNS profile on Avi and allow DNS zone delegation for that in your DNS server. For now, click SAVE. Avi will take couple of seconds registering and collecting some inventory from NSX-T and if all is good you should see NSX-T cloud list with a green circle indicating that connection and registration status are good:

Last step is to create an IPAM profile and define it under NSX-T Cloud instance we just created, from Avi UI navigate to Templates > IPAM/DNS Profiles and CREATE a new IPAM profile

Choose cloud to be your NSX-T cloud (nsx-l-01a) and then choose useable network Avi-vip

Click SAVE, and once created go back and add IPAM profile to our NSX-T which we created step earlier. Next, we need to tell Avi how IP address management on Avi-mgmt and Avi-vip segments will be handled, this either can be using DHCP (enabling DHCP on NSX-T logical segments side) or by using static IP pool allocation in Avi. I have to say, I was not able to get Avi to acquire DHCP addresses for SE management and data networks and ended up using IP allocation, I believe your DHCP server needs to support a specific option for that work so if you figure it out please ping me and share your tips.

To configure static IP pool allocation on Avi, navigate to Infrastructure > Cloud Resources > Networks and make sure that you select your NSX-T cloud from drop-down list:

Click on the small pencil icon on the right and add a subnet and a static IP pool for your service engines data network and VIP placement

Repeat the same for your management network logical segment (Avi-mgmt).

Step 3: configure Avi routing 

As last step in Avi configuration, we need to setup routing since we need to make sure that Avi service engine management interfaces can reach Avi controller. In addition, Avi service engines need to be able to reach Tanzu Cluster nodes (network 10.244.0.80.28) through data network. So we will be adding two default routes on Avi pointing to Avi-Tier1 GW IP as next-hop for both networks. Avi works with the concept of routing contexts (VRFs) so there is routing context for management interfaces and another routing context for data network (data interfaces). For management network, you can configure default route from Avi command line as follows:

shell 

switchto cloud nsx-l-01a 

terminal mode linux_command_line 

configure vrfcontext management --static_routes.1.prefix 0.0.0.0/0 --static_routes.1.next_hop 10.67.100.254 --static_routes.1.route_id 1 

For Avi data network, you can also use command line (setting context to be Avi-vip) and set default route to Tier1 interface

Deploy AKO to Tanzu Workload Cluster

We will be deploying AKO version 1.8.2 (compatible with the versions of Avi controller, NSX-T and vCenter in my home lab) using helm ako chart. If you do not have Helm installed on your machine then you will need first to install Helm following the steps HERE.

Step 1: Download and modify AKO values yaml file

From my Linux machine, I used the following commands to add AKO chart and search for the available AKO version

helm repo add ako https://projects.registry.vmware.com/chartrepo/ako
helm search repo
helm show values ako/ako --version 1.8.2 > ~/ako-values.yml

The last command downloads the values yaml file to our local machine to a file called ako-1.8.2-values.yml, this is AKO’s configuration file where we need to set our environment parameters in addition to how AKO will function. From a text editor open the values file and modify the following sections to match your environment:

You can get the Tier1 API path by logging in NSX-T and navigating to T1 Gateways, click on the 3 vertical dots and choose “Copy Path to Clipboard” paste this to the value of nsxtT1LR shown above.

Last section in the AKO values file is to add controller settings:

Moving down to end of file, add your Avi controller credentials

Save and exit the file.

Step 2: Deploy AKO Pod 

Once all the values mentioned above are correctly modified in the values file, run the following command to create a namespace to host AKO pod and start AKO pod deployment:

kubectl create ns avi-system

helm install  ako/ako  --generate-name --version 1.8.2 -f ako-1.8.2-values.yml --namespace=avi-system

If the deployment is successful you should see the AKO pod in Running state

Step 3: Deploy a demo App and Ingress service to test AKO

To test AKO layer 7 ingress, I created the below test demo application on my Sumi-Gaeshi Tanzu workload cluster:

# This should work on Kubernetes deployments that have LoadBalancer support
apiVersion: v1
kind: Service
metadata:
  name: redis-server
  labels:
    app: redis-server
    tier: cache
spec:
  type: ClusterIP
  ports:
  - port: 6379
  selector:
    app: redis-server
    tier: cache
---
apiVersion: v1
kind: Service
metadata:
  name: yelb-db
  labels:
    app: yelb-db
    tier: backenddb
spec:
  type: ClusterIP
  ports:
  - port: 5432
  selector:
    app: yelb-db
    tier: backenddb
---
apiVersion: v1
kind: Service
metadata:
  name: yelb-appserver
  labels:
    app: yelb-appserver
    tier: middletier
spec:
  type: ClusterIP
  ports:
  - port: 4567
  selector:
    app: yelb-appserver
    tier: middletier
---
apiVersion: v1
kind: Service
metadata:
  name: yelb-ui
  labels:
    app: yelb-ui
    tier: frontend
spec:
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: yelb-ui
    tier: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yelb-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: yelb-ui
      tier: frontend
  template:
    metadata:
      labels:
        app: yelb-ui
        tier: frontend
    spec:
      containers:
      - name: yelb-ui
        image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-ui@sha256:2844e7a275c918fb799da14f1964c620585ee233b302ffabc00d319cf4bf5c29
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-server
      tier: cache
  template:
    metadata:
      labels:
        app: redis-server
        tier: cache
    spec:
      containers:
      - name: redis-server
        image: harbor-repo.vmware.com/challagandlp/mreferre/redis@sha256:3c07847e5aa6911cf5d9441642769d3b6cd0bf6b8576773ae3a0742056b9dd47
        ports:
        - containerPort: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yelb-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: yelb-db
      tier: backenddb
  template:
    metadata:
      labels:
        app: yelb-db
        tier: backenddb
    spec:
      containers:
      - name: yelb-db
        image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-db@sha256:6412d2fe96ee71ca701932d47675c549fe0428dede6a7975d39d9a581dc46c0c
        ports:
        - containerPort: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yelb-appserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: yelb-appserver
      tier: middletier
  template:
    metadata:
      labels:
        app: yelb-appserver
        tier: middletier
    spec:
      containers:
      - name: yelb-appserver
        image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-db@sha256:6412d2fe96ee71ca701932d47675c549fe0428dede6a7975d39d9a581dc46c0c
        ports:
        - containerPort: 4567

Apply the above YAML file, this will create a demo App called yelb with a nodeport service exposing frontend pod UI

I then created a layer 7 ingress with the above nodeport as a backend service, my ingress will have a DNS (host) name sumi-ingress.nsxbaas.homelab I created this entry as well in my DNS server

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ako-ingress
spec:
  ingressClassName: avi-lb
  rules:
  - host: sumi-ingress.nsxbaas.homelab
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: yelb-ui-svc
            port:
              number: 80

Apply the above file, AKO will then kick in and if it is your first ingress, Avi will take sometime to create and configure service engine in vCenter. If all goes fine you should be able to see that your ingress service has got an IP from the 172.10.110.248/29 address pool

From Avi UI, you should be able to see virtual service created successfully mapping to our Tanzu workload cluster nodes:

You can also verify that opening a web browser to http://sumi-ingress.nsxbaas.homelab redirects you to yelb homepage

Hope you have found this post useful!