
Overview
NSX ALB (previously known as Avi) offers rich capabilities for L4-L7 load balancing across different clouds and for different workloads. However if you run vSphere with Tanzu (TKGs) on top of NSX-T networking, NSX-T will deploy standard NSX load balancers to offer L4 load balancing for guest Tanzu clusters cluster-api and subsequent loadbalancer services deployed on top of those guest clusters, this includes Ingress services as well. This adds a challenge for customers who would like to utilise NSX Advanced Load Balancer (Avi) for their guest Tanzu clusters running on vSphere supervisor clusters (vsphere with Tanzu).
In this blog post I am going to discuss how customers running vSphere with Tanzu on NSX-T networking can run and make use of NSX ALB (Avi) for providing Layer 7 Ingress using AKO (Avi Kubernetes Operator) to their deployed workloads. This means, that NSX standard load balancer will still provide L4 load balancing for Tanzu clusters, only Avi AKO will be providing layer 7 load balancing (i.e. Ingress).
What is Ingress and Why?!
By default, when you spin up a pod no external access is allowed to containers running inside that pod, to allow external access to pods you need to expose those pods using a Service.
There are 3 service types available in Kubernetes:
- ClusterIP
- NodePort
- Loadbalancer (L4 only)
The first type which is the default is ClusterIP and this allows pod access from within the kubernetes cluster, while NodePort exposes pods ports on the hosting nodes interface and maps container ports to a local host port, users can then access containers using nodeport ip and port combination.
Last type which is Loadbalancer which by default offers standard Layer 4 TCP load balancing across workload pods.
Why Ingress?
Ingress allows users to define complex http/https rules routing criteria (Layer 7 load balancing) based on which the incoming traffic will be routed to a specific container running our application.
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. (https://kubernetes.io/docs/concepts/services-networking/ingress/) It is important to highlight that Ingress does not expose any container access, so a backend service (one of the mentioned above) needs to be configured before configuring Ingress.
Lab Inventory
For software versions I used the following:
- VMware ESXi 8.0GA
- vCenter server version 8.0GA
- NSX-T 4.0.0.1
- TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
- VyOS 1.4 used as lab backbone router and DHCP server.
- Ubuntu 20.04 LTS as bootstrap machine.
- Ubuntu 20.04.2 LTS as DNS and internet gateway.
- Windows 10 pro as RDP Jumpbox.
- NSX ALB controller 22.1.2
- AKO version 1.8.2
For virtual hosts and appliances sizing I used the following specs:
- 3 x virtualised ESXi hosts each with 12 vCPUs, 2 x NICs and 128 GB RAM.
- vCenter server appliance with 2 vCPU and 24 GB RAM.
Reference Architecture
All the logical segments and T1 routers under the supervisor namespace are created by WCP as part of enabling workload management on a vCenter cluster, while the Avi Service Engine domain logical segments and T1 need to be created and configured by the user. Make sure that Avi Se management segment has access to Avi controller in your lab setup and also make sure that Avi service engines can reach Tanzu cluster nodes, either by connecting Avi-Tier1 to same T0 gateway that is used for T1 created for Tanzu workload cluster or another T0 but ensure that route advertisement is working properly.
Below is a summary of the networks I used and shown above in the reference architecture:
10.244.0.0/28 is subnet from the defined pod CIDR (10.244.0.0/16) which was defined while enabling workload management (WCP) and used to assign IPs to supervisor cluster nodes.
10.244.0.80/28 is subnet from the defined pod CIDR (10.244.0.0/16) which was defined while enabling workload management (WCP) and used to assign IPs to workload cluster Sumi-Gaeshi.
172.10.110.0/24 is an Ingress pool which was also defined while enabling workload management on cluster (by WCP) and used by NSX-T to assign IPs to ingress and load balancer services.
172.10.110.248/29 (gateway interface in Avi-Tier1 logical segment is 172.10.110.254/29) is a subset from the above pool which I will use for Avi VIP and Se data network, I used this subnet since it is already routed by our T0 and thus can be accessed from external network without extra routing.
172.10.120.0/24 is an IP address pool created by NSX-T (also defined while enabling workload management) and is used by NSX-T as Egress IPs (to assign IPs for NAT rules).
10.67.100.0/24 (gateway interface in Avi-Tier1 logical segment is 10.67.100.254/24) is a subnet manually created and will be used to connect Avi service engine management interfaces. This subnet need to be able to reach Avi controller IP, so either advertise it through Avi-Tier1 to T0 and then to external network or add a SNAT rule to Avi-Tier1. I used the latter option and I source natted 10.67.100.0/24 to one of egress IPs from egress pool 172.10.120.0/24.
Deployment Workflow
- A vCenter Cluster with vSphere with Tanzu enabled (workload management) with NSX networking.
- Create a guest Tanzu Cluster on vSphere with Tanzu
- Create a dedicated T1 logical router for Avi service engine segments on NSX.
- Create Avi management and data network logical segments on NSX.
- Adjust NSX DFW and NAT rules.
- Add NSX-T instance as cloud provider in NSX ALB (Avi).
- Deploy AKO in one of Tanzu guest clusters.
- Create a layer 7 Ingress using AKO and verify service engine and virtual service creation in Avi.
vSphere with Tanzu enabled Cluster
The focus of this blog post on how to use Avi AKO on Tanzu guest clusters which are already deployed on top of NSX networking and utilising NSX standard load balancers, so I will not cover the steps of enabling workload management in this post, however you can review some of my previous blog posts if you want to learn more about this topic.
To create a guest Tanzu cluster, login to your supervisor cluster using the command:
kubectl vsphere login --server=https://<supervisor cluster IP> --insecure-skip-tls-verify -u administrator@vsphere.local
I used the following YAML to deploy my guest cluster
apiVersion: run.tanzu.vmware.com/v1alpha2 kind: TanzuKubernetesCluster metadata: name: sumi-gaeshi namespace: homelab spec: topology: controlPlane: replicas: 1 vmClass: best-effort-small storageClass: k8s-storage volumes: - name: etcd mountPath: /var/lib/etcd capacity: storage: 20Gi tkr: reference: name: v1.22.9---vmware.1-tkg.1.cc71bc8 nodePools: - name: worker-pool01 replicas: 2 vmClass: best-effort-xlarge storageClass: k8s-storage volumes: - name: containerd mountPath: /var/lib/containerd capacity: storage: 20Gi tkr: reference: name: v1.22.9---vmware.1-tkg.1.cc71bc8 settings: storage: defaultClass: k8s-storage network: cni: name: antrea services: cidrBlocks: ["198.53.100.0/16"] pods: cidrBlocks: ["192.0.5.0/16"] serviceDomain: cluster.local
Make sure that you have a namespace created under Workload Management before applying (kubectl apply -f filename.yml) the above file (in my case it is called homelab). Wait till your guest cluster is deployed then login to it using:
kubectl-vsphere login --server=https://172.10.110.2 --insecure-skip-tls-verify -u administrator@vsphere.local --tanzu-kubernetes-cluster-name sumi-gaeshi --tanzu-kubernetes-cluster-namespace homelab
verify all nodes are in Ready state
NSX-T Configuration
Step 1: Create dedicated T1 gateway for Avi service engines
Login to your NSX-T instance and deploy a T1 and connected your T1 to T0 gateway, make sure that your T0 gateway redistributes connected T1 subnets to external network so that Avi service engines can reach Avi controller and vice versa. I am not going to cover how to created a T0 and T1 logical routers in this blog post for two reasons:
- If you do not know how to configure a T0 and T1 gateways on NSX-T then probably this post is not for you.
- VMware documentation covers this topic quite good, adding a T0 gateway and adding a T1 gateway.
After you create Avi-Tier1 T1 gateway, make sure to configure route advertisement (connected segments are a must, NAT is only needed if you NAT Avi segments subnets)
Step 2: Create Avi service engine management and data network logical segments
Next step is to create the two logical segments shown above in the reference architecture under Avi Service Engine domain, logical segment Avi-mgmt is an overlay logical segment which will be used to connect SE management interface and logical segment Avi-vip is also an overlay segment which will be used to connect SE data network interfaces.
- Creating logical segments can be referenced in VMware NSX documentation. For both segments I am using Overlay Transport zone.
So under Segments configuration in my NSX segments l highlighted the two segments configured for Avi SE management and data networks
If you check network topology from NSX, this is how my setup looks like
Step 3: Modify DFW to allow SE data network to Tanzu cluster nodes
When a guest Tanzu workload cluster is deployed, WCP process creates a T1 gateway and connects workload cluster nodes to it and adds specific DFW rules to allow only needed traffic to workload cluster nodes. This results in a Drop ANY ANY rule at the bottom of DFW rules which will cause Avi service engines not to be able to connect to workload cluster nodes, to resolve that we need to add an ALLOW DFW rule before the DFW rules policy created by WCP, to allow communication to workload cluster node network (10.244.0.80/28).
NSX ALB (Avi) configuration
Step 1: Add NSX and vCenter user credentials
In order to be able to use NSX ALB AKO to provide layer 7 ingress to Tanzu guest clusters, we need to integrate NSX-T instance as cloud provider in Avi. Remember, that for layer 4 Tanzu clusters load balancing services this will still be handled by NSX standard load balancers, but for layer 7 load balancers (ingress) this will be handled by Avi.
To add NSX-T as cloud in Avi, we need first to add NSX-T and vCenter credentials (on which NSX-T is connected) to Avi in order to authenticate to both clouds. To configure user credentials from Avi UI navigate to Administration and click on User Credentials and then CREATE
Add credentials of your NSX-T Manager instance
Create another credential to your vCenter server instance as well.
Step 2: Add NSX-T Cloud to Avi as cloud provider
Once you have created both NSX-T and vCenter credentials, you need to add NSX-T cloud to Avi, from Avi UI navigate to Infrastructure > Clouds and then click on CREATE NSX-T Cloud
Fill in your NSX-T parameters and under Management Network specify T1 and Logical Segment we created for Avi SE management earlier:
In the above screenshot I specified my overlay transport zone and which T1 and logical segment Avi should attach SE management interfaces to.
Scroll further down, define which data network Avi should use along with adding your vCenter server instance.
We will need to create an IPAM profile but this will be done after saving the cloud configuration so we can use the SE data network as useable network for VIP placement. I did not define a DNS profile since I will be manually creating my ingress host names in my DNS server, if you would like to have Avi create Ingress DNS hostnames for you in your DNS then you need to configure a DNS profile on Avi and allow DNS zone delegation for that in your DNS server. For now, click SAVE. Avi will take couple of seconds registering and collecting some inventory from NSX-T and if all is good you should see NSX-T cloud list with a green circle indicating that connection and registration status are good:
Last step is to create an IPAM profile and define it under NSX-T Cloud instance we just created, from Avi UI navigate to Templates > IPAM/DNS Profiles and CREATE a new IPAM profile
Choose cloud to be your NSX-T cloud (nsx-l-01a) and then choose useable network Avi-vip
Click SAVE, and once created go back and add IPAM profile to our NSX-T which we created step earlier. Next, we need to tell Avi how IP address management on Avi-mgmt and Avi-vip segments will be handled, this either can be using DHCP (enabling DHCP on NSX-T logical segments side) or by using static IP pool allocation in Avi. I have to say, I was not able to get Avi to acquire DHCP addresses for SE management and data networks and ended up using IP allocation, I believe your DHCP server needs to support a specific option for that work so if you figure it out please ping me and share your tips.
To configure static IP pool allocation on Avi, navigate to Infrastructure > Cloud Resources > Networks and make sure that you select your NSX-T cloud from drop-down list:
Click on the small pencil icon on the right and add a subnet and a static IP pool for your service engines data network and VIP placement
Repeat the same for your management network logical segment (Avi-mgmt).
Step 3: configure Avi routing
As last step in Avi configuration, we need to setup routing since we need to make sure that Avi service engine management interfaces can reach Avi controller. In addition, Avi service engines need to be able to reach Tanzu Cluster nodes (network 10.244.0.80.28) through data network. So we will be adding two default routes on Avi pointing to Avi-Tier1 GW IP as next-hop for both networks. Avi works with the concept of routing contexts (VRFs) so there is routing context for management interfaces and another routing context for data network (data interfaces). For management network, you can configure default route from Avi command line as follows:
shell switchto cloud nsx-l-01a terminal mode linux_command_line configure vrfcontext management --static_routes.1.prefix 0.0.0.0/0 --static_routes.1.next_hop 10.67.100.254 --static_routes.1.route_id 1
For Avi data network, you can also use command line (setting context to be Avi-vip) and set default route to Tier1 interface
Deploy AKO to Tanzu Workload Cluster
We will be deploying AKO version 1.8.2 (compatible with the versions of Avi controller, NSX-T and vCenter in my home lab) using helm ako chart. If you do not have Helm installed on your machine then you will need first to install Helm following the steps HERE.
Step 1: Download and modify AKO values yaml file
From my Linux machine, I used the following commands to add AKO chart and search for the available AKO version
helm repo add ako https://projects.registry.vmware.com/chartrepo/ako helm search repo helm show values ako/ako --version 1.8.2 > ~/ako-values.yml
The last command downloads the values yaml file to our local machine to a file called ako-1.8.2-values.yml, this is AKO’s configuration file where we need to set our environment parameters in addition to how AKO will function. From a text editor open the values file and modify the following sections to match your environment:
You can get the Tier1 API path by logging in NSX-T and navigating to T1 Gateways, click on the 3 vertical dots and choose “Copy Path to Clipboard” paste this to the value of nsxtT1LR shown above.
Last section in the AKO values file is to add controller settings:
Moving down to end of file, add your Avi controller credentials
Save and exit the file.
Step 2: Deploy AKO Pod
Once all the values mentioned above are correctly modified in the values file, run the following command to create a namespace to host AKO pod and start AKO pod deployment:
kubectl create ns avi-system helm install ako/ako --generate-name --version 1.8.2 -f ako-1.8.2-values.yml --namespace=avi-system
If the deployment is successful you should see the AKO pod in Running state
Step 3: Deploy a demo App and Ingress service to test AKO
To test AKO layer 7 ingress, I created the below test demo application on my Sumi-Gaeshi Tanzu workload cluster:
# This should work on Kubernetes deployments that have LoadBalancer support apiVersion: v1 kind: Service metadata: name: redis-server labels: app: redis-server tier: cache spec: type: ClusterIP ports: - port: 6379 selector: app: redis-server tier: cache --- apiVersion: v1 kind: Service metadata: name: yelb-db labels: app: yelb-db tier: backenddb spec: type: ClusterIP ports: - port: 5432 selector: app: yelb-db tier: backenddb --- apiVersion: v1 kind: Service metadata: name: yelb-appserver labels: app: yelb-appserver tier: middletier spec: type: ClusterIP ports: - port: 4567 selector: app: yelb-appserver tier: middletier --- apiVersion: v1 kind: Service metadata: name: yelb-ui labels: app: yelb-ui tier: frontend spec: type: NodePort ports: - port: 80 protocol: TCP targetPort: 80 selector: app: yelb-ui tier: frontend --- apiVersion: apps/v1 kind: Deployment metadata: name: yelb-ui spec: replicas: 1 selector: matchLabels: app: yelb-ui tier: frontend template: metadata: labels: app: yelb-ui tier: frontend spec: containers: - name: yelb-ui image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-ui@sha256:2844e7a275c918fb799da14f1964c620585ee233b302ffabc00d319cf4bf5c29 ports: - containerPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: redis-server spec: replicas: 1 selector: matchLabels: app: redis-server tier: cache template: metadata: labels: app: redis-server tier: cache spec: containers: - name: redis-server image: harbor-repo.vmware.com/challagandlp/mreferre/redis@sha256:3c07847e5aa6911cf5d9441642769d3b6cd0bf6b8576773ae3a0742056b9dd47 ports: - containerPort: 6379 --- apiVersion: apps/v1 kind: Deployment metadata: name: yelb-db spec: replicas: 1 selector: matchLabels: app: yelb-db tier: backenddb template: metadata: labels: app: yelb-db tier: backenddb spec: containers: - name: yelb-db image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-db@sha256:6412d2fe96ee71ca701932d47675c549fe0428dede6a7975d39d9a581dc46c0c ports: - containerPort: 5432 --- apiVersion: apps/v1 kind: Deployment metadata: name: yelb-appserver spec: replicas: 1 selector: matchLabels: app: yelb-appserver tier: middletier template: metadata: labels: app: yelb-appserver tier: middletier spec: containers: - name: yelb-appserver image: harbor-repo.vmware.com/challagandlp/mreferre/yelb-db@sha256:6412d2fe96ee71ca701932d47675c549fe0428dede6a7975d39d9a581dc46c0c ports: - containerPort: 4567
Apply the above YAML file, this will create a demo App called yelb with a nodeport service exposing frontend pod UI
I then created a layer 7 ingress with the above nodeport as a backend service, my ingress will have a DNS (host) name sumi-ingress.nsxbaas.homelab I created this entry as well in my DNS server
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ako-ingress spec: ingressClassName: avi-lb rules: - host: sumi-ingress.nsxbaas.homelab http: paths: - path: / pathType: Prefix backend: service: name: yelb-ui-svc port: number: 80
Apply the above file, AKO will then kick in and if it is your first ingress, Avi will take sometime to create and configure service engine in vCenter. If all goes fine you should be able to see that your ingress service has got an IP from the 172.10.110.248/29 address pool
From Avi UI, you should be able to see virtual service created successfully mapping to our Tanzu workload cluster nodes:
You can also verify that opening a web browser to http://sumi-ingress.nsxbaas.homelab redirects you to yelb homepage
Hope you have found this post useful!
Pingback: Insight into positioning NSX ALB (Avi) with VMware Tanzu Offerings - nsxbaas