Monitoring Tanzu Kubernetes Clusters using vROPs

Key day two operations for an enterprise running Tanzu and/or Kubernetes clusters would typically include automated discovery, monitoring and troubleshooting of management and workload clusters. With the radical and exponential increase in the amount of traffic, interconnected flows and relationships between containerised workloads, the need of powerful and insightful monitoring tools is greater than ever.

vRealize suite products like vRealize Operations and vRealize Network Insight provide insightful and automated discovery modes for your Tanzu and upstream Kubernetes clusters. This allows you to monitor your containerised workloads and clusters from a single pane of glass alongside the rest of your infrastructure.

In this blog post I am going to walk you through monitoring Tanzu Kubernetes clusters using vROPs for Kubernetes management pack (previously known as management pack for containers monitoring). If you are interested in monitoring Tanzu and Kubernetes clusters using vRealize Network Insight then you can check my previous blog posts HERE and HERE.

Lab Inventory

For software versions I used the following:

  • VMware ESXi 7.0U3c
  • vCenter server version 7.0U3
  • TrueNAS 12.0-U7 used to provision NFS datastores to ESXi hosts.
  • VyOS 1.4 used as lab backbone router and DHCP server.
  • Ubuntu 18.04 LTS as jumpbox server.
  • Ubuntu 20.04.2 LTS as DNS and internet gateway.
  • Windows Server 2012 R2 Datacenter as management host for UI access.
  • vRealize Operations version 8.6.3 with management pack for Kubernetes 1.7

For virtual hosts and appliances sizing I used the following specs:

  • 3 x virtualised ESXi hosts each with 8 vCPUs, 2 x NICs and 64 GB RAM.
  • vCenter server appliance with 2 vCPU and 24 GB RAM.

Installing vROPs management pack for Kubernetes

vROPs management pack for Kubernetes allows vROPs to monitor your Tanzu/Kubernetes clusters regardless where they run. vRealize Operations Manager can monitor multiple Kubernetes solutions, whether it is VMware Tanzu Kubernetes Grid (TKG), RedHat OpenShift, or Kubernetes on Amazon Web Services EC2, Azure, or Google Virtual Machines.

Step 1: Download and add management pack to vROPs

Please note, if you are using vSphere with Kubernetes to run your containerised workloads then you do not need to use management pack for Kubernetes in vROPs, since workload management monitoring is natively supported and added in vROPs and added automatically when you add your vCenter server as data source.

To download the management pack, navigate to VMware Market place at https://marketplace.cloud.vmware.com/ and make sure that you have an account created so you can download management packs.

In the search field, search for “management pack for kubernetes 1.7” and then click on Download to download the pak file.

Login to vROPs using an admin privileged account, then from the left menu, click Data Sources > Integrations. Go to the Repository tab in the right pane and on the Repository tab, click Add.

Browse to the location where you have downloaded the pak file, choose it and then click on upload (this may take several minutes). Once the upload is successful you should see the below:

Read and accept the EULA, and click Next, when vRealize Operations Management Pack for Kubernetes is installed, click Finish.

Step 2: Preparing Kubernetes Cluster for monitoring

There are different methods by which vROPS management pack for Kubernetes can pull monitoring metrics and resource utilisation from Tanzu/K8s clusters, the one I will use in this blog post is the cadvisor open source agent. cAdvisor is an open-source agent that monitors resource usage and analyses the performance of containers. It collects statistics about the CPU, memory, file, and network usage for all containers.

Install cadvisor pods using cadvisor YAML

Login to your Tanzu cluster that you would like to monitor and paste the contents of the cadvisor YAML file from HERE in a file called cadvisor.yaml and create it on your Tanzu/k8s cluster using the command:

kubectl create -f cadvisor.yaml

You do not need to modify anything in that YAML file if your docker directories are in default locations and no specific port number changes are required.

cadvisor system pods will be created under kube-system namespace and should talk around 3 to 4 minutes to deploy and become running, check the status of the created pods using the command:

kubectl get pods -n kube-system

Wait till the above pods are in Running state then proceed with the following steps.

Adding Tanzu Kubernetes clusters in vROPs

After the cadvisor pods are initialised and in running state, copy and paste the below information in an external file as you will need them when adding your Tanzu or Kubernetes clusters in vROPs.

From your bootstrap machine (machine from which you manage your Tanzu or Kubernetes deployment) run the following command:

Cat ~/.kube/config

From the output you need to get out the following info:

  • Kubernetes Master API URL (by default is https://<k8s cluster mgmt node or cluster ip>:6443)
  • certificate-authority-data
  • client-certificate-data
  • client-key-data

The certificate information need to be pasted into the credentials when adding the Tanzu Cluster in vROPs.

Navigate to your vROPs UI and under Data Sources > Integrations choose Kubernetes

Then fill in the required details to connect to your Tanzu Kubernetes cluster

Validate the connection to the newly added cluster and make sure it is successful

Click on ADD you should then see your Tanzu/Kubernetes cluster added as an integrated account in vROPs

Step 3: Monitoring added Tanzu Kubernetes clusters

Once the above steps are completed you should be able to see your Tanzu/Kubernetes clusters in vROPs and interact with them as you do with standard vSphere workloads. Below are some screenshots from my homelab after completing the above steps.

Final words

vROPS management pack for kubernetes onboards Tanzu and Kubernetes clusters seamlessly into vRealize Operations expanding vrops monitoring, troubleshooting and capacity planning capabilities to Tanzu and Kubernetes deployments deployments.

Bassem Rezkalla

Recent Posts

Gain Insight into Tanzu Kubernetes Clusters using VMware Project Octant

Overview Note: After I have written this blog post I got to hear that work…

4 days ago

Enable Workload Management (vSphere with Tanzu) with NSX ALB

Overview I am just back from VMware Explore in Barcelona after presenting an interesting session…

2 weeks ago

Deploying NSX NAPP on upstream (a.k.a native) Kubernetes – Part II

Overview In the second part of this blog post I will be finalising my NSX…

4 weeks ago

Deploying NSX NAPP on upstream (a.k.a native) Kubernetes – Part I

Overview In a previous blog post (HERE) I deployed NSX Application Platform on top of…

1 month ago

Deploying TKG 2 workload clusters with vSphere 8 with Tanzu

Overview With the release of vSphere 8, VMware introduced Tanzu Kubernetes Grid clusters version 2,…

1 month ago

Deploying TKG workload clusters across vSphere 8 Availability Zones

Overview vSphere 8 introduced zonal supervisor cluster deployments in order to improve Tanzu workload resiliency,…

2 months ago