Monitoring Tanzu Community edition using Prometheus and Grafana

If you have been in the Tanzu/Kubernetes world for a while then you have definitely come across Prometheus and Grafana as open-source monitoring and visualisation tools available for Kubernetes.

Prometheus is free and an open-source event monitoring tool for containers or microservices, while Grafana is a multi-platform visualisation software which provides graphs and charts for visualising your data sources, no matter where they are stored.

During my setup of Prometheus and Grafana to monitor my Tanzu Community edition clusters, I made use of Helm. Helm is a package manager for Kubernetes which is part of the Artifact Hub open project (more about it HERE) I also used a Tanzu Community edition workload cluster which I deployed in a previous blog post (HERE).

Lab Inventory

For software versions I used the following:

• VMware ESXi 7.0U3d
• vCenter server version 7.0U3
• TrueNAS 12.0-U7 used to provision NFS datastores to ESXi hosts.
• VyOS 1.4 used as lab backbone router and DHCP server.
• Ubuntu 20.04 LTS as bootstrap machine.
• Ubuntu 20.04.2 LTS as DNS and internet gateway.
• Windows Server 2012 R2 Datacenter as management host for UI access.
• Tanzu Community Edition version 0.11.0

For virtual hosts and appliances sizing I used the following specs:

• 3 x virtualised ESXi hosts each with 8 vCPUs, 4 x NICs and 32 GB RAM.
• vCenter server appliance with 2 vCPU and 24 GB RAM.

Prerequisites

We need to ensure that we have the following components installed and running before we proceed:

• TCE workload cluster.
• Kubectl command line tool.
• Helm package manager.

Deploying Prometheus & Grafana to TCE workload cluster

Before we proceed with installing and configuring Prometheus and Grafana pods, we will inspect our Tanzu cluster status and switch context to our workload cluster

kubectl cluster list
kubectl config use-context tce-wld-cluster01-admin@tce-wld-cluster01
kubectl get ns

Step1: Create a namespace for Prometheus and Grafana pods

Step 2: Verify helm version and install Prometheus operator

We need to ensure that helm packages are installed before proceeding with installing Prometheus and Grafana. It is important to note that by installing prometheus-operator packages we also install Grafana, so need for an extra step to deploy Grafana pods.

Add prometheus Helm repo and install Prometheus operator CRDs to monitoring namespace

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install my-kube-prometheus-stack \
prometheus-community/kube-prometheus-stack --version 36.0.1 --namespace \
monitoring \

Once all relevant packages are fetched and Prometheus pods are deployed, the output of kubectl get pods -n monitoring should look similar to the below:

Step 3: Expose Prometheus and Grafana UI

By default, Prometheus server (dashboard UI) will listen on port 9090 and Grafana on port 3000. Those ports are not by default exposed outside of the Kubernetes cluster and in order to be able to access those ports we need to expose them using any of the below methods:

• Using Kubectl port forwarding
• Exposing the Prometheus deployment as a service with NodePort or a Load Balancer.
• Adding an ingress object if you have an Ingress controller deployed.

I will be using the first method as it is the simplest one for a lab test. First, note down your Prometheus and Grafana main pods, in my case they are highlighted below in green:

Next, we need to expose both pods on ports 9090 and 3000 respectively

kubectl port-forward --address 0.0.0.0 -n monitoring prometheus-my-kube-prometheus-stack-prometheus-0 9090 &

kubectl port-forward --address 0.0.0.0 -n monitoring my-kube-prometheus-stack-grafana-5d7d448b46-77x5g 3000 &

The above commands will cause both pods to listen to incoming HTTP requests on all interfaces of the main machine running those pods.

Step 4: Accessing Prometheus and Grafana UI

From your web browser use command http://<machine where Promtheus is running>:9090

Navigate to Status then Targets to verify the K8s cluster discovery

To access Grafana, open an extra tap and then navigate to http://<machine where Grafana is running>:3000

default login credentails are:

admin and prom-operator

Navigate to Dashboards and then Browse

You then a get a list of the dahsboards which are added by default and populated by the metrics collected by Prometheus

Click on any metric of an interest and browse through the dashboards and graphs

Final words

Prometheus and Grafana are very powerful open-source metric collection and analysis tools and for customers adopting Tanzu Community edition this is a perfect match to complete an open-source Tanzu deployment.