If you have been in the Tanzu/Kubernetes world for a while then you have definitely come across Prometheus and Grafana as open-source monitoring and visualisation tools available for Kubernetes.
Prometheus is free and an open-source event monitoring tool for containers or microservices, while Grafana is a multi-platform visualisation software which provides graphs and charts for visualising your data sources, no matter where they are stored.
During my setup of Prometheus and Grafana to monitor my Tanzu Community edition clusters, I made use of Helm. Helm is a package manager for Kubernetes which is part of the Artifact Hub open project (more about it HERE) I also used a Tanzu Community edition workload cluster which I deployed in a previous blog post (HERE).
For software versions I used the following:
For virtual hosts and appliances sizing I used the following specs:
We need to ensure that we have the following components installed and running before we proceed:
Before we proceed with installing and configuring Prometheus and Grafana pods, we will inspect our Tanzu cluster status and switch context to our workload cluster
kubectl cluster list kubectl config use-context tce-wld-cluster01-admin@tce-wld-cluster01 kubectl get ns
We need to ensure that helm packages are installed before proceeding with installing Prometheus and Grafana. It is important to note that by installing prometheus-operator packages we also install Grafana, so need for an extra step to deploy Grafana pods.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install my-kube-prometheus-stack \ prometheus-community/kube-prometheus-stack --version 36.0.1 --namespace \ monitoring \
Once all relevant packages are fetched and Prometheus pods are deployed, the output of kubectl get pods -n monitoring should look similar to the below:
By default, Prometheus server (dashboard UI) will listen on port 9090 and Grafana on port 3000. Those ports are not by default exposed outside of the Kubernetes cluster and in order to be able to access those ports we need to expose them using any of the below methods:
I will be using the first method as it is the simplest one for a lab test. First, note down your Prometheus and Grafana main pods, in my case they are highlighted below in green:
Next, we need to expose both pods on ports 9090 and 3000 respectively
kubectl port-forward --address 0.0.0.0 -n monitoring prometheus-my-kube-prometheus-stack-prometheus-0 9090 & kubectl port-forward --address 0.0.0.0 -n monitoring my-kube-prometheus-stack-grafana-5d7d448b46-77x5g 3000 &
The above commands will cause both pods to listen to incoming HTTP requests on all interfaces of the main machine running those pods.
From your web browser use command http://<machine where Promtheus is running>:9090
Navigate to Status then Targets to verify the K8s cluster discovery
To access Grafana, open an extra tap and then navigate to http://<machine where Grafana is running>:3000
default login credentails are:
admin and prom-operator
Navigate to Dashboards and then Browse
You then a get a list of the dahsboards which are added by default and populated by the metrics collected by Prometheus
Click on any metric of an interest and browse through the dashboards and graphs
Prometheus and Grafana are very powerful open-source metric collection and analysis tools and for customers adopting Tanzu Community edition this is a perfect match to complete an open-source Tanzu deployment.
Overview NSX ALB (previously known as Avi) offers rich capabilities for L4-L7 load balancing across…
Overview In part one of this blog post, we deployed a Cloud Director instance and…
Overview Regardless of the type of the cloud services that your organisation is making use…