In this blog post which is the last in series of posts discussing VMware Antrea IDS configuration and visibility, I am going to configure VMware Aria Operations for Logs (formerly known as vRealize LogInsight) to ingest and display VMware Antrea logs including IDS events captured by Antrea IDS Suricata engine.
VMware Aria operations for logs offers a sophisticated engine for log analysis and automatically identify structure in machine-generated, unstructured log data (including application logs, network traces, configuration files, etc.). Using Aria for operations logs we can build dashboard of interest for efficient logs visibility and analysis.
For software versions I used the following:
- VMware ESXi 7.0U3f
- vCenter server version 7.0U3f
- TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
- VyOS 1.4 used as lab backbone router and DHCP server.
- Ubuntu 18.04 LTS as bootstrap machine.
- Ubuntu 20.04.2 LTS as DNS and internet gateway.
- Windows Server 2012 R2 Datacenter as management host for UI access.
- NSX 220.127.116.11
- Vanilla Kubernetes cluster version 1.24
- VMware Aria Operations for Logs version 8.8
For virtual hosts and appliances sizing I used the following specs:
- 3 x virtualised ESXi hosts each with 12 vCPUs, 2 x NICs and 128 GB RAM.
- vCenter server appliance with 2 vCPU and 24 GB RAM.
- VMware Antrea CNI and integrated with NSX (to learn how you might need to check my previous blog posts HERE and HERE)
- NSX TP or ATP license.
- VMware Antrea IDS controller and agents deployed and all in running state.
- VMware Aria Operations for Logs version 8.8 deployed and running (step by step Installation Guide).
Kubernetes logging using Aria Operations for Logs
Aria Operations for Logs utilises Fluentd open source project for collecting Kubernetes cluster logs and send them over to Aria syslog server. Fluentd uses plugins in order to be able to interact and send logs to different collection destinations, such as elasticsearch, Aria and other tools, so the concept is pretty straight forward you need to install the right plugin for the collector that you want to use to collect Kubernetes logs collected by Fluentd.
However, this is easy said than done because installing a plugin to fluentd general image means that you will need to rebuild a fluentd image with the specific plugin you want to use which is a cumbersome task. For that reason, VMware has released specific fluentd image which already has loginsight log collection plugin installed and this is what we are going to use int his blog post. VMware fluentd k8s images are available via the following VMware Harbor repository:
Step 1: Deploy and configure fluentd
Before we deploy the actual fluentd daemonset on our k8s cluster, we need to create a configmap through a configuration file, in order to pass some configuration parameters to fluentd pods, those parameters include:
- Which logs from k8s cluster to be collected by fluentd agents, i.e. log sources.
- What are the log collectors to which fluentd should be sending the collected logs. By default, fluentd agents send collected logs to stdout of fluentd pods, we need to set our Vmware Aria Operations address as the log output destination and this needs to be added to the initial configuration file.
login to your bootstrap machine and create a file called fluent.conf and paste the following contents in it (modify loginsight address as per your setup)
<source> @id in_tail_container_logs @type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag raw.kubernetes.* read_from_head true <parse> @type multi_format <pattern> format json time_key time time_format %Y-%m-%dT%H:%M:%S.%NZ </pattern> <pattern> format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/ time_format %Y-%m-%dT%H:%M:%S.%N%:z </pattern> </parse> </source> <source> @type tail read_from_head true path /var/log/antrea/suricata/eve.alert.* pos_file /var/log/fluentd-suricata.pos tag suricata <parse> @type json time_type string time_format %Y-%m-%dT%H:%M:%S.%6N%z </parse> </source> # Detect exceptions in the log output and forward them as one log entry. <match raw.kubernetes.**> @id raw.kubernetes @type detect_exceptions remove_tag_prefix raw message log stream stream multiline_flush_interval 5 max_bytes 500000 max_lines 1000 </match> <filter kubernetes.**> @type record_transformer <record> environment tanzu_k8s_grid log_type kubernetes </record> watch false </filter> # Enriches records with Kubernetes metadata <filter kubernetes.**> @id filter_kubernetes_metadata @type kubernetes_metadata watch false </filter> <match **> @type vmware_loginsight scheme https ssl_verify false host 18.104.22.168 port 9543 http_method post serializer json rate_limit_msec 0 raise_on_error true include_tag_key true tag_key tag http_conn_debug false </match>
Save and exit the above file, then create a namespace called kube-logging and a configmap from the above file in that namespace:
kubectl create ns kube-logging
kubectl create configmap loginsight-fluentd-config -n kube-logging --from-file fluent.conf
Step 2: Deploy fluentd Daemonset pods and verify deployment
Next step, is to deploy fluentd as Daemonset. This allows fluentd pods to run on all available worker nodes and will be created automatically if deleted. Create a deployment YAML file and paste the below contents into it (make sure to create the deployment file under the same directory as the fluent.conf file).
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: fluentd-loginsight-logging name: fluentd-loginsight-logging namespace: kube-logging --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole rules: - apiGroups: [""] resources: ["namespaces", "pods"] verbs: ["list", "get", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluentd-clusterrole subjects: - kind: ServiceAccount name: fluentd-loginsight-logging namespace: kube-logging --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-loginsight-logging namespace: kube-logging labels: k8s-app: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: name: fluentd-loginsight-logging template: metadata: labels: name: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd-loginsight-logging serviceAccountName: fluentd-loginsight-logging tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-loginsight image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0 command: ["fluentd", "-c", "/etc/fluentd/fluent.conf", "-p", "/fluentd/plugins"] env: - name: FLUENTD_ARGS value: --no-supervisor -q resources: limits: memory: 500Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log readOnly: false - name: config-volume mountPath: /etc/fluentd readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: config-volume configMap: name: loginsight-fluentd-config
Save and exit the above file and then apply it to your k8s cluster:
kubectl apply -f <filename.yaml>
Wait for couple of minutes then check the status of the pods running inside kube-logging namespace, they should be all in running state:
Step 3: Verify log collection on Aria Operations for logs
Login to Aria UI and from left pane select Explore logs, you should be able to see logs coming in from your k8s cluster
In the search field type suricata and press enter, you should get similar output to the below with Antrea IDS logs
Having Aria Operations for logs as central logging system for all your workloads is a great way of keeping close eye to all events and alerts across your entire environment. With integrating Antrea IDS logs with Aria you can create dashboards specifically for IDS events from your Tanzu and/or k8s clusters and with that you have a centralised log collection and visualisation for your containerised workloads as well.
Hope you find this blog post useful.