In this blog post which is the last in series of posts discussing VMware Antrea IDS configuration and visibility, I am going to configure VMware Aria Operations for Logs (formerly known as vRealize LogInsight) to ingest and display VMware Antrea logs including IDS events captured by Antrea IDS Suricata engine.
VMware Aria operations for logs offers a sophisticated engine for log analysis and automatically identify structure in machine-generated, unstructured log data (including application logs, network traces, configuration files, etc.). Using Aria for operations logs we can build dashboard of interest for efficient logs visibility and analysis.
For software versions I used the following:
For virtual hosts and appliances sizing I used the following specs:
Aria Operations for Logs utilises Fluentd open source project for collecting Kubernetes cluster logs and send them over to Aria syslog server. Fluentd uses plugins in order to be able to interact and send logs to different collection destinations, such as elasticsearch, Aria and other tools, so the concept is pretty straight forward you need to install the right plugin for the collector that you want to use to collect Kubernetes logs collected by Fluentd.
However, this is easy said than done because installing a plugin to fluentd general image means that you will need to rebuild a fluentd image with the specific plugin you want to use which is a cumbersome task. For that reason, VMware has released specific fluentd image which already has loginsight log collection plugin installed and this is what we are going to use int his blog post. VMware fluentd k8s images are available via the following VMware Harbor repository:
Before we deploy the actual fluentd daemonset on our k8s cluster, we need to create a configmap through a configuration file, in order to pass some configuration parameters to fluentd pods, those parameters include:
login to your bootstrap machine and create a file called fluent.conf and paste the following contents in it (modify loginsight address as per your setup)
<source> @id in_tail_container_logs @type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag raw.kubernetes.* read_from_head true <parse> @type multi_format <pattern> format json time_key time time_format %Y-%m-%dT%H:%M:%S.%NZ </pattern> <pattern> format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/ time_format %Y-%m-%dT%H:%M:%S.%N%:z </pattern> </parse> </source> <source> @type tail read_from_head true path /var/log/antrea/suricata/eve.alert.* pos_file /var/log/fluentd-suricata.pos tag suricata <parse> @type json time_type string time_format %Y-%m-%dT%H:%M:%S.%6N%z </parse> </source> # Detect exceptions in the log output and forward them as one log entry. <match raw.kubernetes.**> @id raw.kubernetes @type detect_exceptions remove_tag_prefix raw message log stream stream multiline_flush_interval 5 max_bytes 500000 max_lines 1000 </match> <filter kubernetes.**> @type record_transformer <record> environment tanzu_k8s_grid log_type kubernetes </record> watch false </filter> # Enriches records with Kubernetes metadata <filter kubernetes.**> @id filter_kubernetes_metadata @type kubernetes_metadata watch false </filter> <match **> @type vmware_loginsight scheme https ssl_verify false host 188.8.131.52 port 9543 http_method post serializer json rate_limit_msec 0 raise_on_error true include_tag_key true tag_key tag http_conn_debug false </match>
Save and exit the above file, then create a namespace called kube-logging and a configmap from the above file in that namespace:
kubectl create ns kube-logging
kubectl create configmap loginsight-fluentd-config -n kube-logging --from-file fluent.conf
Next step, is to deploy fluentd as Daemonset. This allows fluentd pods to run on all available worker nodes and will be created automatically if deleted. Create a deployment YAML file and paste the below contents into it (make sure to create the deployment file under the same directory as the fluent.conf file).
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: fluentd-loginsight-logging name: fluentd-loginsight-logging namespace: kube-logging --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole rules: - apiGroups: [""] resources: ["namespaces", "pods"] verbs: ["list", "get", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluentd-clusterrole subjects: - kind: ServiceAccount name: fluentd-loginsight-logging namespace: kube-logging --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-loginsight-logging namespace: kube-logging labels: k8s-app: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: name: fluentd-loginsight-logging template: metadata: labels: name: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd-loginsight-logging serviceAccountName: fluentd-loginsight-logging tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-loginsight image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0 command: ["fluentd", "-c", "/etc/fluentd/fluent.conf", "-p", "/fluentd/plugins"] env: - name: FLUENTD_ARGS value: --no-supervisor -q resources: limits: memory: 500Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log readOnly: false - name: config-volume mountPath: /etc/fluentd readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: config-volume configMap: name: loginsight-fluentd-config
Save and exit the above file and then apply it to your k8s cluster:
kubectl apply -f <filename.yaml>
Wait for couple of minutes then check the status of the pods running inside kube-logging namespace, they should be all in running state:
Login to Aria UI and from left pane select Explore logs, you should be able to see logs coming in from your k8s cluster
In the search field type suricata and press enter, you should get similar output to the below with Antrea IDS logs
Having Aria Operations for logs as central logging system for all your workloads is a great way of keeping close eye to all events and alerts across your entire environment. With integrating Antrea IDS logs with Aria you can create dashboards specifically for IDS events from your Tanzu and/or k8s clusters and with that you have a centralised log collection and visualisation for your containerised workloads as well.
Hope you find this blog post useful.
Overview NSX ALB (previously known as Avi) offers rich capabilities for L4-L7 load balancing across…
Overview In part one of this blog post, we deployed a Cloud Director instance and…
Overview Regardless of the type of the cloud services that your organisation is making use…