nsxbaas

Overview

In this blog post which is the last in series of posts discussing VMware Antrea IDS configuration and visibility, I am going to configure VMware Aria Operations for Logs (formerly known as vRealize LogInsight) to ingest and display VMware Antrea logs including IDS events captured by Antrea IDS Suricata engine.

VMware Aria operations for logs offers a sophisticated engine for log analysis and automatically identify structure in machine-generated, unstructured log data (including application logs, network traces, configuration files, etc.). Using Aria for operations logs we can build dashboard of interest for efficient logs visibility and analysis.

Lab Inventory

For software versions I used the following:

• VMware ESXi 7.0U3f
• vCenter server version 7.0U3f
• TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
• VyOS 1.4 used as lab backbone router and DHCP server.
• Ubuntu 18.04 LTS as bootstrap machine.
• Ubuntu 20.04.2 LTS as DNS and internet gateway.
• Windows Server 2012 R2 Datacenter as management host for UI access.
• NSX 4.0.0.1
• Vanilla Kubernetes cluster version 1.24
• VMware Aria Operations for Logs version 8.8

For virtual hosts and appliances sizing I used the following specs:

• 3 x virtualised ESXi hosts each with 12 vCPUs, 2 x NICs and 128 GB RAM.
• vCenter server appliance with 2 vCPU and 24 GB RAM.

Prerequisites

• VMware Antrea CNI and integrated with NSX (to learn how you might need to check my previous blog posts HERE and HERE)
• NSX TP or ATP license.
• VMware Antrea IDS controller and agents deployed and all in running state.
• VMware Aria Operations for Logs version 8.8 deployed and running (step by step Installation Guide).

Kubernetes logging using Aria Operations for Logs

Aria Operations for Logs utilises Fluentd open source project for collecting Kubernetes cluster logs and send them over to Aria syslog server. Fluentd uses plugins in order to be able to interact and send logs to different collection destinations, such as elasticsearch, Aria and other tools, so the concept is pretty straight forward you need to install the right plugin for the collector that you want to use to collect Kubernetes logs collected by Fluentd.

However, this is easy said than done because installing a plugin to fluentd general image means that you will need to rebuild a fluentd image with the specific plugin you want to use which is a cumbersome task. For that reason, VMware has released specific fluentd image which already has loginsight log collection plugin installed and this is what we are going to use int his blog post. VMware fluentd k8s images are available via the following VMware Harbor repository:

projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0

Deployment steps

Step 1: Deploy and configure fluentd

Before we deploy the actual fluentd daemonset on our k8s cluster, we need to create a configmap through a configuration file, in order to pass some configuration parameters to fluentd pods, those parameters include:

• Which logs from k8s cluster to be collected by fluentd agents, i.e. log sources.
• What are the log collectors to which fluentd should be sending the collected logs. By default, fluentd agents send collected logs to stdout of fluentd pods, we need to set our Vmware Aria Operations address as the log output destination and this needs to be added to the initial configuration file.

login to your bootstrap machine and create a file called fluent.conf and paste the following contents in it (modify loginsight address as per your setup)

<source>
  @id in_tail_container_logs
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

<source>
  @type tail
  read_from_head true
  path /var/log/antrea/suricata/eve.alert.*
  pos_file /var/log/fluentd-suricata.pos
  tag suricata
  <parse>
    @type json
    time_type string
    time_format %Y-%m-%dT%H:%M:%S.%6N%z
  </parse>
</source>

# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
  @id raw.kubernetes
  @type detect_exceptions
  remove_tag_prefix raw
  message log
  stream stream
  multiline_flush_interval 5
  max_bytes 500000
  max_lines 1000
</match>

<filter kubernetes.**>
  @type record_transformer
  <record>
  environment tanzu_k8s_grid
  log_type kubernetes
  </record>
  watch false
</filter>

# Enriches records with Kubernetes metadata
<filter kubernetes.**>
  @id filter_kubernetes_metadata
  @type kubernetes_metadata
  watch false
</filter>

<match **>
  @type vmware_loginsight
  scheme https
  ssl_verify false
  host 172.10.40.5
  port 9543
  http_method post
  serializer json
  rate_limit_msec 0
  raise_on_error true
  include_tag_key true
  tag_key tag
  http_conn_debug false
</match>

Save and exit the above file, then create a namespace called kube-logging and a configmap from the above file in that namespace:

kubectl create ns kube-logging

kubectl create configmap loginsight-fluentd-config -n kube-logging --from-file fluent.conf

 

Step 2: Deploy fluentd Daemonset pods and verify deployment

Next step, is to deploy fluentd as Daemonset. This allows fluentd pods to run on all available worker nodes and will be created automatically if deleted. Create a deployment YAML file and paste the below contents into it (make sure to create the deployment file under the same directory as the fluent.conf file).

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: fluentd-loginsight-logging
  name: fluentd-loginsight-logging
  namespace: kube-logging

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-clusterrole
rules:
- apiGroups: [""]
  resources: ["namespaces", "pods"]
  verbs: ["list", "get", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-clusterrole
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-clusterrole
subjects:
  - kind: ServiceAccount
    name: fluentd-loginsight-logging
    namespace: kube-logging

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-loginsight-logging
  namespace: kube-logging
  labels:
    k8s-app: fluentd-loginsight-logging
    app: fluentd-loginsight-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
 selector:
   matchLabels:
     name: fluentd-loginsight-logging
 template:
   metadata:
     labels:
       name: fluentd-loginsight-logging
       app: fluentd-loginsight-logging
       version: v1
       kubernetes.io/cluster-service: "true"
   spec:
     serviceAccount: fluentd-loginsight-logging
     serviceAccountName: fluentd-loginsight-logging
     tolerations:
     - key: node-role.kubernetes.io/master
       effect: NoSchedule
     containers:
     - name: fluentd-loginsight
       image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0
       command: ["fluentd", "-c", "/etc/fluentd/fluent.conf", "-p", "/fluentd/plugins"]
       env:
       - name: FLUENTD_ARGS
         value: --no-supervisor -q
       resources:
         limits:
           memory: 500Mi
         requests:
           cpu: 100m
           memory: 200Mi
       volumeMounts:
       - name: varlog
         mountPath: /var/log
         readOnly: false
       - name: config-volume
         mountPath: /etc/fluentd
         readOnly: true
     volumes:
     - name: varlog
       hostPath:
         path: /var/log
     - name: config-volume
       configMap:
         name: loginsight-fluentd-config

Save and exit the above file and then apply it to your k8s cluster:

kubectl apply -f <filename.yaml>

Wait for couple of minutes then check the status of the pods running inside kube-logging namespace, they should be all in running state:

Step 3: Verify log collection on Aria Operations for logs

Login to Aria UI and from left pane select Explore logs, you should be able to see logs coming in from your k8s cluster

In the search field type suricata and press enter, you should get similar output to the below with Antrea IDS logs

Final Word

Having Aria Operations for logs as central logging system for all your workloads is a great way of keeping close eye to all events and alerts across your entire environment. With integrating Antrea IDS logs with Aria you can create dashboards specifically for IDS events from your Tanzu and/or k8s clusters and with that you have a centralised log collection and visualisation for your containerised workloads as well.

Hope you find this blog post useful.