Securing Containers using VMware Antrea IDS

Overview

With the release of NSX 4.0.0.1 and VMware Antrea 1.5.0 came a very interesting announcement that you can now secure Antrea containers integrated with NSX using NSX IDPS i.e. leveraging threat prevention sigantures and capabilities to secure your containerised workloads.

This feature is however is tech preview only at the moment, which means it is not for production deployment and VMware does not offer any support for this at the moment. The feature also offers only IDS capability at the moment with no GUI integration with NSX IDPS alerting dashboards i.e. you need to either read the IDS events alerts through pod logging or by exporting pod logging to logging tool such as fluent (I will discuss this in later posts.)

VMware Antrea IDPS implementation is based on project Antrea’s Traffic Control capability that was introduced with Antrea 1.7.0 release and uses Suricata (open source IDS/IPS project) as the IDPS engine.

Vmware Antrea IDPS solution is composed of the following main components:

• IDPS Controller which is responsible for signature pulling and validating Antrea registration with NSX.
• Antrea IDPS agents which are the running on all your K8s nodes and will be mirroring traffic handled by Antrea data plane agents to Suricata IDPS engine.
• Suricata IDPS engine deployed as containers within deployment pods and this is where the actual threat detection is happening (nothing to do with NSX IDPS engine).

Lab Inventory

For software versions I used the following:

• VMware ESXi 7.0U3f
• vCenter server version 7.0U3f
• TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
• VyOS 1.4 used as lab backbone router and DHCP server.
• Ubuntu 18.04 LTS as bootstrap machine.
• Ubuntu 20.04.2 LTS as DNS and internet gateway.
• Windows Server 2012 R2 Datacenter as management host for UI access.
• NSX 4.0.0.1

For virtual hosts and appliances sizing I used the following specs:

• 3 x virtualised ESXi hosts each with 8 vCPUs, 4 x NICs and 96 GB RAM.
• vCenter server appliance with 2 vCPU and 24 GB RAM.

Prerequisites

• VMware Antrea CNI and integrated with NSX (to learn how you might need to check my previous blog posts HERE and HERE)
• NSX TP or ATP license.
• Internet connection to Antrea IDPS pods so IDPS signatures can be pulled and updated.

Deployment steps

Step 1: Enable Traffic Control feature gate in Antrea agents

Traffic Control feature gate has been introduced as Beta in project Antrea 1.7.0. Traffic Control is a CRD API that manages and manipulates the transmission of Pod traffic. It allows users to mirror or redirect specific traffic originating from specific Pods or destined for specific Pods to a local network device or a remote destination via a tunnel of various types (ex. vxlan or geneve). It provides full visibility into network traffic, including both north-south and east-west traffic.

To enable Traffic Control on your Antrea agents, connect to your Kubernetes cluster (or from your master node) run the following command to determine the antrea configmap that you need to manipulate:

Now, edit the highlighted configmap using the following command:

kubectl edit configmaps -n kube-system antrea-config

This will open a vi editor for the antrea config agent, search for “TrafficControl” uncomment the line and set the value to true

save and exit vim using :wq! and then perform a rolling restart for Antrea config daemonsets to that changes in configmaps take effect:

kubectl rollout restart ds/antrea-agent -n kube-system

Step 2: Deploy Antrea IDPS Controller and agents

for this step, we need to use a deployment yaml file to roll out the IDPS controller and agents, you can use a sample deployment file I uploaded for you HERE  just copy and paste the contents of this file to a file called idps.yaml (name does not matter) open it with any editor and add your NSX TP or ATP in base64 encoded format to the below section in that deployment file:

Save and exit the file and then apply it using the following command:

kubectl apply -f idps.yml

This will start creating Antrea IDPS controller and agents pods under kube-system system namespace, give it sometime and then check the status of pods using the command:

kubectl get pods -n kube-system

The status of all the pods must be running.

Step 3: Create an IDPS policy to capture a test malicious generated traffic

Once Antrea IDPS is deployed, we need to define an IDPS policy which basically tells the IDPS engine which traffic from which pods should be monitored and which engine to use, either an IDS our IPS. However, in the current tech review release of VMware Antrea, VMware only supports IDS mode, so no need to define the engine type in the the IDPS policy CRD that we are going to create.

Create a file called WebIdsPolicy.yaml and paste in the following contents:

apiVersion: crd.antrea.tanzu.vmware.com/v1alpha1
kind: IDPSPolicy
metadata:
  name: test-ids-policy
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: web

The above will create a test IDPS policy called test-ids-policy which is active on pods having label of app=web. Save and exit the above file (please make sure that your indenting is exactly as shown above).

Run the command “Kubectl apply -f WebIdsPolicy.yaml” to start creating the test IDS policy CRD.

Check that the CRD has been created successfully:

Testing Antrea IDS functionality

In order to verify that our Antrea IDS is working by detecting and reporting malicious traffic patterns and behaviour, we will create a pod running alpine-curl image (or any container image with curl in it) and will label it with app=web so that our previously created IDS policy get applied to this test pod.

One important note, in this test I will be testing an egress malicious attack from my curl pod to http://testmynids.org, this is a very cool website which allows security admins test their IDPS engine against this website. A list of the available attacks and how to launch them can be found on their Github repository as well:

https://github.com/3CORESec/testmynids.org/blob/master/tmNIDS

For this test, I am going to use the most simple one which is a uid/gid scan against their website, so from my curl pod I will run the command:

curl -s http://testmynids.org/uid/index.html

For this I will also need to add my DNS to my Pod configuration, because by default pods will only use default cluster local dns to resolve only local kubernetes services and pod names. To create our testing pod, just copy and paste the following in yaml file and apply it:

apiVersion: v1
kind: Pod
metadata:
      name: curl
      labels:
          app: web
spec:
     containers:
     - name: source
       image: quay.io/cilium/alpine-curl
       command: ["sh", "-c", "while true; do sleep 3600; done"]
     dnsPolicy: "None"
     dnsConfig:
       nameservers:
       - 192.168.0.253

once you apply the above file using the command kubectl apply -f <pod yaml filename>  get on which node this test pod is running using the command:

kubectl get pods -o wide

login to the above shown node and run the following command:

ls -l  /var/log/antrea/suricata

Record the name of the created log file (this is the log file Suricata will be logging events to) then tail this file:

tail -f tail -f /var/log/antrea/suricata/<your eve filename>.json

Now, switch back to your bootstrap machine or your Kubernetes master and generate an egress malicious scan from our testing pod using the following command:

kubectl exec curl -- curl -s http://testmynids.org/uid/index.html

Switch back to the node on which the test pod is running and you should see the following event generated:

If you want to get a more neat output then you can pipe the above command to jq (on Ubuntu you can install it using sudo apt install -y jq)

Final word

With the addition of IDPS capability to Antrea, this will leverage the use of NSX drastically within the modern applications security space. Although the feature is in tech preview it is very promising and I expect this to be a game changer in NSX footprint across VMware customers.

Stay tuned to my next blog post in which I will show you how to visualise Antrea IDPS alerts for a more real life use case.[/vc_column_text]