Overview

NSX Advanced Load Balancer (a.k.a Avi) offers variety of advanced load balancing and application security features, one of which is WAF (Web Application Firewall) which allows security administrators to control traffic to and from web servers behind NSX ALB. This capability extends also to protect web servers hosted in containers, this is achieved by extending NSX ALB WAF functionality to Ingress objects created by AKO in a Kubernetes cluster.

Web Application Firewall can be applied to an Ingress object by defining an AKO HostRule, which is basically a Kubernetes CRD (Custom Resource Definition) that defines extra security rules to a specific virtual service (VS) in NSX ALB. If AKO is used then the virtual service we talk about maps to an Ingress object that controls traffic routing to a web service hosted in containers. If the previous sentence is new to you then stop reading this post and familiarise yourself with concept of Ingress, Kubernetes Load Balancing and AKO before reading.

NSX ALB AKO (Avi Kubernetes Operator) supports the following extra CRDs in Kubernetes:

  • Layer 7: These CRD objects are used to express layer 7 traffic routing rules.

    • Layer 7 rules are categorised into HostRule & HTTPRule.
      • The HostRule CRD is used to express additional virtual host properties. The virtual host FQDN is matched from a Kubernetes Ingress resource.
      • The HTTPRule CRD can be used as a complimentary object to control additional layer 7 properties like algorithm, hash, and TLS re-encrypt use cases.

  • Layer 4: These CRD objects are used to express layer 4 traffic routing rules.

  • Infrastructure: These CRD objects are used to control Avi’s infrastructure components like Ingress class, SE group properties etc.

In this post I will use a HostRule CRD to define some security options to control access to a web application hosted in a Tanzu cluster.

Lab Inventory

For software versions I used the following:

    • VMware ESXi 8.0U1
    • vCenter server version 8.0U1
    • VMware NSX ALB (Avi) 22.1.3
    • TrueNAS 12.0-U7 used to provision NFS data stores to ESXi hosts.
    • VyOS 1.3 used as lab backbone router and DHCP server.
    • Windows Server 2019 as DNS server.
    • Windows 10 pro as management host for UI access.

For virtual hosts and appliances sizing I used the following specs:

    • 6 x ESXi hosts each with 12 vCPUs, 2 x NICs and 128 GB RAM.
    • vCenter server appliance with 2 vCPU and 24 GB RAM.

Deployment Workflow

  • Create an Ingress object mapping to a containerised web service.
  • Create NSX ALB WAF Profile & Policy.
  • Create & Apply AKO HostRule to Ingress.
  • Verify HostRule Operation. 

Create a Secure Ingress object mapping to a containerised web service

In my lab setup I have the following application which has web server running in a pod called frontend and I pre-configured a secure ingress object called onlineshop-ingress

Detailed Ingress YAML configuration is shown below

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/issuer: vault-issuer
  name: onlineshop-ingress
  namespace: microservices
spec:
  rules:
  - host: onlineshop.zonal.nsxbaas.homelab
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: frontend
            port:
              number: 80
  tls:
  - hosts:
    - onlineshop.zonal.nsxbaas.homelab
    secretName: onlineshop-ingress-secret

In the above YAML I used certmanager to request a certificate to be used for HTTPS access to my frontend containers, the details on how I have setup certmanager and my CA issuer can be found in this previous post.

From a Web browser, I will navigate to the URL https://onlineshop.zonal.nsxbaas.homelab Chrome and most web browsers (including also cURL command from Linux) send HTTP/1.1 by default unless instructed otherwise, in the below screenshot as you can see access to my web application works just fine over HTTP/1.1

My web server which is running in the pod/container called frontend, supports only HTTP/1.1 requests so in order to demonstrate how NSX ALB WAF can protect/limit access to web applications I will create a WAF policy that allows only HTTP/2.0.

Create NSX ALB WAF Profile & Policy

Important note before proceeding in this step, in a production environment you will probably need to carry out the following steps in the following order:

  • Generate a certificate from your CA issuer and import it as an SSL template in Avi controller.
  • Create an httppolicyset from Avi CLI (no GUI option for that so far).
  • Create your Ingress object without TLS definition.
  • Create AKO HostRule CRD referencing your Ingress and TLS and policyset sections.

The reason is why I hinted the above sequence is that in my lab setup I used a difference approach (I am not following the above mentioned workflow) to create my secure Ingress object, if you reference the above YAML sample of my Ingress you will see that I added the TLS section directly in YAML which references a certificate which is assigned by certmanager directly to my Ingress i.e. NSX ALB (Avi) will inherit both TLS and httppolicyset configuration from the virtual service mapped to this Ingress so I will need to omit both when I define my AKO HostRule.

Step 1: Create WAF Profile from NSX ALB UI

From NSX ALB UI navigate to Templates> WAF> WAF Profile and click CREATE

In the above screenshot I allowed only HTTP/2.0 as allowed protocol, if you navigate the rest of the available controls you can define in the profile more explanation can be found HERE. Click on SAVE and then navigate to WAF Policy.

Step 2: Create WAF Policy

Next step is to create a WAF policy which will be attached to our HostRule and applied to our Ingress object. From NSX ALB UI navigate to Templates> WAF> WAF Policy and click on CREATE 

The changes I have done in the above is assigning a name to the policy, assigning the profile to created in step1 to this policy and I chose Enforcement mode to make sure that NSX ALB WAF actually blocks any non HTTP/2.0 requests and not just detect and log it. Again, what you can define in WAF policy is very broad and I will not be able to cover all of them in this blog post, but if you look for more details about what can be defined in WAF policy you can reference this document.

Create & Apply AKO HostRule to Ingress

Step 1: Create HTTPPolicyset using Avi CLI

One of the advanced features that we can use in HostRule is the ability to define an httppolicyset, an HTTP policy set defines some advanced custom parameters for HTTP(S) traffic such as rate-limiting, redirection, control attributes and so on. An httppolicyset needs however to be created via command-line or API on Avi controller, no UI option so far to create policysets. In my setup, I logged in to my NSX ALB (Avi) controller via SSH and applied the following highlighted command in sequence to create a simple redirection policy in httppolicyset called waf_post_nsxbaas

You can verify the creation and contents of the policysset from Avi CLI by switching to shell mode (using command shell) and executing the command show httppolicyset <policyset_name> 

Step 2: Create and apply HostRule CRD for a secure Ingress

Login to your Tanzu/Kubernetes cluster where you have AKO already installed and apply the following YAML file

apiVersion: ako.vmware.com/v1alpha1
kind: HostRule
metadata:
   name: secure-onlineshop-hostrule
   namespace: microservices
spec:
  virtualhost:
    fqdn: onlineshop.zonal.nsxbaas.homelab # mandatory
    # tls:
    #sslKeyCertificate:
    #  name: "zonal-cluster01--onlineshop.zonal.nsxbaas.homelab"
    #  type: ref
    #  termination: edge
    httpPolicy:
      policySets:
      - "waf_post_nsxbaas"
      overwrite: false
    wafPolicy: "waf_policy01"
    applicationProfile: "System-Secure-HTTP"

Breakdown of the above YAML is as follow:

  • name: is the name given to this HostRule CRD
  • namespace: is the Kubernetes namespace on which this HostRule will be applied
  • virtualhost: multiple virtual hosts FQDNs can be defined where every virtual host represents an Ingress object which is pre-created.
  • FQDN: is a sub property under virtual host and is mandatory since it defines which Ingress will inherit the configuration defined in this Host Rule.
  • tls: this is an optional section if Ingress object is based on HTTPS, in my setup as mentioned earlier I have an HTTPS Ingress object pre-created which has TLS specs already defined in the earlier shown Ingress YAML definition file, so I omitted the TLS section in the above HostRule YAML.
  • httpPolicy: is the name of a pre-created http policy set (optional section). In step 1 we created an httppolicyset called waf_post_nsxbaas and is being referenced in this section.
  • wafPolicy: is the name of the WAF policy we configured earlier in NSX ALB UI.
  • applicationProfile: Application profile defines some extra options and security for HTTP traffic such as preserving client IPs, NT:M app detection, XFF, rate-limiting and much more. In my setup I use an HTTPS Ingress that’s why I assigned a default HTTPS application profile in my HostRule definition above. The application profile I used is the default in Avi for HTTPS traffic

Save and apply the YAML HostRule file above to your Tanzu/Kubernetes cluster.

Verify HostRule Operation

Step 1: Verify HostRule assignment

After we applied our HostRule CRD above to our Tanzu cluster (kubectl apply -f hostrule_crd_yaml.yaml) we can verify is the HostRule has been successfully applied to the Ingress object or not, see below screenshots

You can also get more details on the HostRule using Kubectl describe command

From NSX ALB UI (Avi controller) you can see also a small shield icon next to the Ingress virtual service indicating that there is a WAF policy attached to the virtual service

Step 2: Verify HTTPS access to Ingress URL

Last step we need to verify if we can still access our online store page (exposed via HTTPS Ingress). Remeber, our WAF policy allows only HTTP/2.0 protocol version however my backed webserver supports only HTTP/1.1 so let us see if we can access https://onlineshop.zonal.nsxbaas.homelab as we have done earlier in this post. Open a web browser and type the URL of your protected Ingress

As you can see the above response is being generated from NSX ALB (Avi) itself since the requested has been intercepted by the WAF policy which allows only HTTP/2 and the response code returned is 403 which means that you do not have enough permissions to access the requested URL.

Hope you have found this blog post useful!