Overview
Multi-tenancy is the ability to offer NSX networking and security services to multiple tenants completely isolated from each other. Every tenant will also have its own RBAC configuration and can be assigned quotas to limit the number of objects that can be created inside a tenant. Multi-Tenancy has been a long awaited feature in NSX which enables not only service providers but also end customers to provide NSX services tailored and scoped down to a department/team level on the same NSX instance, previously that was only possible by deploying different NSX instances per tenant/department.
Multi-Tenancy in NSX is achieved by creating NSX projects, where every project represents a logical container of network and security resources (tenant) where every project can have its own set of users, assigned privileges and quotas. Multi-Tenancy has different use cases such as offering Networkings as a Service, Firewall as a Service and so on.
Multi-Tenancy was introduced in NSX UI starting from VMware NSX 4.1 and it uses two tier data model, first tier is called Infra tier which is referred to as Default space, Default space contains non-isolated objects and is accessible to Enterprise admin and other system wide users who are not member of projects. In short, the Default view contains NSX objects that do not belong to any project. The other data model is referred to as Org model (branch) under which projects (tenants) provision their resources, which implies that every tenant (project) will also have a sub-Infra branch with only objects that are created and available to that project (tenant).
In the below diagram is the reference architecture of the lab setup I am using in this blog post
Lab Inventory
For software versions I used the following:
- VMware ESXi 8.0
- vCenter server version 8.0
- VMware NSX 4.1.0
- VMware Antrea 1.6.0
- VMware NSX ALB (Avi) AKO for Ingress provisioning.
- TrueNAS 12.0-U7 as backend storage system.
- VyOS 1.3 used as lab backbone router, NTP and DHCP server.
- Ubuntu 20.04 LTS as Linux jumpbox.
- Windows Server 2019 R2 Standard as DNS server.
- Windows 10 Pro as UI jump box.
- 3 x Ubuntu 18.04 VMs as 1 x Kubernete controller and 2 x nodes.
For virtual hosts and appliances sizing I used the following specs:
- 3 x virtualised ESXi hosts each with 12 vCPUs, 2x NICs and 128 GB RAM.
- vCenter server appliance with 2 vCPU and 24 GB RAM.
- NSX Manager medium appliance
Deployment Workflow
In this post I will deploy an NSX multi-tenant environment with two tenants (projects) Prod and Dev and eventually demo DFW as a Service offering to both tenants. The deployment workflow is shown in the below steps:
- Add Projects Web and Dev, create and assign two admin users for both projects.
- Define and apply quota limits on both projects to limit DFW resource consumption.
- Create segments, security groups for VMs under prod & dev projects.
- Create and verify DFW rules within both tenants (projects).
Add Projects Web and Dev and create project level administrators
From NSX UI we will add two projects, p-prod and p-dev, to do this login to NSX UI and from the projects switching drop-down menu labeled “Default” click on it and choose Manage
Note I: When choosing your Edge Clusters, the overlay transport zone connected to those edges need to be the default transport zone defined in your NSX instance, so you need to make sure that overlay TZ used is labeled as default
Note II: the short log identifies is very important during troubleshooting, since this will be added as a label for all logging related to that project.
Click on SAVE to create p-prod project, then we need to create our p-prod-admin user as Local User and then assign it as project admin to p-prod project. From NSX UI navigate to System > User Management > Local Users and click on ADD you then need to add and activate the user to be used in your project (in my case it is called p-prod-admin)
Navigate back to Manage projects view and click on 3 dots beside your project and chose edit and then click on set to start adding users to your project
On the next window, click on ADD ROLE ASSIGNMENT and choose Local User (ignore the blue warning in my screenshot below)
Specify the name of the local user we added couple of steps above (p-prod-admin) and then click on Set to assign the project admin role to that user
Choose the Project Admin role and then click on Apply
Repeat the above steps to create user p-dev-admin, p-dev project and assign project admin role to p-dev-admin. Below is how my final projects view look like
Assign DFW quota limits to projects
In the following step we will be assigning quota limits on both projects so no user can create more than 10 DFW rules. From project drop down menu click Manage and choose Quotas and then ADD QUOTA
Define quota name and on which project it will be assigned and then click on Set to set quota limits
Under objects choose Distributed Firewall Rules and set limit to 10, click on ADD and then APPLY
Repeat the same to assign a quota on DFW rules for p-dev
Create segments, security groups for VMs under prod & dev projects
In this section, we will be creating project level networking resources (T1 and logical segment) in addition to security resources (security group). Login to NSX manager UI using user p-prod-admin we created earlier:
In the above screenshot, you can see that user p=prod-admin can only see limited view to the project scope, now lets navigate to Networking and see what are the available objects than p-prod-admin can create and modify
From the above screenshot you can see that project admins can create only T1 and logical segments per project and no T0 nor VRFs (those are shared across all projects and managed by Enterprise Admins).
Now lets create a T1 called p-prod-T1 and a logical segment called p-prod-LS
Click on SAVE and then navigate to Segments and create p-prod-LS and connect it to p-prod-T1
Now, I am switching to my vCenter and connecting my prod VMs to p-prod-LS, notice in the below screenshot the NSX Port Group ID for project based logical segments and for the “Default” space logical segments:
Project based objects will have /orgs/default/projects/<project name> in their object path while non-project objects (global) will have the default reference path /infra.
Navigate back to NSX UI as p-prod-admin and from Inventory pane click on Groups and ADD GROUP to add a generic security group called Prod-VMs and I based group membership criteria based on VM names containing “prod”
Repeat the same for project (tenant) p-dev
Create and verify DFW rules within both tenants (projects)
Last step, I created a DFW policy called prod-dfw-policy which allows only SSH to prod project VMs and allows HTTP communication only between those VMs. Below is how my DFW policy looks like under p-prod project view
To verify the above DFW policy, I SSHed to my ProdVM (IP 172.160.120.20) and will try to ping my WebProd VM (172.160.120.10) this should fail since ICMP is not allowed by the above DFW, then I will HTTP to the same VM, this should succeed and see a welcome page:
Repeat the same for Dev project if needed.
Hope you have found this blog post useful!
