Configuring Multi-tenant routing using NSX-T VRF-lite

Lab Inventory

For software versions I used the following:

• VMware ESXi, 7.0.2, 17867351
• vCenter server version 7.0U3
• NSX-T 3.2.0.1
• TrueNAS 12.0-U7 used to provision NFS datastores to ESXi hosts.
• VyOS 1.4 used as lab backbone router.
• Ubuntu 20.04 LTS as Linux jumpbox.
• Ubuntu 20.04.2 LTS as DNS and internet gateway.
• Windows Server 2012 R2 Datacenter as management host for UI access.
• 2 x VyOS VMs as tenant CEs (Customer Edges).

For virtual hosts and appliances sizing I used the following specs:

• 3 x virtualised ESXi hosts each with 8 vCPUs, 4 x NICs and 32 GB RAM.
• vCenter server appliance with 2 vCPU and 24 GB RAM.
• NSX-T Manager medium appliance
• 2 x medium edges with no resource reservations.

Introduction

VRF stands for Virtual Routing and Forwarding and it is a layer 3 segregation mechanism in order to isolate different routing information on a given L3 device (router). VRF is similar to VLAN but on L3 while VLAN is of course an isolation mechanism on L2

There are two types of VRFs in general, VRF and VRF-Lite the former makes use of concept of vpn-id/route tagging so that a VRF can be “truncated”/stretched across different L3 routers. this requires special routing protocols such as MP-BGP (Multi-Protocol BGP) to propagate route tags along with routing updates from every VRF (similar to vlan tag in L2 trunks). VRF-Lite on the other hand, offers only local L3 isolation on a given L3 gateway, and this is what is currently supported on NSX.

In this blog post I am going to showcase how can you setup a simple VRF-Lite environment using NSX-T VRF gateways and utilising BGP as routing protocols between those VRF gateways and my core network.

Topology and lab breakdown

Logically the VRFs setup along with the connected tenant VMs will look like the below figure:

However in order to be able to create the above VRF gateways for both tenants we need first to create a “parent” T0 gateway as we are going further in this blog post.

For my core routing in my physical network I am using VyOs (https://support.vyos.io/en/downloads) with the below configuration:

VyOS core router configuration:

Creating tenant VRFs:
set vrf name tenant-1 table '110'
set vrf name tenant-2 table '120'
The above table id is similar to assigning a vlan id to a vlan, however this is local identification only for local routing tables within VRFs.

Creating interfaces, VLANs and assigning them to VRFs:
set interfaces ethernet eth1 vif 210 address '172.16.210.254/24'
set interfaces ethernet eth1 vif 210 description 'tenant-1'
set interfaces ethernet eth1 vif 210 mtu '1700'
set interfaces ethernet eth1 vif 210 vrf 'tenant-1'
Note: in VyOS an interface vif is creating a VLAN sub-interface on an interface and assigning it to that vlan.

set interfaces ethernet eth1 vif 220 address '172.16.220.254/24'
set interfaces ethernet eth1 vif 220 description 'tenant-2'
set interfaces ethernet eth1 vif 220 mtu '1700'
set interfaces ethernet eth1 vif 220 vrf 'tenant-2'

Configure BGP per tenant VRF
set vrf name tenant-1 protocols bgp local-as '210'
set vrf name tenant-1 protocols bgp neighbor 172.16.210.1 remote-as '65000'
set vrf name tenant-1 protocols bgp neighbor 172.16.210.1 update-source 'eth1.210'


set vrf name tenant-2 protocols bgp local-as '220'
set vrf name tenant-2 protocols bgp neighbor 172.16.220.1 remote-as '65000'
set vrf name tenant-2 protocols bgp neighbor 172.16.220.1 update-source 'eth1.220'

NSX-T configuration

Before we actually start creating our parent T0 and tenant VRFs, we need to create a set of segments that we will use later as T0 and VRFs uplinks in addition to downlinks towards the VMs (Customer Edges) which we will use to test routing within and across VRFs.

Login to your NSX-T UI, navigate to Networking > Segments

The below table gives more details about the segments created in the above diagram:

The below section is more details about the above segments configuration from NSX-T side:

T0-VRF0-uplink1

Tenant01-segment

Tenant01-vlan210-uplink

Tenant02-segment

Tenant01-vlan220-uplink

Creating parent T0 and tenants VRF-Gateways

As mentioned in the beginning of the blog post, VRF is a L3 isolation mechanism, in which routing table of a router can be virtualised into multiple isolated routing instances (VRFs). So, before you can create VRFs for your tenants you need to create a parent T0 gateway and then create VRFs on top of it.

You do not need to configure full BGP on your T0 but BGP needs to be enabled as VRF T0’s will inherit some BGP config from the parent T0 (such as AS number). For route redistribution, route maps and so on, this can be configured per VRF basis, also BGP neighbours can be configured per VRF.

From NSX UI, navigate to Networking > Tier-0 Gateways and click on ADD GATEWAY and then Tier-0

For the T0-GW we just need to create it on an edge cluster, assign uplink(s) interface(s) and enable BGP. As mentioned earlier, no need to fully configure BGP neighbours or route redistribution on the T0, since we will be doing this on per VRF tenant basis.

For the external uplink below is more details over it:

For the tenant VRFs, it is almost the same as creating the parent T0 gateway, you just need to select VRF and configure your VRF-T0 as follows:

The below is the external interface of Tenant1-VRF that will carry BGP session to core router:

Notice that the interface inherited the VLAN ID (210) from the assigned logical segment (Tenant01-vlan210-uplink) do not re-enter the ID again.

For BGP, it inherits the same AS as the one configured under VRF0 (parent T0). While for BGP neighbours this needs to be configured. I added oen BGP neighbour corresponding to Tenant-1 VRF on my core router:

Last step is to configure route redistribution into BGP to allow tenant-1 VMs to communicate to my core router. We can configure route redistribution by expanding route redistribution section under BGP and set the following route redistribution parameters:

You can select whatever subnets you need to redistribute into BGP sessions, in my case I only selected directly connected segments to my tenant-1 VRF which is the downlink to the tenant-1 VM (subnet 10.10.210.0/24)

Repeat the same above steps to create Tenant2-VRF and a neighbour BGP session to external core router.

Tenants BGP sessions and route verification

From my VyOS prompt I used the command “show ip bgp vrf <vrf-name> summary” to verify that both BGP neighbours from both Tenants VRFs are up and running:

Then we can verify which prefixes are being advertised from Tenants VRFs using the command “show ip bgp vrf <vrf-name>” on VyOS

Routing verification

The expectation is that tenants can only reach network prefixes that are present and advertised within their corresponding VRFs. To verify this, I am logging in to my tenant-1 CE router and will ping the above networks:

From vCenter, I am checking that my CE router of tenant-1 is connected to the logical segment Tenant01-segment which is created earlier and connected as downlink interface to VRF Tenant1-VRF

Since I have not setup my core routing in my core router to route within the VRFs, I am using a remote console to verify connectivity from Tenant1-CE to gateway VRF interface (10.10.210.254) and to core router interface 172.16.210.254 which is also part of Tenant-1 VRF on core router

If now I try to reach interface on Tenant-2 CE (10.10.220.10) it will fail, since I have not configured any inter-VRF routing at any stage so far

If we want to have route leaking (inter VRF routing) then we need to configure this either on the physical (core) network or by adding multi-tier routing in NSX i.e. T1 gateways underneath every T0-VRF gateway and setup static routing between them on the VRF-T0s level.

Conclusion

VRF lite provides a L3 routing information isolation which is the basis for multi-tenant routing. This allows tenants to reuse subnets without any communication conflict. However, inter-VRF routing is needed in case of tenant-to-tenant communication is required and extreme care and proper design documentation need to be followed to ensure that no undesired inter-tenant communication might occur.