DPDK with OVS and Open Flow Installation: Getting Ready for User Space Switching in a Virtual Environment

DPDK with OVS and Open Flow Installation: Getting Ready for User Space Switching in a Virtual Environment

Andrei Negulescu Behnam Dezfouli

Santa Clara University

1. Introduction

This document details the installation and verification workflow of the DPDK data plane library and the Open vSwitch (in this case used as DPDK control plane) in a virtual machine environment. The objective is having a virtual switch with both control and forwarding running entirely in the user space of a Linux virtual machine. In our approach we used two different virtualization environments running on a Windows host, i.e. 1) Oracle VM VirtualBox and 2) VMware Workstation. We start by building a virtual machine on the Oracle VM VirtualBox that we will later convert to the vmdk format (and open it in VMware Workstation).

2. Building the VirtualBox Ubuntu Virtual Machine

The VirtualBox VM can be built using the VirtualBox application starting with Ubuntu iso 14.04 release or newer. Two options are available, build it yourself or use a cloud pre-built version. We decided to use an already built, Vagrant cloud release, that gives more options in terms of configuration and virtual interface management.

The currently released version of ubuntu/trusty64 box is available at https://app.vagrantup.com/ubuntu/boxes/trusty64. For the complete guide on how to use and install cloud based vagrant VirtualBox images please consult the following instructions:

The trusty-server-cloudimg-i386.ova should be installed for example in C:\Users\user\Desktop\vagrant. An example of creating the ubuntu VM follows:

#vagrant init ubuntu

#vagrant up

#vagrant ssh //use virtual box

The versions that we used are the following:

  • Linux vagrant-ubuntu-trusty-64 3.13.0-170-generic x86_64
  • ubuntu 14.04.6

We recommend the following updates on the VM side:

#sudo vim /etc/apt/sources.list

Include: deb http://archive.canonical.com/ubuntu/ trusty partner

Include: deb-src http://archive.canonical.com/ubuntu/ trusty partner

#sudo apt-get install iperf

The installation can follow using the ubuntu version 14.04, however we decided to upgrade to the latest version of ubuntu: 18.04.2.

#sudo do-release-upgrade

3. I/O module driver installation for dpdk usage

At this stage we recommend the installation of the uio-module-drv-dkms deb package. For more choices available and depending on use case please consult: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

#sudo apt-get install uio-module-drv-dkms

4. NUMA library update

DPDK requires the installation of NUMA library:

#sudo apt-get update

#sudo apt-get install libnuma-dev

5. DPDK Installation

The following steps are required for the installation of DPDK:

#sudo apt-get install git build-essential linux-headers-`uname -r`

  • Download:

#sudo git clone http://dpdk.org/git/dpdk

  • Build:

#cd dpdk

#export RTE_SDK=/home/user/dpdk

#export RTE_TARGET=x86_64-native-linux-gcc

#make install T=x86_64-native-linux-gcc

#make config T=x86_64-native-linux-gcc

#make

  • Install kernel modules (I/O):

#sudo modprobe uio

#sudo insmod build/kmod/igb_uio.ko

  • Configure “Hugepages” per DPDK recommendation (also based on the intended application and available hardware resources):

#echo 512 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

#sudo mkdir /mnt/huge

#sudo mount -t hugetlbfs nodev /mnt/huge

#sudo mkdir -p /mnt/huge_2mb

#sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB

#sudo mount -t hugetlbfs none /dev/hugepages

  • Verify the status of DPDK service

#systemctl status dpdk.service

#sudo service networking restart

  • Bind the desired network interface to the DPDK driver (the interface will have to be disabled before the bind operation occurs. Once an interface bind is successful you will not be able to list it with the ifconfig command since it is no longer seen in the kernel space).

#cd ~/dpdk

#sudo ifconfig eth2 down

#sudo ./usertools/dpdk-devbind.py –status

#sudo ./usertools/dpdk-devbind.py –status-dev net

#sudo ./usertools/dpdk-devbind.py –bind=igb_uio eth2

#sudo ./usertools/dpdk-devbind.py -u 0000:00:09.0

#sudo ./usertools/dpdk-devbind.py -b e1000 0000:00:09.0 //”move” the driver back to kernel

This should be the bind command result (use sudo ./usertools/dpdk-devbind.py –status).

########

Network devices using DPDK-compatible driver

============================================

0000:00:09.0 ‘82545EM Gigabit Ethernet Controller (Copper) 100f’ drv=igb_uio unused=e1000

########

6. OVS Installation

  • On ubuntu 18.04 install the openvswitch with dpdk version and update alternatives in order to force the use of the dpdk version instead of the no dpdk support version of openvswitch.

#sudo apt-get install openvswitch-switch-dpdk

#sudo update-alternatives –set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

  • Modify the following openvswitch config file:

#sudo vim /etc/default/openvswitch-switch

# This is a POSIX shell fragment                -*- sh -*-

# FORCE_COREFILES: If ‘yes’ then core files will be enabled.

# FORCE_COREFILES=yes

# OVS_CTL_OPTS: Extra options to pass to ovs-ctl.  This is, for example,

# a suitable place to specify –ovs-vswitchd-wrapper=valgrind.

# OVS_CTL_OPTS=

# DPDK options – see /usr/share/doc/openvswitch-common/INSTALL.DPDK.md.gz

DPDK_OPTS=’–dpdk -c 0x1 -n 4′

  • GRUB: Set the hugepage size and number then reboot the VM (example follows):

(Also included are the vfio iommu settings for “pass-through” and “SR-IOV”, more information on vfio iommu groups: https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html);

#sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT=”default_hugepagesz=1G hugepagesz=1G hugepages=2 hugepagesz=2M hugepages=1024 iommu=pt intel_iommu=on”

       #sudo vim /etc/dpdk/dpdk.conf

NR_1G_PAGES=2

NR_2M_PAGES=1024

       #sudo update-grub

#sudo reboot

  • Verify the above settings:

#grep HugePages /proc/meminfo

#cat /proc/cmdline

  • Verify the installation version. In a successful installation, OVS needs the following: (brief description attached, more details are available on openvswitch.org):

– ovsdb (user space): this is the OVS database, plus the OVSDB protocol that allows users to monitor the database content;

– ovs-vswitchd (user space): daemon that implements the switch, queries ovsdb-server in order to get its configuration from ovsdb;

– ovs-vsctl (user space): configuration utility used to configure ovs-vswitchd through changes made to the ovsdb database via ovsdb-server;

– ovs-ofctl (user space): configuration utility used for ovs-vswitchd OpenFlow features configuration;

DPDK is the forwarding plane that OVS controls in the current setup (control + forwarding in user space).

# sudo ovs-vswitchd –version

ovs-vswitchd (Open vSwitch) 2.9.2

DPDK 17.11.2

7. OVS with DPDK Initialization

For OVS configuration and management we need to create the ovsdb database. We recommend the following directory structure:

#sudo mkdir -p /etc/openvswitch

#sudo mkdir -p /var/run/openvswitch

#sudo rm /etc/openvswitch/conf.db

#sudo ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema

Once the ovsdb database is created we can start the ovsdb-server process that will answer to OVSDB queries (the following examples show the options with or without SSL support):

  • No SSL support:

#sudo ovsdb-server –remote=punix:/var/run/openvswitch/db.sock –remote=db:Open_vSwitch,Open_vSwitch,manager_options –pidfile –detach

  • With SSL support:

#ovsdb-server –remote=punix:/var/run/openvswitch/db.sock \

               –remote=db:Open_vSwitch,Open_vSwitch,manager_options \

               –private-key=db:Open_vSwitch,SSL,private_key \

                –certificate=Open_vSwitch,SSL,certificate \

               –bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert –pidfile –detach

Now we ca initialize the database:

               #sudo ovs-vsctl –no-wait init

8. OVS with DPDK Configuration

  • Once we have the ovsdb-server process running we can start ovs-vswitchd.

This configuration depends on the available hardware (CPU, memory and network interfaces) as well as on the intended network topology and use cases design. We will give an example on the hardware detection and network use case scenario in order to complete our workflow. Due to the highly configurable characteristic of the OVS with DPDK environment, users can experiment with a variety of computing and networking options. As noticed in the following examples, the “other_config” option is used to pass dpdk configuration parameters to OVS. Any change in parameter values will require to restart the ovs-vswitchd process (forwards frames and answers OpenFlow queries).

  • We have to set dpdk-init in order to have OVS support for DPDK interfaces:

# sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-init=true

  • Detect the NUMA nodes (as required by DPDK lib) and pass the pmd-cpu-mask parameter:

By passing the pmd-cpu-mask parameter to OVS, PMD (poll-mode driver) threads are created and pinned to specific CPU cores. This is the basic mechanism used by dpdk to bypass kernel system calls (avoids handling interrupts received by the cpu while processing I/O operations):

1 NUMA node

# numactl -H

available: 1 nodes (0)

node 0 cpus: 0 1 2 3

In order to find out the set of cpus a thread is eligible to run on, execute the following:

# taskset -c -p 1

pid 1’s current affinity list: 0-3

If we want to use cpu cores 1, 2 and 3 on NUMA 0 (only NUMA node available) for dpdk packet forwarding we pass the following config parameters to ovsdb:

                       #sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xE

2 NUMA nodes

                       # numactl -H

available: 2 nodes (0-1)

node 0 cpus: 0 2 4 6 8 10 12 14

node 1 cpus: 1 3 5 7 9 11 13 15

If we want to use cpu cores 2 and 4 on NUMA node 0 and cpu cores 3 and 5 on NUMA node 1 for dpdk packet forwarding we pass the following config parameters to ovsdb:

                        #sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x3C

  • DPDK lcore setting

This is an example of how to set dpdk-lcore-mask, assigning non-datapath OVS-DPDK threads to run on cpu core 2:

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x4

  • DPDK socket memory allocation is allocating hugepages on desired dpdk sockets (the following example represents two socket or one socket allocations of 1024MB and 32MB respectively):

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-socket-mem=”1024,1024″

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-socket-mem=”32″

  • The synthesis of the above would be the following sequence of commands:

sudo ovs-vsctl get Open_vSwitch . other_config

{}

#export DB_SOCK=/var/run/openvswitch/db.sock

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-init=true

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x4

#sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-socket-mem=”32″

#sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xE

#sudo ovs-vsctl get Open_vSwitch . other_config

{dpdk-init=”true”, dpdk-lcore-mask=”0x4″, dpdk-socket-mem=”32″, pmd-cpu-mask=”0xE”}

  • Starting Open vSwitch with dpdk-init option=true activates the use of the DPDK forwarding.

        # export DB_SOCK=/var/run/openvswitch/db.sock

        #sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-init=true

        #sudo ovs-vswitchd unix:$DB_SOCK –pidfile –detach

        #sudo ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-socket-mem=”1024″

        #sudo ovs-vswitchd unix:$DB_SOCK –pidfile –detach

  • Verify if the “Hugepages” are in use by ovs-dpdk. If this is the case OVS has become the control plane for DPDK.

#grep HugePages_ /proc/meminfo

#cat /proc/cmdline

9. OVS with DPDK In Use

At this point we have several options available, such as:

Import the ova format in VMware Workstation (different choice of NIC drivers and network configuration options);

Install KVM/QEMU for the virtual hosts using OVS with DPDK userspace switch;

We can also explore different network topologies based on the hardware and network driver availability. Brief examples of usage are given (http://docs.openvswitch.org, http://docs.openvswitch.org/en/latest/topics/dpdk) leaving the choice of network architecture and use cases as future work.

  • NIC driver options: Once moving to VMware Workstation, we replaced the e1000 nic driver used in the Oracle VirtualBox by “vmxnet3”. This required the modification of the vm descriptor file. We have also used the pass-through vfio-pci driver for the virtual machine nics.
  • VFIO should be supported by the kernel before using it with dpdk. Several examples are presented.    

#sudo modprobe vfio-pci

        #sudo chmod 777 /dev/vfio

        #sudo chmod 777 /dev/vfio/*

 #cd ~/dpdk

#sudo ifconfig eth2 down

#sudo ./usertools/dpdk-devbind.py –status

#sudo ./usertools/dpdk-devbind.py –status-dev net

#sudo ./usertools/dpdk-devbind.py –bind= vfio-pci eth2

//////////Example 1:

# ./usertools/dpdk-devbind.py –status

Network devices using DPDK-compatible driver

============================================

0000:0b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ drv=igb_uio unused=vmxnet3

0000:13:00.0 ‘VMXNET3 Ethernet Controller 07b0’ drv=igb_uio unused=vmxnet3

Network devices using kernel driver

===================================

0000:03:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth0 drv=vmxnet3 unused=igb_uio *Active*

0000:1b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth3 drv=vmxnet3 unused=igb_uio *Active*

No ‘Baseband’ devices detected

==============================

////////// Example 2:

Network devices using DPDK-compatible driver

============================================

0000:13:00.0 ‘VMXNET3 Ethernet Controller 07b0’ drv=igb_uio unused=vmxnet3,vfio-pci

Network devices using kernel driver

===================================

0000:03:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth0 drv=vmxnet3 unused=igb_uio,vfio-pci *Active*

0000:1b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth3 drv=vmxnet3 unused=igb_uio,vfio-pci *Active*

Other Network devices

=====================

0000:0b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ unused=vmxnet3,igb_uio,vfio-pci

////////// Example 3:

Network devices using kernel driver

===================================

0000:02:00.0 ‘82545EM Gigabit Ethernet Controller (Copper) 100f’ if=eth4 drv=e1000 unused= *Active*

0000:03:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth0 drv=vmxnet3 unused= *Active*

0000:0b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth1 drv=vmxnet3 unused= *Active*

0000:13:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth2 drv=vmxnet3 unused= *Active*

0000:1b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=eth3 drv=vmxnet3 unused= *Active*

//////////

  • Once the appropriate drivers have been bounded to dpdk we can create OVS-DPDK bridges and ports (Notice the type netdev in the example as well as dpdkvhostuser for the KVM/QEMU virtual host installation):

# sudo ovs-vsctl add-br br0 — set bridge br0 datapath_type=netdev

#sudo ovs-vsctl add-port br0 vhost_user1 — set Interface vhost_user1 type=dpdkvhostuser

#sudo ovs-vsctl add-port br0 vhost_user2 — set Interface vhost_user2 type=dpdkvhostuser

#sudo ovs-vsctl show

  • Another example of adding a bridge and dpdk ports in userspace, with the addition of Open Flow rules:

#sudo ovs-vsctl add-br br0 — set bridge br0 datapath_type=netdev

#sudo ovs-vsctl add-port br0 dpdk_port0 — set Interface dpdk_port0 type=dpdk options:dpdk-devargs=0000:13:00.0 ofport_request=1

#sudo ovs-vsctl add-port br0 dpdk_port1 — set Interface dpdk_port1 type=dpdk options:dpdk-devargs=0000:0b:00.0 ofport_request=2

#sudo ovs-ofctl del-flows br0

#sudo ovs-ofctl add-flow br0 in_port=1,action=output:2

#sudo ovs-ofctl add-flow br0 in_port=2,action=output:1

#sudo ovs-ofctl dump-flows br0

10 Conclusion and Future Work

We described the installation and initialization workflow of the Open vSwitch and DPDK for an entirely control and forwarding userspace virtual switch application. The environment is highly configurable, depending on the hardware and I/O intended to be used. Although, we started with an Oracle Virtual Box environment, this step is not necessary.

Network design and architecture are part of use cases that we did not include in this document.

Appendix

Useful customization and debug:

#!/bin/bash

//Include db config: /etc/openvswitch/conf.db

//Include system id config: system-id.conf

ovsdb-server –remote=punix:/var/run/openvswitch/db.sock \

                     –remote=db:Open_vSwitch,Open_vSwitch,manager_options \

                     –private-key=db:Open_vSwitch,SSL,private_key \

                     –certificate=db:Open_vSwitch,SSL,certificate \

                     –bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \

                     –pidfile –detach

#sudo ovs-vsctl –no-wait init

#sudo ovs-vswitchd –pidfile –detach

//////////

#find /sys/kernel/iommu_groups/

#readlink -f /sys/bus/pci/devices/0000\:1b\:00.0/iommu_group/

#ls /sys/bus/pci/devices/0000\:1b\:00.0/iommu_group/devices/

//////////

#sudo kill -9  “vswitchd_pid”

#sudo kill -9 “ovsdb-server_pid”

#sudo  rm /etc/openvswitch/conf.db

#sudo rm /var/run/openvswitch/db.sock

Stop or start ovs using the ovs-ctl script as follows:

#sudo /usr/share/openvswitch/scripts/ovs-ctl stop

#sudo /usr/share/openvswitch/scripts/ovs-ctl start

//OVS switch logs

#sudo vim /var/log/openvswitch/ovs-vswitchd.log

#sudo vim /var/log/openvswitch/ovsdb-server.log

References and Recommended Reading

Install DPDK on Ubuntu (VirtualBox VM example),https://gist.github.com/ConradIrwin/9077440

OVS DPDK INSTALL GUIDE, https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md

OVS-DPDK Parameters: Dealing with multi-NUMA, https://developers.redhat.com/blog/2017/06/28/ovs-dpdk-parameters-dealing-with-multi-numa/

DPDK Linux drivers, https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

Poll Mode Driver for Paravirtual VMXNET3 NIC, https://doc.dpdk.org/guides/nics/vmxnet3.html

Ubuntu 18.04 – VFIO PCIe Passthrough, https://forum.level1techs.com/t/ubuntu-18-04-vfio-pcie-passthrough/127011

IOMMU Groups, inside and out, https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html

Open vSwitch over DPDK on Ubuntu, https://feisky.xyz/posts/2015-03-02-open-vswitch-over-dpdk-on-ubuntu/

DPDK Physical Ports, http://docs.openvswitch.org/en/latest/topics/dpdk/phy/

Connecting VMs Using Tunnels (Userspace), http://docs.openvswitch.org/en/latest/howto/userspace-tunneling/

Using Open vSwitch with DPDK, http://docs.openvswitch.org/en/latest/howto/dpdk/

Using Open vSwitch with DPDK, https://www.openvswitch.org/support/dist-docs-2.5/INSTALL.DPDK.md.txt

Open vSwitch with DPDK, http://docs.openvswitch.org/en/latest/intro/install/dpdk/

Open vSwitch, Release 2.8.0, https://buildmedia.readthedocs.org/media/pdf/ovs-istokes/stable/ovs-istokes.pdf

Vhost-user-ovs-dpdk, https://wiki.qemu.org/Documentation/vhost-user-ovs-dpdk

Troubleshooting Resources

VirtulBox / Vagrant

https://askubuntu.com/questions/771871/16-04-virtualbox-vm-from-vhd-file-hangs-at-non-blocking-pool-is-initialized

https://www.vagrantup.com/docs/networking/public_network.html

OVS-VSWITCHD

https://software.intel.com/en-us/comment/1863239

Openvswitch

Linux-KVM-RedHat
https://www.linuxtechi.com/install-configure-kvm-ubuntu-18-04-server/

https://bugzilla.redhat.com/show_bug.cgi?id=1397550

OVS-DPDK

OVS-DPDK End to End Troubleshooting Guide, https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/ovs-dpdk_end_to_end_troubleshooting_guide/index

https://stackoverflow.com/questions/53530589/unable-to-connect-testpmd-to-ovsdpdk

http://mails.dpdk.org/archives/users/2018-August/003326.html

http://mails.dpdk.org/archives/users/2017-April/001811.html

Leave a Reply

Your email address will not be published. Required fields are marked *