4. Installation with DPDK¶
This section describes how to install DPDK-enabled Lagopus on Linux installed bear-metal server with DPDK-enabled NICs and its basic configuration. Lagopus software switch leverages DPDK for high-performance packet processing and forwarding. DPDK [1] provide a set of high-performance network-related library and user-space polling-based drivers in order to accelerate network I/O and forwarding performance on top of PC servers with standard NICs. DPDK allows both packet processing and NIC device handling in user-space not in kernel-space, therefore, special configuration are required compared with the Lagopus with raw-socket configuration (none-DPDK configuration).
To enable DPDK configuration on Lagopus software switch, you have to configure DPDK environment before its execution. Usually Ethernet NIC are under the control of Linux kernel. Therefore you can see these NIC name by ip
command, for example, ethX
or emX
. DPDK-enabled NICs are supposed to be managed under the userspace DPDK process and userspace drivers. As a result, DPDK-enabled NICs are supposed to be out of the control of Linux kernel. Then you will not see the name of DPDK-controlled NICs by ip
command.
DPDK-enabled Lagopus can run on a hypervisor-based virtual machine, such as KVM, VMware, and VirtualBox. Therefore, you can test DPDK-enabled Lagopus on your environment. This installation steps are confirmed on VirtualBox VM, which should be identical to the steps on bear-metal server. If you want to run Lagopus with raw-socket configuration instead of DPDK-enabled Lagopus, refer to Installation with raw-socket configuration.
[1] | For detail DPDK information, check http://dpdk.org/ |
4.1. Minimum hardware requirement¶
Lagopus software switch with DPDK configuration uses multiple CPU cores to achieve high-performance network I/O and processing performance. Therefore, you have to prepare a host machine that meets the following minimum hardware requirements.
- # of CPU cores: 2 or more
- Memory size: 2 GB or more
- NIC: DPDK-enabled NICs
Warning
You need 2 or more processors (CPU cores) to run Lagopus software switch using DPDK.
4.2. Software Versions¶
- Lagopus : Lagopus software switch 0.2.10
- OS: Ubuntu Server 16.04 LTS
- DPDK: 2.2 or above
4.3. Installation steps¶
Install Ubuntu to a bear-metal server or a VM.
Install necessary packages.
$ sudo apt-get update $ sudo apt-get install build-essential linux-headers-$(uname -r) \ libexpat-dev libgmp-dev \ libssl-dev libpcap-dev libnuma-dev byacc flex git \ python-dev python-pastedeploy python-paste python-twisted
Download Lagopus source code.
You have two options to get Lagopus source code by downloading tar file or by
git
.$ wget https://github.com/lagopus/lagopus/archive/v0.2.10.tar.gz $ tar xvf v0.2.10.tar.gz $ ls lagopus-0.2.10 v0.2.10.tar.gz $
Or
$ git clone -b v0.2.10 --recursive https://github.com/lagopus/lagopus.git $ ls lagopus $
Compile DPDK-enabled Lagopus software switch
The source code of DPDK library is automatically download by git submodule mechanism. You don’t have to download by manual.
$ ./configure $ make
Note
If you find an error in configure process or make process, check git submodule setting. Sometimes git submodule operation may be failed due to firewall or proxy setting on your environment.
/home/<usr>/lagopus-0.2.10/mk/make_dpdk.sh /home/<usr>/lagopus-0.2.10 src/dpdk \ "x86_64" "linuxapp" gcc fatal: Not a git repository (or any of the parent directories): .git make[1]: *** [dpdk] Error 1 make[1]: Leaving directory `/home/<usr>/lagopus-0.2.3' make: *** [prerequisite] Error 2 configure: error: Prerequisite failure.
Install lagopus package
$ sudo make install
4.4. Setup DPDK¶
To enable DPDK on your environment, you have three steps, kernel module setup, hugepage setup, DPDK-enabled NIC assignment. The above compilation process of Lagopus software switch, DPDK library and drivers are compiled automatically. Thus you just perform the following DPDK configurations.
4.4.1. Setup kernel module¶
DPDK provides two kernel modules, igb_uio
and rte_kni
, to realize userspace NIC drivers and network packet processsing. The igb_uio
is a wrapper module for DPDK-enabled NICs on top of UIO module of Linux kernel. The igb_uio
allows userspace DPDK driver access to memory-mapped registers on NICs directly.
The rte_kni
module enables packet frame exchanges between an user space DPDK application and network stack in kernel space.
Load UIO and kernel modules.
The kernel modules built are available in the
src/dpdk/build/kmod
directory.$ sudo modprobe uio $ cd lagopus $ sudo insmod ./src/dpdk/build/kmod/igb_uio.ko $ sudo insmod ./src/dpdk/build/kmod/rte_kni.ko $ lsmod | egrep 'uio|kni' rte_kni 282624 0 igb_uio 16384 0 uio 20480 1 igb_uio $
Note
You have to perform the above steps above after the OS reboot.
4.4.2. Setup Hugepages¶
Make hugepages available to DPDK. You can setup hugepage in two ways:
- Manual configuration: Repeate steps after reboot if you select this.
- Script configuration: Select this to keep it permanent after reboot.
Note
When configured manually with , you need the same steps after rebooting OS.
Manual confguration (undone after reboot)
Perform the following commands
$ sudo sh -c "echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages" $ sudo mkdir -p /mnt/huge $ sudo mount -t hugetlbfs nodev /mnt/huge
Script configuration (permanent after reboot)
Reserve 256 pages of 2 MB hugepages in linux by adding the following line in
/etc/sysctl.conf
.$ sudo vi /etc/sysctl.conf vm.nr_hugepages = 256 $
Enable permanent across reboots. Add a directory for hugepages and the following line to
/etc/fstab
so that mount point can be made permanent across reboots.$ sudo mkdir -p /mnt/huge $ sudo vi /etc/fstab nodev /mnt/huge hugetlbfs defaults 0 0
Confirm HugePages are configured correctly by the below commands.
$ grep -i "HugePages" /proc/meminfo AnonHugePages: 0 kB HugePages_Total: 256 HugePages_Free: 256 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB $ mount | grep huge nodev on /mnt/huge type hugetlbfs (rw) $
4.4.3. NIC (Network Interface Card) assignment¶
The following steps detatch the control and management of NICs, which will be used for DPDK application, from Linux kernel.
In this example, the host has three NICs, we are going to use eth1
, eth2
for Lagopus software swtich.
Perform the following steps to enable NICs for DPDK application.
Note
- You need to unbound NIC from kernel (ixgbe driver) before using it with DPDK.
- You will lose connection to the OS if you unbound NIC used for management plane (ex: ssh).
Check PCI ID of the NICs you want to use for DPDK with
dpdk-devbind.py
script.dpdk-devbind.py
script displays DPDK-enabled NIC information, such as PCI ID and interface name in Linux kernel.$ cd lagopus $ sudo ./src/dpdk/tools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused= *Active* 0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth1 drv=e1000 unused= 0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused= Other network devices ===================== <none> $
In this example, You can see PCI IDs of
eth1
andeth2
from the output,0000:00:08.0
and0000:00:09.0
.¶ PCI ID Linux IF name Bounded by Linux 0000:00:03.0 eth0 Yes 0000:00:08.0 eth1 Yes 0000:00:09.0 eth2 Yes Unbound NICs from ixgbe driver and registerd with igb_uio driver.
Perform the dpdk-devbind with the PCI IDs to be unbounded from Linux kernel.
~/lagopus$ sudo ./src/dpdk/tools/dpdk-devbind.py --bind=igb_uio 0000:00:08.0 0000:00:09.0 ~/lagopus$ sudo ./src/dpdk/tools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused= 0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused=igb_uio *Active* Other network devices ===================== <none>
Memorize the DPDK NIC configuration
After the unbound NICs from Linux kernel, the NIC which was bounded to
eth1
is accesssed byDPDK port #0
or PCI ID (0000:00:08.0
) directly. The NIC which was bounded byeth2
is also accessed byDPDK port #1
or PCI ID (0000:00:09.0
).¶ PCI ID Linux IF name Bounded by Linux DPDK ready DPDK port # 0000:00:03.0 eth0 Yes No 0000:00:08.0 eth1 No Yes 0 0000:00:09.0 eth2 No Yes 1
4.5. Setup Lagopus configuration file¶
Example Lagopus configuration (DSL format) can be found at “misc/examples/lagopus.dsl”.
lagopus.dsl
file must be located at the same directory of the executable of lagopus
, or under /usr/local/etc/lagopus/
.
Copy sample configuration file under
/usr/local/etc/lagopus/
.$ sudo mkdir /usr/local/etc/lagopus/ $ cd ~/lagopus-0.2.10 $ sudo cp misc/examples/lagopus.dsl /usr/local/etc/lagopus/lagopus.dsl
Edit configuration file suited to your environment.
Example:
- One OpenFlow controller: “127.0.0.1”
eth0
: management interface. (Thus does not appear in the configuration)DPDK port #0
which waseth1
andDPDK port #1
which waseth2
: Lagopus dataplane ports. These two ports are accessed with DPDK.
$ sudo vi /usr/local/etc/lagopus/lagopus.dsl channel channel01 create -dst-addr 127.0.0.1 -protocol tcp controller controller01 create -channel channel01 -role equal -connection-type main interface interface01 create -type ethernet-dpdk-phy -port-number 0 interface interface02 create -type ethernet-dpdk-phy -device :0000:00:09.0 # Since v0.2.10, interface can be specified by PCI ID. port port01 create -interface interface01 port port02 create -interface interface02 bridge bridge01 create -controller controller01 -port port01 1 -port port02 2 -dpid 0x1 bridge bridge01 enable $
4.6. Running / Stopping Lagopus software switch¶
4.6.1. DPDK command option¶
In order to run Lagopus softwarwe switch with DPDK configuration, you need to specify DPDK-related options in lagopus
command: which CPU cores are assigned to packet processing (-c
), how many memory channels the host has (-n
).
In this example, the host has four CPU cores and two memory channel and you try to assign CPU core #1 and CPU core #0 for network I/O and processing and DPDK port #0 and DPDK port #1.
With -c
option, you should specify which CPU cores are assigned to Lagopus software switch. The value of -c
option uses the hexadecimal notation that its N -bit shows whehter CPU core # N is used or not for DPDK. The following table help your understanding of CPU flags.
cpu core # 3 | cpu core # 2 | cpu core # 1 | cpu core # 0 | flag in binary | flag in hexadecimal |
---|---|---|---|---|---|
0 | 0 | 1 | 1 | 0x0011 | 0x3 |
4.6.2. Run Lagopus software switch¶
Peform the following command to run Lagopus software switch in foreground with debug mode.
$ sudo lagopus -d -- -c3 -n2 --
Or
Peform the following command to run Lagopus software switch in background.
$ sudo lagopus -- -c3 -n2 --
4.6.3. Operation of Lagopus software switch with lagosh
¶
Enter
show version
command from lagosh to confirm it’s running.Enter
stop
command from lagosh to stop Lagopus vswitch.$ lagosh Lagosh> show version { "product-name": "Lagopus", "version": "0.2.10-release" } Lagosh> stop Lagosh> show version Socket connection refused. Lagopus is not running? Lagosh> exit $
For more configuration options, refer to Lagopus software switch command options and examples
For more information about lagosh, refer to Using Lagopus CLI (lagosh)
Note
Lagopus software switch with DPDK use pooling-based packet processing. Therefore you will see lagopus consume 100% of CPUs by top
command.
top - 15:50:26 up 6 min, 1 user, load average: 0.34, 0.13, 0.07
Tasks: 83 total, 2 running, 81 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 3081312 total, 686804 used, 2394508 free, 20676 buffers
KiB Swap: 3143676 total, 0 used, 3143676 free. 78792 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1205 root 20 0 1037728 16264 6764 S 100.2 0.5 0:25.58 lagopus