# $Id$ [@background] {tr, Luigi Rizzo, 17 may 2013} {bl sz50, @@sz10,img/unipi.jpg@@} /* See http://www.bsdcan.org/2013/schedule/events/391.en.html High speed network communication is challenging on bare metal, and even more so in virtual machines. There we have to deal with expensive I/O instruction emulation, format manipulation, and handing off data through multiple threads, device drivers and virtual switches. Common solutions to the problem rely on hardware support (such as PCI passthrough) to make portions of the NIC directly accessible to the guest operating system, or specialized drivers (virtio-net, vmxnet, xenfront) built around a device model that is easier to emulate. These solutions can reach 10 Gbit/s and higher speeds (with suitably large frames), one order of magnitude faster than emulated conventional NICs (e.g. Intel e1000). Despite popular belief, NIC emulation is not inherently slow. In this paper we will show how we achieved VM-to-VM throughputs of 4 Mpps and latencies as low as 100us with only minimal modifications to an e1000 device driver and frontend running on KVM. Our work relies on four main components, which can be applied independently: 1) proper emulation of certain NIC features, such as interrupt mitigation, which greatly contribute to reduce the emulation overhead; 2) modified device drivers that reduce the number of I/O instructions, much more expensive on virtual machines than on real hardware; 3) a small extension of the device model, which permits shared-memory communication with the hypervisor without requiring a completely new device driver 4) a fast network backend (VALE), based on the netmap framework, which can sustain multiple millions of packets per second; With the combination of these techniques, our VM-to-VM throughput (two FreeBSD guests running on top of QEMU-KVM) went from 80 Kpps to almost 1 Mpps using socket based applications, and 4 Mpps with netmap clients running on the guest. Similarly, latency was reduced by more than 5 times, reaching values of less than 100 us. It is important that these techniques can be applied independently depending on the circumstances. In particular, #1 and #4 modify the hypervisor but do not require any change in the guest operating system. #2 introduces a minuscule change in the guest device driver, but does not touch the hypervisor. #4 relies on device driver and hypervisor changes, but these are limited to a few hundreds of lines of code, compared to the 3-5 Klines that are necessary to implement a new device driver and its corresponding frontend on the hypervisor. */ [Lightning fast networking in your VM] {title, Lightning fast networking in your virtual machine} {cc, Luigi Rizzo, Universita` di Pisa [[http://info.iet.unipi.it/\~luigi/vale/]] } // {cc, @@sz60,img2/tower-800x516.jpg@@} {cc, @@sz45,img2/tower-800x516.jpg@@ @@sz45,img2/google_campus2_cropped.jpg@@ } {cc, joint work with Giuseppe Lettieri and Vincenzo Maffione} /* */ [What is this about ?] {b ti,Goals of this work:} -, accelerate network I/O within VMs - typical packet rates \~1~Mpps/core, much less than line rate at 10~Gbit/s (14.88~Mpps with 64-byte frames) - 10x worse in Virtual Machines -, not just for bulk TCP connections - we may want to run high PPS applications within the VM: - software routers/middleboxes - transactions, short lived connections - Virtual Machines - Software Defined Networking -, demistify "device emulation is inherently slow" - xen, virtio, vmxnet suggest a different device model - but we show same results with e1000 within a VM -, find an application for our bag of tricks (netmap, VALE, batching...) [Main tools] {b red,Netmap (2011)} - basic packet I/O framework, OS and device independent - like raw sockets or PF_PACKET, but 10..40 times faster - 14.88 Mpps with 1 core at 900 MHz ., {b red,VALE (2012)} - Virtual Local Ethernet - generic interconnect, but targeted at Virtual Machines - up to 20~Mpps or 70~Gbit/s Both Netmap and VALE are in FreeBSD HEAD and stable/9, Linux module also available (some effort to import it there, too) [Our result] {b ti,Fast QEMU (BSDCan 2013)} - accelerated network I/O path in qemu - paravirtualized e1000 - guest-guest rates on top of e1000: - \>1~Mpps with sockets (about 6~Gbit/s with 1500-byte frames) - 5~Mpps with netmap clients in the guest (25..40 Gbit/s with 1500-byte frames) - changes are small (100..300 lines per subsystem) - FreeBSD driver patches about to be committed - qemu patches, linux guest driver patches available [Netmap] @@my40,sosp-rizzo/shared_mem.jpg@@ A new API to send/receive raw frames - relies heavily on batching - buffers and rings exposed to userspace via shared memory - select()'able file descriptor for synchronization - minimal modifications to device drivers - libpcap library eases porting of apps [Netmap API] @@my40,atc-images/shmem.gif@@ {b ti,Access} - {tt,open("/dev/netmap")} returns a file descriptor - {tt,ioctl(fd, NIOCREG, arg)} puts an interface in netmap mode - {tt,mmap(..., fd, 0)} maps buffers and rings {b ti,Transmit} - fill up to {tt,avail} buffers, starting from slot {tt,cur}. - {tt,ioctl(fd,NIOCTXSYNC)} queues the packets {b ti,Receive} - {tt,ioctl(fd,NIOCRXSYNC)} reports newly received packets - process up to {tt,avail} buffers, starting from slot {tt,cur}. {b ti, {tt,poll()} and {tt,select()} are used for synchronization} - POLLIN and POLLOUT select the rings to monitor - work as expected, returning when {tt,avail>0} - interrupt mitigation delays are propagated up to the userspace process [Netmap performance] @@fr sz50,sosp-rizzo/plot-tx-clock-201204.jpg@@ * 65 cycles/packet between the wire and userspace (14.88 Mpps at 900 MHz) * good scalability with CPU frequency and number of cores Ported several apps (OVS, Click, ipfw, ...) with 5-10x speedup - netmap exposes bottlenecks in applications - some kernel functions may now run more efficiently in userspace! [VALE: Extending netmap to virtual switching] It was relatively simple to extend the netmap code to implement a virtual switch @@cc sz50,talk-vale/img/conext-fig-sw.png@@ - NIOCREGIF {tt,valeX:Y} dynamically creates switch instances (X) and ports (Y) - each switch runs the Learning Bridge algorithm - access with exactly the same API as physical netmap ports {b,Operation is sender-driven:} - each incoming packet is dispatched to the correct destination(s) - writes are non-blocking, packets dropped when input queue full - reads are blocking - the cost of forwarding/copying is charged to the sender. [VALE performance] A straightforward implementation (one packet at a time) reaches 2..5~Mpps - much better than native bridging - but can still be improved by a large factor - locking overhead and cache misses are the performance bottlenecks {b ti,Multi-stage forwarding} @@fr sz40,talk-vale/img/fwd2.png@@ # Fetch batches of packets of {b,bounded} maximum size - prefetch payload, reducing cache miss penalties - control maximum latency # Compute destinations for each packet in the batch - allow further batching during the next stage # Forward traffic iterating on interfaces - one lock/unlock per interface {red b,18-20~Mpps with minimum-size frames} {red b,70~Gbit/s with 1500-byte frames} [Performance comparison] VALE: our software switch NIC: switch/VMDq supported forwarding (link or PCIe limited) TAP: linux bridge, BSD bridge, in-kernel OpenvSwitch {cc, @@sz40,talk-vale/img/plot-rate-ports.png@@ @@sz40,talk-vale/img/plot-rate-size.png@@} VALE is 10 .. 20 times faster than native bridging, and also faster than a 10~Gbit/s NIC [@-Summary of results] Experiments on QEMU/KVM, guest-to-guest, same host - e1000 emulation (virtio only for comparison) - linux bridge or VALE switch - FreeBSD or linux guests \\ {b ti,Main results:} - up to 1~Mpps and 6~Gbit/s using UDP sockets on the guest - up to 5~Mpps and 25~Gbit/s using netmap on the guest - More at [[http://info.iet.unipi.it/\~luigi/papers/20130206-qemu.pdf]] \\ Coding effort: - only small changes (\~600 lines in total) to hypervisor and guest - easily applicable to other hypervisors /* [Netmap and friends] Our recent work is focused on moving packets at high speed: - on bare metal ([[http://info.iet.unipi.it/\~luigi/netmap]]) - on switches and virtual switches (ovs enhancements, [[http://info.iet.unipi.it/\~luigi/vale]]) - on virtual machines (qemu and device driver enhancements) [Netmap] @@mr fr sz40,atc-images/shmem.gif@@ Netmap is our first attempt at achieving fast network I/O - OS network stack bypass, provides efficient access to raw packets I/O for user applications - line rate performance with limited CPU overhead \\ Similar in principle to other commercial alternatives (DPDK, OpenOnLoad, mellanox, ...) - netmap is OS and device independent - FreeBSD and Linux versions - Intel, RealTek, nVidia, Mellanox, Broadcom... - pcap compatibility library - minimal OS modifications - the OS is on charge of protection [Netmap key ideas and performance] @@mr fr sz60,sosp-rizzo/plot-tx-clock-201204.jpg@@ Move expensive operations outside critical paths: - one packet format - pre-allocation of data - batching - all processing within user threads [Netmap applications] 5-10x speedup on several packet capture/forwarding apps: - userspace version of ipfw+dummynet (up to 6~Mpps) - userspace openvswitch (up to 3~Mpps) - userspace Click (up to 4~Mpps) Just replacing the I/O not always sufficient: - better batching - add prefetching - remove other sources of overhead (e.g. memory allocation) [OpenvSwitch acceleration] The original openvswitch code suffered from several limitations: - expensive I/O library - expensive event loop management. Reached up to 3~Mpps in the fast path by: - creating a separate forwarding thread with reduced overhead - handling multiple packets per iteration - running on top of netmap [VALE] In-kernel software switches normally limited to about 1~Mpps, - well below line rate - share most of the bottlenecks of socket API @@fr sz50,talk-vale/img/fwd.png@@ VALE is a fast software switch based on netmap - uses netmap API on connection ports - several performance enhancements techniques 3-phase forwarding # build batches and prefetch # compute destinations # per-port aggregation and forwarding [VALE performance] @@ctr sz60,talk-vale/img/plot-rate-ports.png@@ [VALE performance] @@ctr sz50,talk-vale/img/plot-rate-size.png@@ */ [VM background] @@ctr sz60,img2/vmm-arch.gif@@ - modern CPUs run guest OS on the actual CPU (in virtual mode) - each VCPU appears to the host as a thread - main sources of overhead: VM exits and interrupt dispatching [Virtual Machine network performance] @@fr sz40,img2/vmm-arch.gif@@ Emulation of network peripherals historically slow - register access 100 times more expensive than on bare metal - VM exits and interrupt management impacts performance - host backend performance (VMM datapaths, software switches, tunnels) Paravirtualized peripherals (virtio, vmxnet, xenfront) address only one part of the problem: - PROS: reduced VM exits - CONS: new driver and device model Still need to address bottlenecks in the hypervisor and the switch [Our work] {b red,Demistifying "device emulation is inherently slow"} - data representation very similar to virtio (possibly even simpler) - all it takes is to reduce VM exits # proper interrupt moderation # send combining # e1000 paravirtualization ., {b red,More performant backend} - use netmap/VALE to replace TAP - useless unless we... ., {b red,Clean up the backend-frontend datapath} - too many abstractions and indirections - (still limited by single lock, qemu-specific) [Interrupt moderation] Real hardware can limit the maximum interrupt rate + helps building batches in the RX path + reduces interrupt load {b,BUG:} not properly emulated in hypervisors. ., {b,IMPLEMENTATION:} - \~100 lines of code, only in the frontend - fidelity depends on timer resolution ., {b,PERFORMANCE:} helps in the 1-VCPU case, but not much with 2 VCPUs: . {red tt, TX:~ 24 -> 80 Kpps (1 VCPU), 65 -> 87 Kpps (2 VCPU)} + TX still suffers from VM exists on each transmission + the second CPU absorbs most of the interrupt load . {red tt, RX: 250 -> 300 Kpps (1 VCPU), 480 -> 490 Kpps (2 VCPU)} + results depend heavily on the burstiness of the input + livelock depends on the OS [Send combining] @@fr sz50,img2/sc.jpg@@ Send combining reduces VM exits on TX: . {red, when there are pending interrupt, postpone transmissions until the next interrupt arrives} - especially useful when combined with moderation ., {b,IMPLEMENTATION:} \~100 lines, only in the guest device driver {b,PERFORMANCE: TX rate, Kpps (guest-guest, TAP backend)} . <pre class="red"> Function 1 VCPU 2 VCPU ------------------------------------ moderation: 24 -> 90 65 -> 87 send comb.: 24 -> 23 65 -> 301 both: 24 -> 322 65 -> 334</pre> [e1000 paravirtualization] @@fr sz50,img2/csb.jpg@@ Paravirtualization reduces VM exits by providing a shared memory mailbox (CSB) to implement communication between the VCPU and the IO thread + initially, an I/O or interrupt acts as a "kick" to wake up a process on the other side; + subsequently, guest driver and I/O thread chase each other exchanging read/write position through the CSB; + data buffers and descriptors are in shared memory (guest physical, host virtual) {b,IMPLEMENTATION} - no need to create a brand new device, easy addon for any modern NIC - \~100 lines in the guest driver, \~100 lines in the frontend [e1000 paravirtualization performance] Almost completely remove kicks and interrupts as packet rate increases . {red tt, 24 -> 492 Kpps (1 VCPU), 65 -> 507 Kpps (2 VCPU) } + performance equivalent to that of {tt,virtio} + does not pay the latency of interrupt moderation + reliability/portability/stability advantages \\ The bottlenecks now move to the backend / virtual switch [VALE as a virtual switch] @@mr fr sz30,atc-images/shmem.gif@@ A netmap-based QEMU backend connects to a VALE switch: + use the netmap API to connect to the switch + heavy use of batching and prefetching to improve throughput + switch performance 20~Mpps or 70~Gbit/s depending on packet size Backend uses \~350 lines of code - can talk directly to the NIC without needing PCI passthrough \\ {red,Not effective without improving the datapath in the hypervisor} [Cleaning up the hypervisor datapath] @@mr fr sz60,img2/hypervisor.jpg@@ Data moves from frontend -\> backend -\> switch + access guest descriptors and buffers + copy to hypervisor memory (backend) + pass to the switch Initial throughput was limited to \~3~Mpps even before going through the switch - removed unnecessary address translation - optimized copy routine Current code can drive the switch at 10~Mpps [Guest-guest, TAP, sockets] . <pre> \ normal send_combining paravirt. itr \ 1 cpu 2 cpu 1 cpu 2 cpu 1 cpu 2 cpu \ +-------------------------------------------------------- | 0 | 24 65 23 301 492 507 | | | 1 | 22 68 23 303 | | | 100 | 80 79 322 334 | | | 1000 | 90 87 293 323 | | | </pre> [Guest-guest, TAP vs VALE, sockets] . <pre> \ normal send_combining paravirt. itr \ 1 cpu 2 cpu 1 cpu 2 cpu 1 cpu 2 cpu \ +-------------------------------------------------------- | 0 | 24 65 23 301 492 507 | | <span class="red">27 112 27 650 1080 1080</span> | 1 | 22 68 23 303 | | <span class="red">25 97 24 670</span> | 100 | 80 79 322 334 | | <span class="red">129 125 850 860</span> | 1000 | 90 87 293 323 | | <span class="red">147 140 960 930</span> | </pre> Black: TAP, {red, Red: VALE} about 500~Kpps with 1500-byte frames Using netmap within the guest we reach \~5~Mpps or 25-Gbit/s depending on packet size. [Status and availability] There are three set of changes involved (we are actively working on some performance enhancements): {b ti,Guest OS} - FreeBSD changes for if_lem.c ready for commit - experimenting with if_em.c which supports TSO - Linux changes for e1000 also available (will likely be distributed as patches) - we are also generalizing the mechanism to support "paravirtualization" on more NICs, e.g. rtl8139 {b ti,Hypervisor} - QEMU backend sent to the qemu list a few months ago - in the process, a number of minor patches and bugfixes have been submitted to QEMU, some of them committed. - changes are non intrusive, and the same techniques are applicable to other hypervisors {b ti,Host side} - not really a change, you just need to load the netmap/VALE module [Conclusions] Netmap and related tools give a great boost to packet I/O speeds - suitable for bare metal and virtual machines - readily available for FreeBSD and Linux - extremely simple to use Many contributions come from colleagues and former students: . Matteo Landi, Gaetano Catalli, Marta Carbone, Giuseppe Lettieri, Vincenzo Maffione Funding from EU Projects (CHANGE, OPENLAB), and companies (Netapp, Google). [[http://info.iet.unipi.it/\~luigi/vale/]] {tt red,rizzo@iet.unipi.it} [Appendix - netmap/VALE status update] There is a lot of ongoing work on netmap (also thanks to Giuseppe Lettieri, Vincenzo Maffione, Michio Honda) - Aug.'12: userspace netmap-enabled ipfw+dummynet - Feb.'13: "transparent mode": a client can mark packets that should be sent to "the other side". - Apr.'13: hook NICs to a VALE switch - summer'13: use VALE as the datapath for OVS (linux only for now) - summer'13: support for scatter-gather I/O - helps TSO, fragmentation & reassembly [the end]