# $Id: demo-plot.txt 8558 2011-04-23 22:10:21Z luigi $ [@background] {tr, Luigi Rizzo, UNIPI, Aug. 2011} {bl, @@sz5,img/unipi.jpg@@} {br, @@fr sz10,img/change-logo-small.jpg@@} [Netmap] {title, netmap: a novel framework for high speed packet access} @@logo,img/bolt.jpg@@ {b author, Luigi Rizzo, Universita` di Pisa \\ {b fn60,with help from \\ Gaetano Catalli, Marta Carbone, Matteo Landi} } {author, supported by EC Project CHANGE \\ [[http://info.iet.unipi.it/~luigi/netmap/]] } [Motivations] We have been doing packet processing in software on general purpose OS for the past 30 years + routing/switching; + firewalls + traffic analysis; + network monitoring; + network testing. Plenty of this software in use even in production systems [Motivations] Network speeds have increased by 3 orders of magnitude in the meantime\\ @@sz40,img/morse1.jpg@@ @@sz40,img/10g.jpg@@ But software architectures are still the same - raw socket, BPF, libpcap - mbuf/skbuf/NdisPacket encapsulation // - one system call per packet, poor parallelism Even with faster clock speeds, there are some things that do not scale: - memory and bus latency, system calls [Motivations] At 10 Gbit/s the packet rate is 14.88 Mpps, or 67.2 ns/pkt.\\ \\ {h2 ctr red,How do we deal with such rates ?} <table class="fn80"><tr><td> use more powerful hardware \\ @@sz50,img/2403851423_eabf683a59_z.jpg@@ </td> <td> use existing hardware the best we can\\ @@sz80,img/10g.jpg@@ </td></tr></table> [So, what have we got ?] {tt,netmap} is a novel framework (available on FreeBSD) for fast packet I/O from userspace. @@my50,plot-tx-clock.jpg@@ * 70..90 cycles per packet between the wire and userspace code (14.88 Mpps at 1.2 GHz) * good scalability with CPU frequency and number of cores * easy API * {tt,libpcap} emulation library for easy porting of applications [And how ?] Summary of the talk - existing options for packet I/O (and what's wrong with them) - how {tt,netmap} addresses these problems - the netmap API - internals - performance - experience on porting applications - future work [Existing options for packet I/O] {green b,userspace:} rich and friendly development/runtime environment + packet I/O uses various kinds of sockets, or BPF, or libpcap + {b red,expensive system calls and data copies} just to move packets from/to the kernel {green b, in-kernel:} dangerous and constrained environment + modules using netfilter/pfilter/Ndis hooks + direct device management (Click) {green b, some experimental proposals:} not for everyone - various libpcap optimizations (mmap, batching...): only solve half of the problem. - direct device access (UIO-IXGBE, DNA): as dangerous as in-kernel options - custom hardware (NetFPGA, DAG, ...): hw-dependent [In-kernel operation] Even remaining in the kernel is expensive - the NIC can handle lists of buffers without CPU intervention - but the OS adds a {red b,costly encapsulation} (per-packet {b,malloc/free}, refcounted pointers, data sharing, ...); @@ctr sz70,img/mbufs.jpg@@ [Netmap goals and approach] {ctr red b, netmap focuses on packet I/O for userspace applications.} Fix the weaknesses of existing APIs, taking good ideas from existing proposals: - packet encapsulation, system call costs, data copies - remove unnecessary allocations - {green,don't be afraid of change (netmap could have been done 10 years ago!)} Most of these ideas have been proposed and used before. {red b,But separately.}\\ A collection of ideas does not make a system, and we want to design and build one that is {red,usable} and {red,maintainable}: - simple API, device indepedent - easy to import and maintain, compatible with existing applications [Netmap data structures] We need an efficient representation for batches of packets - amortize syscall costs - remove allocations - reduce data copies Hint: look at the NIC ring @@ctr sz50,img/mbufs.jpg@@ [The netmap ring] @@my50,img/netmap-ring.jpg@@ A shadow copy of the NIC ring - global fields:\\ {tt b,cur}: current rx/tx position,\\ {tt,avail}: available slots - compact (8 bytes) slots:\\ {tt, length, buffer index, flags} - all addresses are relative (position independent). Data structures are in {tt,mmap()}-ed memory. {b, Ownership} is well defined: - {red,global fields, and {tt,[cur..cur+avail-1]} :} owned by the application\\ synchronized with the kernel during {tt,poll()} and {tt,ioctl(fd,NIOC*XSYNC)} - {red,other slots/bufs:} owned by the kernel these rules permit lock free access to the rings and buffers [Netmap API] @@my50,img/netmap-ring.jpg@@ {h1,Access} - {tt,open("/dev/netmap")} returns a file descriptor - {tt,ioctl(fd, NIOCREG, arg)} puts an interface in netmap mode - {tt,mmap(..., fd, 0)} maps buffers and rings {h1,Transmit} - fill up to {tt,avail} buffers, starting from slot {tt,cur}. - {tt,ioctl(fd,NIOCTXSYNC)} queues the packets {h1,Receive} - {tt,ioctl(fd,NIOCRXSYNC)} reports newly received packets - process up to {tt,avail} buffers, starting from slot {tt,cur}. [Netmap API - synchronization] @@my50,img/netmap-ring.jpg@@ - {tt,ioctls(fd,NIOC*SYNC)} are non blocking - {tt,poll()} and {tt,select()} work as expected, returning when {tt,avail>0} - POLLIN and POLLOUT decide which sets of rings to work on - interrupt mitigation delays are propagated up to the userspace process - can use the {tt,ioctl()}s for busy-waiting [Multiqueue/Multicore support] @@my50,img/netmap-ring.jpg@@ Multiqueue cards and multicore systems are fully supported: + one netmap ring per physical ring on the NIC; + by default, the file descriptor operates on all rings + {tt,ioctl(fd, NIOCREG, arg)} can restrict the association to a single ring; + multiple {tt,open("/dev/netmap")} on the same card can be bound to different rings (and issued by different thread); + {tt,setaffinity()} can be used to map threads to cores [Netmap and the host stack] @@my30,img/host-stack.jpg@@ While in netmap mode, the control path remains unchanged: - ifconfig, ioctl's, etc still work as usual - the OS still believes the interface is there The datapath is detached from the host stack: - packets from the net end up in RX netmap rings - packets from TX netmap rings are sent to the net {red b,The host stack is attached to an extra pair of netmap rings:} - packets from the host go to a {red,SW RX} netmap ring - packets from a {red,SW TX} netmap ring are sent to the host - the two extra rings are also managed using the netmap API. [Example of use] Using the native API is very simple. Real code for a traffic generator (handles multiple queues, does line rate at 10 Gbit/s) @@fr sz40,img/netmap-ring.jpg@@ . <pre class="fn80"> fds.fd = open("/dev/netmap", O_RDWR); strcpy(nmr.nm_name, "ix0"); ioctl(fds.fd, NIOCREG, &nmr); p = mmap(0, nmr.memsize, fds.fd); nifp = NETMAP_IF(p, nmr.offset); fds.events = POLLOUT; for (;;) { poll(fds, 1, -1); for (r = 0; r < nmr.num_queues; r++) { ring = NETMAP_TXRING(nifp, r); while (ring->avail-- > 0) { i = ring->cur; buf = NETMAP_BUF(ring, ring->slot[i].buf_index); ... store the payload into buf ... ring->slot[i].len = ... \/\/ set packet length ring->cur = NETMAP_NEXT(ring, i); } } } </pre> [Multiple interfaces] All rings and buffers for all interfaces in netmap mode are in the {b,same} mmapped region: - can do zero-copy forwarding by just swapping buffer indexes @@fr sz40,img/netmap-ring.jpg@@ . <pre> src = &src_nifp->slot[i]; dst = &dst_nifp->slot[j]; tmp = dst->buf_index; dst->buf_index = src->buf_index; src->buf_index = tmp; dst->len = src->len; dst->flags = src->flags = BUF_CHANGED; </pre> Zero-copy also works with the rings from/to the host stack. - firewalls, NAT boxes, IDS can use this mechanism [Safety and isolation] {ctr red b,netmap is meant to be used by trusted processes running with root privileges.} The content of NIC rings is validated by the syscalls: - no chance to crash the kernel or overwrite other memory - protection at the page-level provided by the OS Packet corruption still possible if a process is misbehaving - but compare with the alternatives:\\ in-kernel modules or direct NIC access are even more dangerous {h1,Planned work: implement {em,virtual} per-process netmap rings} - needs data copying for full isolation - needs to design a mechanism to demux incoming packets - still saves on the syscall and in-kernel representation - can we do it as a userspace process ? [The implementation] Current prototype developed on FreeBSD (HEAD and RELENG_8) {b,kernel side:} 2000 lines of (heavily commented) common code and headers, small device-specific patches . <pre class="fn80"> $ wc -l net/* dev/netmap/* 283 net/netmap.h 98 net/netmap_user.h 393 dev/netmap/if_bge_netmap.h 425 dev/netmap/if_em_netmap.h 375 dev/netmap/if_lem_netmap.h 436 dev/netmap/if_re_netmap.h 425 dev/netmap/ixgbe_netmap.h 306 dev/netmap/netmap.4 1691 dev/netmap/netmap.c 225 dev/netmap/netmap_kern.h </pre> {b,Userspace:} just a few macros in netmap_user.h {b,Extra:} a small libpcap emulation layer is enough to port many existing applications <pre class="fn80"> 756 examples/pcap.c </pre> [Internals] The netmap code is designed to minimize changes to the OS sources - the common code is a completely independent module - per-device code has all additional handlers in an external file <pre class="fn60"> int (*nm_register)(struct ifnet *, int onoff); void (*nm_lock)(void *, int what, u_int ringid); int (*nm_txsync)(void *, u_int ring, int lock); int (*nm_rxsync)(void *, u_int ring, int lock); </pre> plus very small patches to the ring initalization and interrupt service functions <pre class="fn60"> ... +#ifdef DEV_NETMAP +#include \<dev/netmap/if_re_netmap.h> +#endif ... +#ifdef DEV_NETMAP + re_netmap_tx_init(sc); +#endif ... +#ifdef DEV_NETMAP + if (ifp->if_capenable & IFCAP_NETMAP) { + selwakeuppri(&NA(ifp)->rx_rings->si, PI_NET); + return 0; + } +#endif ... </pre> [Internals (2)] Devices in netmap mode are essentially insensitive to livelock: - the interrupt service routine does nothing but a {tt,selwakeup()} - NIC rings are scanned by the upper half of the kernel in {tt,ioctl()} and {tt,poll()} - the system remains perfectly controllable even under severe overload {tt,poll()} and {tt,select()} are optimized for common usage patterns: - unnecessary calls (those with {tt,avail \> 0}) are lock-free - pending transmissions are sent out even if POLLOUT is not specified - {tt,poll()} and {tt,select()} set a timestamp in the ring These two features typically save two system calls per iteration of eventloops. [Libpcap emulation] @@fr sz30,img/netmap-ring.jpg@@ Many applications use libpcap so we built a simple emulation library (many stubs, plus the two real I/O functions): <pre class="fn80"> int pcap_inject(pcap_t *p, void *buf, size_t size) { int si, idx; for (si = p->begin; si < p->end; si++) { struct netmap_ring *ring = NETMAP_TXRING(p->nifp, si); if (ring->avail == 0) continue; idx = ring->slot[ring->cur].buf_idx; bcopy(buf, NETMAP_BUF(ring, idx), size); ring->slot[ring->cur].len = size; ring->cur = NETMAP_RING_NEXT(ring, ring->cur); ring->avail--; return size; } return -1; } </pre> [Libpcap emulation (2)] @@fr sz30,img/netmap-ring.jpg@@ <pre class="fn80"> int pcap_dispatch(pcap_t *p, int cnt, pcap_handler callback, u_char *user) { int si, ret = 0; for (si =p->begin; si < p->end; si++) { struct netmap_ring *ring = NETMAP_RXRING(p->nifp, si); while ((cnt == -1 || ret != cnt) && ring->avail > 0) { int i = ring->cur; int idx = ring->slot[i].buf_idx; p->hdr.len = p->hdr.caplen = ring->slot[i].len; callback(user, &p->hdr, NETMAP_BUF(ring, idx)); ring->cur = NETMAP_RING_NEXT(ring, i); ring->avail--; ret++; } } return ret; } </pre> [Performance] Most of the cost is per-packet, so the relevant metric is packets per second (pps), not bytes per second - transmit and receive - forwarding (native and with pcap emulation) - ported applications (OpenvSwitch, Click userspace) All experiments using intel 82599 dual-port card, i7-870 CPU (2.93 GHz, 4 cores) if not specified otherwise. {h1,Warning/disclaimer:} - we try to measure only the packet I/O overhead, not the application - CPU-intensive applications will benefit less from efficient I/O - many cards (even 1 Gbit models) cannot do line rate irrespective of CPU speed - buses, caches, CPU and memory speeds also affect your results Anyways, the code is available, you can try it yourself. [Tx throughput versus clock speed] Tx performance with 1, 2, 4 cores at variable clock speeds (0.15..2.93 GHz).\\ 10 Gbit/s interface, 64-byte packets ({tt,examples/pkt-gen.c}) @@ctr sz60,plot-tx-clock.jpg@@ [Tx throughput versus burst size] Tx performance vs burst size. \\ 10 Gbit/s interface, 64-byte packets, 2.93 GHz. @@ctr sz60,plot-tx-burst.jpg@@ [Rx throughput versus packet size] Receive does line rate at multiple of 64 bytes, at other sizes the hardware limits the throughput (reported by others as well). Measured with {tt,examples/pkt-gen.c} in receive mode. @@ctr sz60,plot-rx-size.jpg@@ [Forwarding performance] Forwarding speed between two interfaces, using 1 core. Measured with {tt,examples/bridge.c} and {tt,examples/pcap.c} @table l r {b,Configuration} | {b,Mpps} native API | 10.66 native API + timestamp | 9.42 libpcap emulation | 7.50 freebsd bridging | 0.69 @endtable [Ported applications] The libpcap emulation was used to port/test various applications. - just run with LD_LIBRARY_PATH=path/to/new_libpcap - many simple apps (such as trafshow) just run much faster out of the box - others needed tweaks to exploit the faster library {h1,Case studies} - OpenvSwitch, userspace module (from 65 Kpps to 3 Mpps) - Click userspace (from 400 Kpps to 3.95 Mpps) [OpenvSwitch] OpenvSwitch is a software implementation of an OpenFlow controller plus an OpenFlow switch - two forwarding engines: in-kernel and userspace (experimental) - written in portable C, but uses Linux APIs We ported the userspace module to FreeBSD using libpcap - initial performance was poor (50..65 Kpps, both for Linux and FreeBSD) due to what we believe an oversight in the design - fixing the problem brought the userspace performance to 780 Kpps (comparable with FreeBSD bridging, 690 Kpps) - replacing libpcap with our version went up to 3 Mpps {ctr red h1,Even the optimized version runs almost 4 times faster} [Click] Click is a popular packet processing platform. Elements build packet processing graphs: {ctr tt,FromDevice(ix0) -> Queue -> ToDevice(ix1)\\ FromDevice(ix1) -> Queue -> ToDevice(ix0)} - in kernel version with modified device drivers (Linux) - userspace version: 400 Kpps with libpcap, 1.3 Mpps with netmap - the C++ allocator was the bottleneck. Fixes now in Click 2.0 Forwarding performance with one core: . @table l r r {b,Version} | {ctr b,Kpps (orig.) } | {ctr b,Kpps (new)} kernel (estimated) | 3-4000 | - libpcap | 400 | 495 netmap | 1300 | 3950 @endtable {ctr red h1,Click userspace is now competitive or better than the in-kernel version} [Related work] Various mechanisms available for raw packet I/O + most of them retain the driver encapsulation (mbuf/skbuf): - netslice, PF_RING, libpcap variations with mmap support; - Click-based prototypes (including RouteBricks) - in-kernel modules: netfilter/pfilter/Ndis clients, netgraph. reduced syscall/copy overhead, still high per-packet overhead. + some control directly the hardware:\\ - PF_RING-DNA, UIO-IXGBE need hardware-specific libraries;\\ as dangerous as kernel applications (unless IO-MMU is used) + custom hardware and associated drivers - (DAG, NetFPGA) hardware specific [Related work (2)] The PacketShader Packet I/O engine [[http://shader.kaist.edu/packetshader/io_engine/]] is the closest to our work in terms of performance and architecture - reported performance similar to netmap - exports data structures similar to netmap rings - uses ioctl() to syncronize kernel and userspace Limitations - custom device driver, only supports one specific device - does not seem to support {tt,select()/poll()} - no pcap emulation - no documentation [Availability] [[http://info.iet.unipi.it/~luigi/netmap/]] is netmap's home, continuously updated with - code snapshots (kernel, libraries, examples) - bootable images (PicoBSD-based) - papers and presentations Currently works on - FreeBSD HEAD (and RELENG_8 with minimal effort) - ixgbe, e1000, re drivers - user-contributed igb and bce drivers [Future work] Boring but useful tasks - small performance optimizations - support more cards - port/fix more applications (either on libpcap or native API) - linux port Explore architectural issues - per-user virtual rings - improved safety and isolation - bring subsystems to userspace - firewalls (ipfw) and schedulers - routing tables [Conclusions / Acks / Links] {ctr red h1, Line-rate at 10 Gig in userspace is possible without any magic.} {h1,Thanks to:} + the Change Project @@sz10,img/change-logo-small.jpg@@ + Intel Research Berkeley + my students Gaetano Catalli, Marta Carbone, Matteo Landi {h1,Netmap home page} + [[http://info.iet.unipi.it/~luigi/netmap/]]