busdma
-The busdma interfaces permit hardware device drivers to operate - on a variety of platforms avoiding the encoding of platform-specific - access methods into drivers. This lowers the maintenance costs - for drivers across platforms, and improves the chances that a driver - will "just work" on a new platform. Modifying a driver to make use - of busdma is relatively straight forward, but does require - familiarity with both the device driver and busdma primitives. - For busdma to be used in FreeBSD, two sets of changes are generally - required: adaptation of the busdma implementation to run on all - platforms, and adaptation of drivers to use the framework. As such, - status information on this project is broken down into platform - support, and driver support (sorted by category). Completing - this work requires a thorough audit of the system device drivers, - then prioritized conversion of drivers. Drivers are also - expected to use bus_space functions, and this column is - sometimes used to denote a driver in need of conversion to - bus_space as well.
- -INTR_MPSAFE
-Hardware drivers register their interrupt handler with the
- bus_setup_intr() function. Setting the flag
- INTR_MPSAFE tells the system interrupt code
- to call the interrupt routine without holding the Giant
- mutex. This can give a significant performance gain on SMP
- systems.
Drivers can set this flag even if they are not fully locked - down as long as their interrupt routine is careful about not - touching other data structures in the driver. An easy way to - do this is to check and clear the hardware interrupt status - registers and then schedule the interrupt processing for a - taskqueue or kernel thread.
- -SMPng locked
-Drivers should employ mutexes and sx locks to protect their - data structures and hardware registers from competing threads. - Mutex operations are somewhat expensive, so a good strategy is - combine as many atomic operations into a single mutex acquisition - as possible.
- -p!=a safety
-Intel PAE support requires that pointers and physical address - representations be of differing sizes. This means that drivers - must be written to use vm_paddr_t or bus_addr_t rather than - assuming that physical addresses can be represented using a - void *. In addition, format strings and casts must be - carefully handled.
- -The task list below is not intended to be complete, but does - represent a set of relevant and/or important components of the - overall work. The "Responsible" field identifies a developer who - has expressed willingness to be responsible for completing the - identified task; this doesn't preclude others working on it, - but suggests that coordination with the responsible party might - be appropriate so as to avoid unnecessary duplication of work, - and to maximize forward progress. If beginning work on a new - area of substantial size, or one that appears unclaimed, it may - be worth dropping an e-mail to &a.mux; to see if any progress - has been made.
- -The definition of the date field varies depending on the status - of a task. For completed tasks, it refers to the date completed - or reported completed. For in-progress tasks, it refers to the - date of the last update of the entry. For stalled tasks, it - refers to the date that the task was declared stalled. For - new tasks, it refers to the date the task was added to the list.
- -Tasks are sorted first by status, then by date.
- - -A series of manual pages related to this project can be found - here:
- -| Task | -Responsible | -Last updated | -Status | -Details | -
|---|---|---|---|---|
| amd64 | -&a.peter; | -July 1, 2003 | -&status.done; | -Fully supported. | -
| arm | -&a.cognet; | -December 23, 2005 | -&status.done; | -Fully supported. | -
| ia64 | -&a.marcel; | -December 10, 2002 | -&status.done; | -There may be problems for systems with large amounts - of memory. | -
| i386 | -&a.sam; | -December 9, 2002 | -&status.done; | -Fully supported. | -
| powerpc | -&a.grehan; | -January 15, 2003 | -&status.done; | -Fully supported. | -
| sparc64 | -&a.tmm; | -January 6, 2003 | -&status.done; | -Fully supported. | -
| Driver | -Responsible | -Last updated | -busdma | -INTR_MPSAFE | -SMPng locked | -a!=p | -Notes | -
|---|---|---|---|---|---|---|---|
| if_an | -- | December 23, 2005 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_ar | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -kvtop() | -
| if_arl | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_ath | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_aue | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_awi | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_axe | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_bce | -- | August 25, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_bfe | -&a.silby; | -August 23, 2006 | -&status.done; | -&status.wip; | -&status.wip; | -&status.wip; | -- |
| if_bge | -&a.wpaul; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_cdce | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_cnw | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_cp | -&a.rik; | -October 31, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_cs | -&a.imp; | -December 23, 2005 | -&status.new; | -&status.new; | -&status.new; | -&status.unknown; | -Needs bus_space conversion | -
| if_ct | -&a.rik; | -October 31, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_cue | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_cx | -&a.rik; | -June 24, 2004 | -&status.done; | -&status.wip; | -&status.wip; | -&status.new; | -- |
| if_dc | -&a.mux; | -August 19, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_de | -&a.mux; | -August 17, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_ed | -&a.imp; | -December 23, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_em | -&a.pdeuskar; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_en | -&a.harti; | -November 2, 2005 | -&status.done; | -&status.new; | -&status.new; | -&status.done; | -Locking present; not yet marked INTR_MPSAFE? | -
| if_ep | -&a.mdodd;,&a.imp; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_ex | -&a.imp; | -Dcember 23, 2005 | -&status.done; | -&status.new; | -&status.new; | -&status.done; | -- |
| if_fatm | -&a.harti; | -November 2, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_fwe | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -- |
| if_fxp | -&a.mux; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_gem | -&a.tmm; | -July 31, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_hatm | -&a.harti; | -November 2, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_hme | -&a.tmm; | -January 30, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_idt | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| if_ipw | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_iwi | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_ixgb | -- | August 21, 2006 | -&status.untested; | -&status.new; | -&status.new; | -&status.new; | -- |
| if_kue | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_le | -&a.marius; | -January 31, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_lge | -- | November 23, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_lmc | -- | February 11, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.unknown; | -Untested on PAE | -
| if_mn | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys(). Please contact &a.phk; for info/hardware. | -
| if_mxge | -&a.gallatin; | -August 23, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_my | -- | August 17, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_nfe | -- | August 23, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_nge | -- | August 17, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_nve | -- | November 23, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_pcn | -&a.obrien; | -August 19, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_pdq | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -Mostly busdma, except for vtophys(). | -
| if_ral | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_ray | -- | August 27, 2006 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -- |
| if_re | -- | May 30, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_rl | -&a.wpaul; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_rue | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_sf | -- | August 19, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_sis | -&a.wpaul; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_sk | -- | August 24, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_sn | -&a.imp; | -December 23, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_snc | -- | December 23, 2005 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -pc98 only device (although it could work with many - cardbus bridges) | -
| if_sr | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| if_ste | -- | August 31, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -vtophys() | -
| if_stge | -&a.yongari; | -August 23, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_ti | -- | December 13, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_tl | -- | September 15, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_tx | -&a.mux; | -April 19, 2003 | -&status.done; | -&status.new; | -&status.new; | -&status.untested; | -- |
| if_txp | -- | September 22, 2005 | -&status.wip; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_udav | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_ural | -- | September 3, 2006 | -&status.na; | -&status.new; | -&status.unknown; | -&status.unknown; | -- |
| if_vge | -- | August 23, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_vr | -- | April 23, 2004 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_vx | -- | September 22, 2005 | -&status.na; | -&status.done; | -&status.done; | -&status.new; | -Uses PIO to copy mbufs to and from hardware. | -
| if_wb | -- | September 22, 2005 | -&status.new; | -&status.done; | -&status.done; | -&status.new; | -- |
| if_wi | -&a.sam;, &a.imp; | -November 4, 2003 | -&status.unknown; | -&status.done; | -&status.unknown; | -&status.unknown; | -This driver needs lots of help | -
| if_xe | -&a.imp; | -December 23, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| if_xl | -&a.mux; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| Driver | -Responsible | -Last updated | -busdma | -INTR_MPSAFE | -SMPng locked | -a!=p | -Notes | -
|---|---|---|---|---|---|---|---|
| aac | -- | January 31, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -Not endian clean. | -
| adv | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| aha | -- | April 13, 2004 | -&status.done; | -&status.wip; | -&status.wip; | -&status.new; | -Uses BUSDMA, but may pun bus address with host address. | -
| ahb | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| ahc | -&a.gibbs; | -January 31, 2005 | -&status.done; | -&status.new; | -&status.new; | -&status.done; | -- |
| ahd | -&a.gibbs; | -January 31, 2005 | -&status.done; | -&status.new; | -&status.new; | -&status.done; | -- |
| aic | -- | December 23, 2005 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -Neeeds evaluation | -
| amd | -- | December 14, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| amr | -- | January 30, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| asr | -- | January 4, 2003 | -&status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys(). Requires major work. A new I2O framework would be desirable. | -
| ata | -&a.sos; | -December 9, 2002 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| buslogic | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| ciss | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| ct | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -- |
| dpt | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| fdc | -- | December 23, 2005 | -&status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -Needs evaluation | -
| ida | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| iir | -- | March 1, 2006 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -64-bit DMA without bouncing is possible, but needs work. | -
| isp | -- | February 8, 2003 | -&status.done; | -&status.done; | -&status.new; | -&status.new; | -- |
| ips | -- | January 30, 2005 | -&status.done; | -&status.done; | -&status.done; | -&status.done; | -- |
| mlx | -- | February 8, 2003 | -&status.done; | -&status.wip; | -&status.wip; | -&status.new; | -- |
| mly | -- | February 8, 2003 | -&status.done; | -&status.wip; | -&status.wip; | -&status.new; | -- |
| mpt | -- | December 9, 2002 | -&status.done; | -&status.done; | -&status.new; | -&status.new; | -- |
| ncr | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys(). Please contact &a.phk; for a possible source - of hardware. | -
| ncv | -- | - | &status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -Needs evaluation | -
| nsp | -- | - | &status.unknown; | -&status.unknown; | -&status.unknown; | -&status.unknown; | -Needs evaluation | -
| pst | -- | April 11, 2003 | -&status.new; | -&status.done; | -&status.new; | -&status.new; | -vtophys() | -
| stg | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -At least, it looks like it may well be. | -
| sym | -- | December 19, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| trm | -&a.cognet; | -December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| twe | -- | December 9, 2002 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| wds | -- | February 2, 2005 | -&status.done; | -&status.new; | -&status.new; | -&status.new; | -- |
| Driver | -Responsible | -Last updated | -busdma | -INTR_MPSAFE | -SMPng locked | -a!=p | -Notes | -
|---|---|---|---|---|---|---|---|
| agp | -- | October 31, 2005 | -&status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| bktr | -&a.cognet; | -January 15, 2003 | -&status.wip; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| digi | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| drm | -&a.anholt; | -October 27, 2003 | -&status.wip; | -&status.done; | -&status.done; | -&status.wip; | -vtophys(). The locking could use some review. | -
| fb | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| firewire | -&a.simokawa; | -April 17, 2003 | -&status.done; | -&status.new; | -&status.new; | -&status.done; | -vtophys() | -
| hfa | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys() | -
| hifn | -&a.sam; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -- |
| musycc | -- | - | &status.new; | -&status.new; | -&status.new; | -&status.new; | -vtophys(). Please contact &a.phk; for info/hardware. | -
| pcm | -&a.cognet; | -February 20, 2003 | -&status.done; | -&status.done; | -&status.new; | -&status.new; | -- |
| ubsec | -&a.sam; | -April 13, 2004 | -&status.done; | -&status.done; | -&status.done; | -&status.new; | -vtophys() is used in debugging printf. | -
| usb | -&a.jmg; | -July 24, 2003 | -&status.done; | -&status.new; | -&status.new; | -&status.untested; | -a!=p should be clean, but requires further testing. | -
| Task | -Responsible | -Last updated | -Status | -Notes | -
|---|---|---|---|---|
| Manual pages for the busdma API | -&a.hmp; | -January 15, 2003 | -&status.done; | -- |
The ideas page has moved to the FreeBSD wiki.
- - - Index: head/en_US.ISO8859-1/htdocs/projects/ideas/style.css =================================================================== --- head/en_US.ISO8859-1/htdocs/projects/ideas/style.css +++ head/en_US.ISO8859-1/htdocs/projects/ideas/style.css @@ -1,38 +0,0 @@ -BODY { -} - -BODY TD { - font-size: 13px; -} - -BODY SMALL { - width: 615px; - font-size: 11px; -} - -.heading { - font-size: 15px; - background-color: #cbd2ec; -} - -.section { - font-size: 15px; - font-weight: bold; - background-color: #e7e9f7; -} - -.notes { - font-size: 13px; - font-weight: normal; -} - -.main { - width: 615px; - height: auto; - text-align: justify; -} - -.list { - width: 550px; - height: auto; -} Index: head/en_US.ISO8859-1/htdocs/projects/netperf/Makefile =================================================================== --- head/en_US.ISO8859-1/htdocs/projects/netperf/Makefile +++ head/en_US.ISO8859-1/htdocs/projects/netperf/Makefile @@ -1,18 +0,0 @@ -# Summary for busdma project status -# -# $FreeBSD$ - -MAINTAINER= rwatson - -.if exists(../Makefile.conf) -.include "../Makefile.conf" -.endif -.if exists(../Makefile.inc) -.include "../Makefile.inc" -.endif - -DOCS= index.xml -DOCS+= cluster.xml -DATA= style.css - -.include "${DOC_PREFIX}/share/mk/web.site.mk" Index: head/en_US.ISO8859-1/htdocs/projects/netperf/cluster.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/projects/netperf/cluster.xml +++ head/en_US.ISO8859-1/htdocs/projects/netperf/cluster.xml @@ -1,141 +0,0 @@ - - - - - -]> - - - -The netperf cluster provides a multi-node, SMP-capable, network - functionality and performance test capability for the FreeBSD Project, supporting a variety of on-going - sub-projects including the netperf project, - and on-going work on - high performance threading. The cluster is available on a check out - basis for developers, who must request accounts be created by - contacting one of the netperf cluster admins. - The cluster includes both 1gbps and 10gbps test segments, with - network hardware from a number of vendors.
- - -The netperf cluster was made possible through the generous donation - of a number of organizations, including:
- -Sentex Data Communications, - who not only host the complete cluster, provide front-end build - system, several test systems, and the management infrastructure - (remote power, serial console, network switch, etc), but also appear - to be endlessly willing to help configure, reconfigure, and - troubleshoot at almost any time of day or night.
FreeBSD Systems, - who through a generous matching grant with the FreeBSD Foundation, - provide the majority of testing hardware used in the cluster, - including three dual-Xeon test systems.
The FreeBSD - Foundation, who provided a matching grant for the purposes of - purchasing testing hardware, as well as taking ownership of hardware, - offering tax receipts to donors in its role as a non-profit, and - participating in cluster planning.
IronPort Systems, who have - donated a test server.
iXsystems, who have - donated several test servers.
Google, Inc., who have - donated two test servers.
Cisco, Inc., who have - donated a 10gbps switch.
Chelsio Communications, - who have donated two 10gbps network cards.
Myricom, Inc., who have - donated two 10gbps network cards.
Intel Corporation, who - have donated two 10gbps network cards.
Meinberg, who - have donated a LANTIME M600 IEEE 1588 Grandmaster Clock.
&a.gnn;, who has donated a quad-core AMD test - system.
&a.rwatson;, who has donated a dual-CPU PIII system and a - Portmaster terminal server.
Donations to support the netperf cluster have an immediate and - substantial impact on the success of a number of on-going performance - projects, providing access to high-end hardware to a large number of - developers. If you or your company are interested in helping to - support continued development of the netperf cluster as a resource for - FreeBSD development, please contact the netperf - cluster admins.
- - -The FreeBSD netperf cluster is managed by a small team of - developer/administrators to support SMP development and performance - testing on high-end hardware. If you have any questions, including - questions about access to the cluster as a developer, or about possible - future donations of testing hardware, please feel free to contact them - via netperf-admin at FreeBSD.org.
- - -The Netperf cluster consists of several systems interconnected using a - management network, as well as individual back-to-back gigabit ethernet - links for a test network. Please refer to the Test - Cluster One Reservations page on the &os; wiki for the - current allocations.
- - -As the netperf cluster is a centrally managed and shared resource, - understanding and consistent following of its procedures is important. - In particular, following of the procedures makes it easier for - developers to have reasonable expectations about the configuration of - systems in the cluster, as well as to avoid treading on each others - toes.
- -Pointers and documentation on Netperf Cluster procedures can be found - on the Test - Cluster One Pointers page on the FreeBSD wiki.
- - Index: head/en_US.ISO8859-1/htdocs/projects/netperf/index.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/projects/netperf/index.xml +++ head/en_US.ISO8859-1/htdocs/projects/netperf/index.xml @@ -1,386 +0,0 @@ - - - - -N/A"> -Done"> -Prototyped"> -Merged to HEAD; RELENG_5 candidate"> -Not done"> -Unknown"> -]> - - - -The netperf project is working to enhance the performance of the - FreeBSD network stack. This work grew out of the - SMPng Project, which moved the FreeBSD kernel from - a "Giant Lock" to more fine-grained locking and multi-threading. SMPng - offered both performance improvement and degradation for the network - stack, improving parallelism and preemption, but substantially - increasing per-packet processing costs. The netperf project is - primarily focussed on further improving parallelism in network - processing while reducing the SMP synchronization overhead. This in - turn will lead to higher processing throughput and lower processing - latency.
- - -Robert Watson
- -The two primary focuses of this work are to increase parallelism - while decreasing overhead. Several activities are being performed that - will work toward these goals:
- -The Netperf project has completed locking work for all components - of the network stack; as of FreeBSD 7.0 we have removed non-MPSAFE - protocol shims, and as of FreeBSD 8.0 we have removed non-MPSAFE - device driver shims.
Optimize locking strategies to find better balances between - locking granularity and locking overhead. In the first cut at locking - for the kernel, the goal was to adopt a medium-grained locking - approach based on data locking. This approach identifies critical - data structures, and inserts new locks and locking operations to - protect those data structures. Depending on the data model of the - code being protected, this may lead to the introduction of a - substantial number of locks offering unnecessary granularity, where - the overhead of locking overwhelms the benefits of available - parallelism and preemption. By selectively reducing granularity, it - is possible to improve performance by decreasing locking overhead. -
Amortize the cost of locking by processing queues of packets or - events. While the cost of individual synchronization operations may - be high, it is possible to amortize the cost of synchronization - operations by grouping processing of similar data (packets, events) - under the same protection. This approach focuses on identifying - places where similar locking occurs frequently in succession, and - introducing queueing or coalescing of lock operations across the - body of the work. For example, when a series of packets is inserted - into an outgoing interface queue, a basic locking approach would - lock the queue for each insert operation, unlock it, and hand off to - the interface driver to begin the send, repeating this sequence as - required. With a coalesced approach, the caller would pass off a - queue of packets in order to reduce the locking overhead, as well as - eliminate unnecessary synchronization due to the queue being - thread-local. This approach can be applied at several levels in the - stack, and is particularly applicable at lower levels of the stack - where streams of packets require almost identical processing. -
Introduce new synchronization strategies with reduced overhead - relative to traditional strategies. Most traditional strategies - employ a combination of interrupt disabling and atomic operations to - achieve mutual exclusion and non-preemption guarantees. However, - these operations are expensive on modern CPUs, leading to the desire - for cheaper primitives with weaker semantics. For example, the - application of uni-processor primitives where synchronization is - required only on a single processor, and optimizations to critical - section primitives to avoid the need for interrupt disabling. -
Modify synchronization strategies to take advantage of - additional, non-locking, synchronization primitives. This approach - might take the form of making increased use of per-CPU or per-thread - data structures, which require little or no synchronization. For - example, through the use of critical sections, it is possible to - synchronize access to per-CPU caches and queues. Through the use of - per-thread queues, data can be handed off between stack layers - without the use of synchronization.
Increase the opportunities for parallelism through increased - threading in the network stack. The current network stack model - offers the opportunity for substantial parallelism, with outbound - processing typically taking place in the context of the sending - thread in kernel, crypto occurring in crypto worker threads, and - receive processing taking place in a combination of the receiving - ithread and dispatched netisr thread. While handoffs between - threads introduces overhead (synchronization, context switching), - there is the opportunity to increase parallelism in some workloads - through introducing additional worker threads. Identifying work - that may be relocated to new threads must be done carefully to - balance overhead, and latency concerns, but can pay off by - increasing effective CPU utilization and hence throughput. For - example, introducing additional netisr threads capable of running on - more than one CPU at a time can increase input parallelism, subject - to maintaining desirable packet ordering (present in FreeBSD - 8.0).
| Task | -Responsible | -Last updated | -Status | -Notes | -
|---|---|---|---|---|
| Prefer file descriptor reference counts to socket reference - counts for system calls. | -&a.rwatson; | -20041124 | -&status.done; | -Sockets and file descriptors both have reference counts in order - to prevent these objects from being free'd while in use. However, - if a file descriptor is used to reach the socket, the reference - counts are somewhat interchangeable, as either will prevent - undesired garbage collection. For socket system calls, overhead - can be reduced by relying on the file descriptor reference count, - thus avoiding the synchronized operations necessary to modify the - socket reference count, an approach also taken in the VFS code. - This change has been made for most socket system calls, and has - been committed to HEAD (6.x). It has also been merged to RELENG_5 - for inclusion in 5.4. | -
| Mbuf queue library | -&a.rwatson; | -20041124 | -&status.prototyped; | -In order to facilitate passing off queues of packets between - network stack components, create an mbuf queue primitive, struct - mbufqueue. The initial implementation is complete, and the - primitive is now being applied in several sample cases to determine - whether it offers the desired semantics and benefits. The - implementation can be found in the rwatson_dispatch Perforce - branch. Additional work must also be done to explore the - performance impact of "queues" vs arrays of mbuf pointers, which - are likely to behave better from a caching perspective. | -
| Employ queued dispatch in interface send API | -&a.rwatson; | -20041106 | -&status.prototyped; | -An experimental if_start_mbufqueue() interface to struct ifnet - has been added, which passes an mbuf queue to the device driver for - processing, avoiding redundant synchronization against the - interface queue, even in the event that additional queueing is - required. This has not yet been benchmarked. A subset change to - dispatch a single mbuf to a driver has also been prototyped, and - benchmarked at a several percentage point improvement in packet send - rates from user space. | -
| Employ queued dispatch in the interface receive API | -&a.rwatson; | -20041106 | -&status.new; | -Similar to if_start_mbufqueue, allow input of a queue of mbufs - from the device driver into the lowest protocol layers, such as - ether_input_mbufqueue. | -
| Employ queued dispatch across netisr dispatch API | -&a.rwatson; | -20090601 | -&status.done; | -Pull all of the mbufs in the netisr queue into a thread-local - mbuf queue to avoid repeated lock operations to access the queue. - This work was completed as part of the netisr2 project, and will - ship with 8.0-RELEASE. | -
| Modify UMA allocator to use critical sections not mutexes for - per-CPU caches. | -&a.rwatson; | -20050429 | -&status.done; | -The mutexes protecting per-CPU caches require atomic operations - on SMP systems; as they are per-CPU objects, the cost of - synchronizing access to the caches can be reduced by combining - CPU pinning and/or critical sections instead. This change has now - been committed and will appear in 6.0-RELEASE; it results in a - several percentage performance in UDP send from user space, and - there have been reports of 20%+ improvements in allocation - intensive code within the kernel. In micro-benchmarks, the cost - of allocation on SMP is dramatically reduced. | -
| Modify malloc(9) allocator to use per-CPU statistics with - critical sections to protect malloc_type statistics rather than - global statistics with a mutex. | -&a.rwatson; | -20050529 | -&status.done; | -Previously, malloc(9) used a single statistics structure - protected by a mutex to hold global malloc statistics for each - malloc type. This change moves to per-CPU statistics structures, - which are coalesced when reporting memory allocation statistics to - the user, and protects them using critical sections. This reduces - cache line contention for common allocation types by avoiding - shared lines, and also reduces synchronization costs by using - critical sections to synchronize access instead of a mutex. While - malloc(9) is less frequently used in the network stack than uma(9), - it is used for socket address data, so is on performance critical - paths for datagram operations. This has been committed and appeared - 6.0-RELEASE. | -
| Optimize critical section performance | -&a.jhb; | -20050404 | -&status.done; | -Critical sections prevent preemption of a thread on a CPU, as - well as preventing migration of that thread to another CPU, and - maybe used for synchronizing access to per-CPU data structures, as - well as preventing recursion in interrupt processing. Currently, - critical sections disable interrupts on the CPU. In previous - versions of FreeBSD (4.x and before), optimizations were present - that allowed for software interrupt disabling, which lowers the - cost of critical sections in the common case by avoiding expensive - microcode operations on the CPU. By restoring this model, or a - variation on it, critical sections can be made substantially - cheaper to enter. In particular, this change lowers the cost - of critical sections on UP such that it is approximately the same - cost as a mutex, meaning that optimizations on SMP to use critical - sections instead of mutexes will not harm UP performance. This - change has now been committed, and appeared in 6.0-RELEASE. | -
| Normalize socket and protocol control block reference model | -&a.rwatson; | -20060401 | -&status.done; | -The socket/protocol boundary is characterized by a set of data - structures and API interfaces, where the socket code acts as both - a consumer and a service library for protocols. This task is to - normalize the reference model by which protocol state is attached - to and detached from socket state in order to strengthen - invariants, allowing the removal of countless unused code paths - (especially error handling), the removal of unnecessary locking - in TCP, and a general improve the structure of the code. This - serves both the immediate purpose of improving the quality and - performance of this code, as well as being necessary for future - optimization work. These changes have been prototyped in - Perforce, and now merged to 7-CURRENT. They will be merged into - RELENG_6 once they have been thoroughly tested. | -
| Add true inpcb reference count support | -&a.mohans;, &a.rwatson;, &a.peter; | -20081208 | -&status.done; | -Historically, the in-bound TCP and UDP socket paths relied on - global pcbinfo info locks to prevent PCBs being delivered to from - being garbage collected by another thread while in use. This set - of changes introduces a true reference model for PCBs so that the - global lock can be released during in-bound process, and appear - in 8.0-RELEASE. | -
| Fine-grained locking for UNIX domain sockets | -&a.rwatson; | -20070226 | -&status.done; | -UNIX domain sockets in FreeBSD 5.x and 6.x use a single global - subsystem lock. This is sufficient to allow it to run without - Giant, but results in contention with large numbers of processors - simultaneously operating on UNIX domain sockets. This task - introduced per-protocol control block locks in order to reduce - contention on a larger subsystem lock, and the results appeared in - 7.0-RELEASE. | -
| Multiple netisr threads | -&a.rwatson; | -20090601 | -&status.done; | -Historically, the BSD network stack has used a single network - software interrupt context, for deferred network processing. With - the introduction of multi-processing, this became a single - software interrupt thread. In FreeBSD 8.0, multiple netisr - threads are now supported, up to the number of CPUs present in the - system. | -
Through the generous donations and investment of Sentex Data - Communications, FreeBSD Systems, IronPort Systems, and the FreeBSD - Foundation, a network performance testbed has been created in Ontario, - Canada for use by FreeBSD developers working in the area of network - performance. A similar cluster, made possible through the generous - donation of Verio, is being prepared for use in more general SMP - performance work in Virginia, US. Each cluster consists of several SMP - systems inter-connected with giga-bit ethernet such that relatively - arbitrary topologies can be constructed in order to test host-host, IP - forwarding, and bridging performance scenarios. Systems are network - booted, have serial console, and remote power, in order to maximize - availability and minimize configuration overhead. These systems are - available on a check-out basis for experimentation and performance - measurement to FreeBSD developers working on the Netperf project, and - in related areas.
- -More detailed information on the netperf - cluster can be found by following this link.
- - -The following paper(s) have been produced by or are related to the - Netperf Project:
- - - - -Some useful links relating to the netperf work:
- -SMPng Project -- Project to introduce - finer grained locking in the FreeBSD kernel.
Robert - Watson's netperf web page -- Web page that includes a change log - and performance measurement/debugging information.