diff --git a/en/smp/index.sgml b/en/smp/index.sgml index dda084e05c..d0f2a6d7a1 100644 --- a/en/smp/index.sgml +++ b/en/smp/index.sgml @@ -1,2768 +1,2768 @@ - + %navincludes; %includes; Done"> In progress"> Stalled"> Not Started"> Resolved"> Unresolved"> %developers; ]> &header;

Contents

Project Goal

The FreeBSD Symmetric MultiProcessing (SMP) project, often referred to as SMPng (SMP next generation), is focused on introducing parallelism into the FreeBSD kernel. While earlier versions of FreeBSD (3.x, 4.x) supported parallelism in user processes, the kernel was limited to executing on a single processor at a time, using what is referred to as a "Giant lock" around the kernel. For many interesting workloads, this results in a substantial speed-up, as significant computation occurs in user processes, especially for applications such as rendering and compilation. However, for kernel-intensive applications, such as intensive network or file system I/O, contention on the kernel lock results in little speed-up. The end goal of the SMPng Project is to decompose the Giant lock into a number of smaller locks, resulting in reduced contention (and improved SMP performance). However, important steps along the way include redesigning significant portions of the FreeBSD kernel architecture around the notion of ubiquitous parallelism: that at any moment, many processors might enter the kernel at the same time. This includes the introduction of more mature threading and synchronization primitives, interrupt threads, cache-aware allocation and scheduling, and topology-aware scheduling.

SMPng debuted in FreeBSD 5.0-RELEASE in January of 2003, and involved over five years of continuous development by a large number of members of the FreeBSD development team, as well as many external contributors. Since 5.0, the implementation has substantially matured; in the 5.0 release, the basic architectural changes required to support SMPng were complete, including new kernel memory allocators, synchronization routines, the move to ithreads, and the removal of the Giant lock from activities such as process scheduling and several common forms of IPC. Since 5.0, the architecture has been refined in a number of ways, including optimizing synchronization approaches, stability and performance testing on larger systems (up to 12 processors), and the removal of Giant from several significant parts of the operating system, including large parts of Virtual Memory (VM), the Virtual File System (VFS), UNIX File System (UFS), most parts of the network stack, including IPv4, IPV6, FAST_IPSEC, UNIX domain sockets, and NetGraph, and also from additional inter-process communication primitives. The SMP-aware kernel slab allocator is now used almost universally, and the focus has changed from "make it work", to "optimize it!". Simultaneous work on KSE 1:1 and M:N threading has also allowed applications to take advantage of new kernel parallelism. The FreeBSD 5.3 kernel introduced Giant-free network stack execution for most relevant code paths, and the FreeBSD 6.x kernel introduces MPSAFE VFS, as well as wide-spread performance optimization.

Continuing work on SMPng includes work sweeping up the "loose ends" that remain under Giant, such as parts of NFS, less widely used file systems such as NTFS, and less commonly used network stack components, such as SLIP. Another important focus is performance measurement and optimization, which build on and refine the SMPng architecture: the introduction of features such as the kernel trace facility (KTR), hardware performance monitor counters (hwpmc), lock profiling, and improved memory monitoring play important roles in this process. Other important debugging and testing facilities include WITNESS, a run-time kernel lock order verifier, and wide-spread use of lock assertions and run-time invariants testing.

The FreeBSD Project recognizes, in particular, the contributions of the BSD/OS development team, including architectural direction and suggestions, as well as for initial locking strategies for some system components, and source code for some of the starting primitives, not to mention BSDI's contributions in staffing and resources for early parts of this project.

Project Plan

This web page contains information relating to the SMPng effort; because of the large amount of work and rapid pace of development, it can fall a bit behind reality.

The task list below is not intended to be complete, but does represent a set of relevant and/or important components of the overall work. The "Responsible" field identifies a developer who has expressed willingness to be responsible for completing the identified task; this doesn't preclude others working on it, but suggests that coordination with the responsible party might be appropriate so as to avoid unnecessary duplication of work, and to maximize forward progress. If beginning work on a new area of substantial size, or one that appears unclaimed, it may be worth dropping an e-mail to the FreeBSD SMP mailing list to see if any progress has been made.

The definition of the date field varies depending on the status of a task. For completed tasks, it refers to the date completed or reported completed. For in-progress tasks, it refers to the date of the last update of the entry. For stalled tasks, it refers to the date that the task was declared stalled. For new tasks, it refers to the date the task was added to the list.

Locking down of individual device drivers is tracked at the busdma and SMPng driver conversion webpage.

Network stack locking information is available at Robert Watson's netperf web page and the FreeBSD.org Netperf web page. An SMP network performance cluster has also been created for the purposes of testing.

Tasks are sorted first by status, then by date.

Resources and Links

Subsystems and Staffing

This is an incomplete list of high-level kernel subsystems and current, active staff working on SMP architecture and stability.

Subsystem Status Last updated Staffing
Newbus &status.wip; 5 October 2003 &a.imp;
VM &status.wip; 4 October 2003 &a.alc;
Buffer cache &status.wip; 4 October 2003 &a.jeff;, &a.phk;
VFS &status.wip; 4 October 2003 &a.jeff;
Processes and thread operations &status.wip; 5 May 2003 &a.jhb;
Scheduler &status.done; 23 April 2004 &a.jhb;, &a.jeff;
GEOM &status.done; 5 February 2003 &a.phk;
File descriptors &status.done; 5 February 2003 &a.alfred;, &a.tanimura; &a.rwatson;
TTY subsystem &status.wip; 24 July 2004 &a.phk;
Pipe IPC &status.done; 4 October 2003 &a.alfred;
Socket structures and system calls &status.done; 25 November 2004 &a.sam;, &a.rwatson;
KQueue &status.done; 24 November 2004 &a.jmg;, &a.green;
IPv4 &status.done; 23 April 2004 &a.jennifer;, &a.hsu;, &a.sam;, &a.rwatson;, &a.gnn;
IPv6 &status.wip; 01 July 2005 &a.rwatson;, &a.gnn;
IPX/SPX &status.done; 09 January 2005 &a.rwatson;
netatalk &status.done; 02 February 2005 &a.rwatson;
Network stack infrastructure &status.wip; 4 October 2003 &a.hsu;, &a.sam;, &a.rwatson;, &a.mlaier;, &a.luigi;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org>, &a.brooks;, &a.rik;
NFS Client &status.wip; 23 April 2004  
NFS Server &status.wip; 24 November 2004 &a.rwatson;

Tasks

Following is an incomplete list of general tasks.

- +
Task Responsible Last updated Status
Convert the giant lock from spinning to blocking, add the scheduler lock, add per-CPU idle processes. &a.dillon; 25 June 2000 &status.done;
Port the BSD/OS locking primitives (i386). &a.jake; 3 July 2000 &status.done;
Implement heavy-weight interrupt threads (i386). &a.grog; 3 August 2000 &status.done;
Rewrite the low level interrupt code (i386 UP). &a.grog; 3 August 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (i386 UP). -smp developers 12 August 2000 &status.done;
Port the BSD/OS locking primitives (alpha). &a.dfr; 24 August 2000 &status.done;
Stub out (disable) spl()s. &a.grog; 30 August 2000 &status.done;
Port the BSD/OS ktr code. &a.grog;, &a.jhb; 30 August 2000 &status.done;
Rewrite the low level interrupt code (i386 SMP). &a.jhb; 1 September 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (i386 SMP). -smp developers 6 September 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (alpha). -smp developers 6 September 2000 &status.done;
Make malloc and friends thread-safe. &a.jasone; 10 September 2000 &status.done;
Implement msleep(), make tsleep() an msleep() wrapper. &a.jake; 11 September 2000 &status.done;
Make fxp driver thread-safe. &a.cp; 17 September 2000 &status.done;
Make mbuf's thread-safe. &a.bmilekic; 29 September 2000 &status.done;
Lock manager re-work. &a.jasone; 3 October 2000 &status.done;
Implement heavy-weight interrupt threads (alpha). &a.jhb;, &a.dfr; 5 October 2000 &status.done;
Rewrite the low level interrupt code (alpha). &a.dfr;, &a.jhb; 5 October 2000 &status.done;
Process accounting. &a.tegge;, &a.jhb; 5 October 2000 &status.done;
Make ethernet drivers thread-safe. &a.wpaul; 15 October 2000 &status.done;
Make the mutex headers mostly machine-independent. &a.jhb; 20 October 2000 &status.done;
Rename SMP_DEBUG to MUTEX_DEBUG. &a.jhb; 20 October 2000 &status.done;
Give each soft interrupt its own thread. &a.cp; 25 October 2000 &status.done;
Make sf_bufs (sendfile(2)) thread-safe. &a.bmilekic; 5 November 2000 &status.done;
Make the witness code work correctly. &a.jhb; 18 November 2000 &status.done;
Split the ktr-specific code out of db_interface.c. &a.jhb; 15 December 2000 &status.done;
Convert the sio driver to using a spin mutex. &a.jhb; 18 December 2000 &status.done;
Implement condition variables. &a.jake;, &a.jasone; 15 January 2001 &status.done;
Add a flag to mtx_init() (MTX_RECURSE) that denotes whether a mutex is allowed to recurse. &a.bmilekic; 19 January 2001 &status.done;
Make the zone allocator thread-safe. &a.des; 21 January 2001 &status.done;
Convert simplelocks to mutexes. &a.jasone; 24 January 2001 &status.done;
Make kernel preemptive with respect to interrupts. &a.jake; 31 January 2001 &status.done;
Cleanup of mutex API. &a.bmilekic; 8 February 2001 &status.done;
Remove COM_LOCK. &a.markm; 11 February 2001 &status.done;
Merge various scheduling classes into one run queue. Modify scheduler to support preemptable kernel. &a.jake; 11 February 2001 &status.done;
Make priority propagation work correctly. &a.jake; 11 February 2001 &status.done;
Make most of the interrupt thread code MI and shared between hardware and software interrupts. &a.jhb; 18 February 2001 &status.done;
Add protection to struct jail and jail-related functionality. &a.rwatson; 20 February 2001 &status.done;
Implement sx (shared/exclusive) locks. &a.jasone; 5 March 2001 &status.done;
Generalize/improve witness to handle more complex locking primitives (mtx, sx). &a.jhb; 28 March 2001 &status.done;
Convert the allproc and proctree locks from lockmgr locks to sx locks. &a.jhb; 28 March 2001 &status.done;
Make mbuf system use condition variables instead of msleep()/wakeup(). &a.bmilekic; 2 April 2001 &status.done;
Remove <sys/mutex.h> includes from other kernel headers such as <vm/vm_zone.h>, <sys/resourcevar.h>, <sys/ucred.h>, and <sys/mbuf.h>. &a.markm; 15 May 2001 &status.done;
Cleanup the various mp_machdep.c's, unify various SMP API's such as IPI delivery, etc. &a.jhb; 15 May 2001 &status.done;
Make most of the forward_* and forwarded_* functions MI. &a.jhb; 15 May 2001 &status.done;
Complete the MD support for SMP on the Alpha platform. &a.gallatin;, &a.dfr;, &a.jhb; 15 May 2001 &status.done;
Convert select() to use condition variables. &a.tanimura; 15 May 2001 &status.done;
Add a "giant" lock around the VM subsystem. &a.alfred; 13 June 2001 &status.done;
Introduce a modified slab allocator for the mbuf subsystem. &a.bmilekic; 21 June 2001 &status.done;
Add a witness_assert() function to handle lock assertions. &a.jhb; 27 June 2001 &status.done;
Extend sx locks to support try lock operations. &a.jhb; 27 June 2001 &status.done;
Document KTR. &a.jhb; 28 June 2001 &status.done;
Make fork_return, fork_exit, ast, and userret MI. &a.jhb; 29 June 2001 &status.done;
Make sched_lock's savecrit a per-process property saved and restored in mi_switch and initialized in fork_exit. &a.jhb; 30 June 2001 &status.done;
Make ast() loop. &a.jhb; 10 August 2001 &status.done;
Add upgrade/downgrade sx lock operations. Alexander Kabaev, &a.jasone; &a.kan;, &a.jasone; 13 August 2001 &status.done;
Implement semaphores. &a.jasone; 14 August 2001 &status.done;
Add support for upgrade/downgrades in witness. &a.jhb; 23 August 2001 &status.done;
Make most of cpu_wait() and cpu_exit() MI. &a.peter; 9 September 2001 &status.done;
Split NFS into client and server. &a.peter; 18 Oct 2001 &status.done;
Lock taskqueues. &a.arr;, &a.jhb; 25 October 2001 &status.done;
Add a per-thread ucred reference. &a.jhb; 25 October 2001 &status.done;
Make most of the per-CPU stuff MI. &a.jhb; 11 December 2001 &status.done;
Make critical section saved state per-thread instead of per-lock so that interlocking spin locks work properly. &a.jhb; 17 December 2001 &status.done;
Replace the APIC-specific imen_mtx with a MI-named icu_lock to protect interrupt controllers and associated data within the kernel for both i386 and alpha. &a.jhb; 20 December 2001 &status.done;
Use the per-thread critical section nesting level in the mutex and interrupt thread code to automatically determine when to not preempt. This makes the MTX_NOSWITCH, SWI_SWITCH, and SWI_NOSWITCH flags obsolete as the kernel will be able to figure out the proper behavior on its own. &a.jhb; 5 January 2002 &status.done;
Lock struct filedesc and struct file. &a.tanimura;, &a.alfred; 12 January 2002 &status.done;
Lock struct pgrp, struct session, and struct sigio. &a.tanimura; 23 February 2002 &status.done;
Lock pipe implementation, but not sigio/fown, VM interactions. &a.alfred; 27 February 2002 &status.done;
Move to explicit reference counting for soft vnode references. &a.phk; 8 March 2002 &status.done;
Initialize mutex pools early enough that sx locks can be used for VM. &a.green; 14 March 2002 &status.done;
Place a global lock (sellock) around selinfo structures to fix a variety of lock order reversals, and make select() MP-safe. &a.alfred;, &a.davidc; 14 March 2002 &status.done;
Push down Giant on read, write, pread, pwrite system calls, acquiring Giant in the per-subsystem fileop layer for sockets, VFS, etc. &a.alfred; 15 March 2002 &status.done;
Lock down kernel module structures. &a.arr; 18 March 2002 &status.done;
Lock down kernel linker globals. &a.arr; 18 March 2002 &status.done;
Rewrite kernel memory allocator to be a slab allocator that uses per-cpu caches. &a.jeff; 21 March 2002 &status.done;
Replace incorrect use of MD critical section API to disable interrupts with a specific interrupt disable API. &a.imp;, &a.dfr;, &a.benno;, &a.jhb; 21 March 2002 &status.done;
Lock down access to the shared p_args "process arguments" structure through appropriate protection of that structure and references to it. &a.mini; 31 March 2002 &status.done;
Move from flags/tsleep lock to sx locks to protect sysctl tree from updates during sysctl operations. &a.mini; 1 April 2002 &status.done;
Create/port userland tool to manage KTR event dumps. &a.jake; 1 April 2002 &status.done;
Create MTX_SYSINIT and SX_SYSINIT macros that allow for initializing locks that are subsystem independent. &a.arr; 2 April 2002 &status.done;
Lock down the global securelevel variable. &a.arr; 2 April 2002 &status.done;
Make grow_stack() MI. Possibly even a macro or inline. &a.alc; 6 April 2002 &status.done;
Lock use of p_fd, which otherwise can result in corrupted p_fd panics during heavy operation. Start with a global, and move to per-proc locking. &a.alfred;, &a.tanimura; 8 April 2002 &status.done;
Lock struct pargs. &a.mini; 9 April 2002 &status.done;
Lock sysctl hierarchy. &a.mini; 9 April 2002 &status.done;
Make {o,}sigreturn() MPSAFE. &a.alc; 11 April 2002 &status.done;
Rewrite kernel memory allocator so that Giant is not required for malloc() or free(). &a.jeff; 2 May 2002 &status.done;
Replace complex shared/exclusive locking scheme in the VM system with a purely exclusive lockmgr locking scheme, simplifying locking and removing potential livelock/deadlock scenarios. &a.green;, &a.alc; 3 May 2002 &status.done;
Push down Giant into readv/writev system calls in style of read/write/pread/pwrite once malloc no longer requires Giant in the handling of iovec structures for uio. &a.alc; 9 May 2002 &status.done;
Push down Giant in mprotect(), minherit(), and madvise() so that it is no longer acquired and released directly. &a.alc; 18 May 2002 &status.done;
Update suser() and p_can*() APIs to accept threads instead of processes. &a.jhb; 18 May 2002 &status.done;
Broadly transition to td_ucred from p_ucred once KSE dependencies are in place. &a.jhb; 18 May 2002 &status.done;
Add a witness_sleep() check to uma_zalloc() to catch code calling malloc() or uma_zalloc() while holding non-sleepable locks. &a.jhb; 20 May 2002 &status.done;
Optimize UP support by changing spin locks to only perform critical section enter and exits. &a.jhb; 21 May 2002 &status.done;
Make sleep mutexes spin if the current lock holder is executing on another CPU. &a.jhb; 21 May 2002 &status.done;
Add support for the IA32 pause instruction to spin loops in locks. &a.jhb; 21 May 2002 &status.done;
Make KTRACE write into tracefiles asynchronously. &a.jhb; 7 June 2002 &status.done;
Remove Giant from modnext(2), modfnext(2), modstat(2),and modfind(2). &a.arr; 25 June 2002 &status.done;
Fix synchronization of TLB flushes and invlpg() on x86 SMP. &a.peter; 12 July 2002 &status.done;
Add KTR(9) tracing for mutex contention. &a.iedowse; 26 Augist 2002 &status.done;
Make cpu_coredump MI. &a.peter; 7 September 2002 &status.done;
Add a subsystem lock to the accounting code. &a.arr; 11 September 2002 &status.done;
Allow KTR(9) to write trace records to alq(9) record facility. &a.jeff; 22 September 2002 &status.done;
Create mechanism in cdevsw structure to protect thread-unsafe drivers. &a.phk; 27 September 2002 &status.done;
Fix SIGXPU and other #if 0'd things in mi_switch(). &a.jhb; 30 September 2002 &status.done;
Lock down TrustedBSD MAC implementation. &a.rwatson; 11 November 2002 &status.done;
Lock eventhandlers. &a.msmith;, &a.mini;, &a.jhb; 11 March 2003 &status.done;
Fix PHOLD() so that it blocks to guarantee PS_INMEM. &a.jhb; 22 April 2003 &status.done;
Fix various procfs_machdep.c to not use sched_lock. &a.jhb; 22 April 2003 &status.done;
Lock all references to process credentials and remove Giant from process credential-related system calls. &a.jhb; 1 May 2003 &status.done;
Merge the procsig and sigacts structures, move the new sigacts structure out of the U-area and add appropriate locking. &a.jhb; 13 May 2003 &status.done;
Remove Giant from the kill() and killpg() system calls. &a.jhb; 13 May 2003 &status.done;
Enhance the mutex pool implementation to allow creation and use of multiple, dynamically allocated pools with adjustable pool sizes and mutex options. &a.truckman; 16 July 2003 &status.done;
Create mutex profiling tool for the kernel so as to measure contention and behavior of kernel mutexes. &a.eivind;, &a.des; 31 March 2002 &status.done;
Lock down linker_file_t structures in the kernel linker. &a.arr; 19 June 2002 &status.done;
Lock pipe implementation: VM optimizations.   4 October 2003 &status.done;
Reimplement i386 interrupt and SMP code so that SMP kernels work on UP boxes and SMP can be enabled in GENERIC. &a.jhb; 3 November 2003 &status.done;
Implement generic turnstiles to use when blocking on non-sleepable locks. &a.jhb; 11 November 2003 &status.done;
Split witness_lock() into witness_checkorder() and witness_lock(). witness_checkorder() would be called before acquiring a lock to increase the changes of detecting and warning about a reversal prior to deadlocking. witness_lock() would simply update witness' internal state to note that a lock has been acquired. &a.jhb; 24 January 2004 &status.done;
Lock per-process resource limits. &a.mtm;, &a.jhb; 4 February 2004 &status.done;
Implement a sleep queue abstraction to be used by both msleep() and condition variables. This new abstraction should use a hash table of sleep queues with a spin lock on each sleep queue chain similar to turnstile chain locks to make sched_lock finger grained. &a.jhb; 27 February 2004 &status.done;
Remove Giant from jail(2). &a.arr;, &a.rwatson; 23 April 2004 &status.done;
Add subsystem locking to NFSv2, NFSv3 server, permitting upcalls and other network-related elements to run Giant-free. &a.rwatson; 24 July 2004 &status.done;
Add KTR(9) tracing for UMA allocation/free events. &a.rwatson; 05 August 2004 &status.done;
Add KTR(9) tracing for GEOM I/O events. &a.rwatson; 21 October 2004 &status.done;
Add KTR(9) tracing for busdma events. &a.rwatson; 23 October 2004 &status.done;
Add KTR(9) tracing for critical sections. &a.rwatson; 07 November 2004 &status.done;
Make the kernel fully preemptive. &a.jhb; 24 November 2004 &status.done;
Lock pipe implementation: sigio/fown-related evil. &a.alfred; 24 November 2004 &status.done;
Lock down the SysV IPC code. &a.alfred; 24 November 2004 &status.done;
Lock contention measurement tool to measure heat of various locks, including Giant, and permit more directed performance and locking strategy optimization. &a.rwatson; 24 November 2004 &status.done;
Add KTR(9) tracing to scheduler run queues. &a.jeff; 26 December 2004 &status.done;
Review locking strategy and correctness of VFS operations and fix up various failure modes associated with enabling VFS locking assertions. &a.jeff; 01 January 2005 &status.done;
Document in-vnode locking strategy, clean it up. &a.jeff; 01 January 2005 &status.done;
Run cross-file system VFS without Giant, acquiring Giant conditionally based on a file system flag. &a.jeff; 01 January 2005 &status.done;
Run UFS file system MPSAFE. &a.jeff; 01 January 2005 &status.done;
Add KTR(9) tracing for buffer cache events. &a.jeff; 24 January 2005 &status.done;
Break out critical section and spin lock APIs, and re-optimize critical sections to not disable interrupts in hardware due to the high cost on some hardware architectures. &a.jhb; 04 April 2005 &status.done;
Modify uma(9) to use critical sections to protect per-CPU statistics, instead of mutexdes, in order to optimize access. &a.rwatson; 29 April 2005 &status.done;
Migrate malloc(9) to per-CPU statistics, and use critical sections to optimize access to those statistics. &a.rwatson; 29 May 2005 &status.done;
Add KTR(9) support for KTR_VFS to trace additional VFS events, rather than mechanically inserted KTR_VOP events. &a.jeff; 11 June 2005 &status.done;
Push the grabbing of Giant into Linux i386 ABI system calls. &a.jhb; 13 July 2005 &status.done;
Push the grabbing of Giant into Linux AXP ABI system calls. &a.jhb; 13 July 2005 &status.done;
Push the grabbing of Giant into SVR4 i386 ABI system calls. &a.jhb; 13 July 2005 &status.done;
Push the grabbing of Giant into OSF/1 AXP ABI system calls. &a.jhb; 13 July 2005 &status.done;
Push the grabbing of Giant into IBCS i386 ABI system calls. &a.jhb; 13 July 2005 &status.done;
Add a new witness check for exiting threads to verify that an exiting thread holds no locks. &a.jhb; 2 September 2005 &status.done;
Implement atomic_fetchadd() for ints. &a.jhb; 27 September 2005 &status.done;
Implement a simple reference count API using atomic operations and use this to replace locks that just protect a reference count. &a.jhb; 27 September 2005 &status.done;
Lock struct proc. &a.jhb; 20 February 2001 &status.wip;
Lock down the tty subsystem. Dick Garner, Jeremy Scofield, &a.tmm;, &a.phk; 24 July 2004 &status.wip;
Fix clock locking to be the same on all platforms. &a.jhb; 16 November 2001 &status.wip;
Make use of process locking and process reference counting to protect debugging interfaces (and procfs). &a.jhb; 27 February 2002 &status.wip;
Make use of process locking to protect process monitoring sysctls, including those employed by 'ps' and related tools. &a.jhb; 27 February 2002 &status.wip;
Lock down newbus infrastructure to support driver fine-graining. &a.imp; 28 February 2002 &status.wip;
Remove the MP safe syscall flag from the syscall table and add explicit mtx_lock/unlock's of Giant to all syscalls. &a.dillon;, &a.mux;, &a.rwatson; 24 July 2004 &status.wip;
SMPng architecture document. &a.jhb;, &a.rwatson; 28 February 2002 &status.wip;
Move to shared lock for VOP_GETATTR() to reduce blocking during frequent lightweight VFS operations. Modify namei() to provide a LOOKUP_SHARED flag to indicate when the lock required may be shared instead of exclusive. &a.jeff; 11 March 2002 &status.wip;
Document existing vm_map locking and verify it's correctness. &a.alc; 18 May 2002 &status.wip;
Document existing vm_object locking and verify it's correctness. &a.alc; 4 May 2002 &status.wip;
Implement lazy interrupt thread switching (context stealing) on i386. &a.bmilekic;, &a.kan; 10 December 2002 &status.wip;
Implement lazy interrupt thread switching (context stealing) on sparc64. &a.jake; 10 December 2002 &status.wip;
Switch from using lockmgr in VM to using a mutex or exclusive sxlock. Push down Giant on all VM except for vm_object/VFS and vm_page/pmap components. &a.alc; 10 December 2002 &status.wip;
Modify device driver API to permit drivers to more easily split "in interrupt context" and "in interrupt thread" code so as to acknowledge interrupts faster. This will permit lower latency in interrupt handling. &a.peter;, &a.scottl; 1 July 2005 &status.wip;
Split the interrupt handler list out of struct ithread into its own structure and only start up kthreads for interrupt vectors that actually have threaded interrupt handlers. &a.jhb; 18 October 2005 &status.wip;
Make printf() safe to call in almost any situation to avoid deadlocks. &a.cp; 15 May 2001 &status.stalled;
Conditionalize atomic ops in the SMP code that are used for debugging statistics. &a.peter; 15 March 2001 &status.new;
Axe schedcpu() in favor of event driven priority updates as much as possible.   7 September 2001 &status.new;
Fix *hold (e.g. crhold) to return reference to object.   7 September 2001 &status.new;
Add witness checking for lockmgr locks.   7 September 2001 &status.new;
Add ICU spin locks on ia64.   4 January 2002 &status.new;
Expand mutex profiling tool to also profile sx locks. &a.eivind;, &a.des; 1 April 2002 &status.new;
Add a witness_sleep() check to copyin/out() and s/fuword(). &a.jhb; 7 June 2002 &status.new;
Lock aio(4).   19 October 2005 &status.new;
Remove Giant from mi_startup() and push Giant down into individual SYSINIT functions. Many SYSINIT functions probably do not need Giant anyway.   21 October 2005 &status.new;

This table lists the todo subtasks for multithreading the network stack.

Task Responsible Last updated Status
Protect network interface queues. &a.jlemon; 24 November 2000 &status.done;
Lock up IP. &a.jennifer;, &a.hsu; 10 June 2002 &status.done;
Lock up TCP. &a.jennifer;, &a.hsu;, &a.sam;, &a.rwatson; 24 November 2004 &status.done;
Lock up UDP. &a.jennifer;, &a.hsu;, &a.rwatson; 24 November 2004 &status.done;
Lock ifaddr reference counts. &a.hsu; 18 December 2002 &status.done;
Lock up ifnet list. &a.hsu; 21 December 2002 &status.done;
Lock radix trees. &a.hsu; 23 December 2002 &status.done;
Lock up ARP. &a.hsu; 16 January 2003 &status.done;
Lock up raw IP. &a.sam;, &a.rwatson; 24 July 2004 &status.done;
Lock divert sockets. &a.sam; 4 October 2003 &status.done;
Lock ipfw2. &a.sam; 4 October 2003 &status.done;
Lock DUMMYNET. &a.sam; 4 October 2003 &status.done;
Lock ethernet bridge. &a.sam; 4 October 2003 &status.done;
Lock IP fragment queues. &a.rwatson; 4 October 2003 &status.done;
Lock routing entries. &a.sam; 4 October 2003 &status.done;
Lock FAST_IPSEC. &a.sam; 4 October 2003 &status.done;
Permit parallel entry into isr processing. &a.rwatson;, &a.sam; 11 October 2003 &status.done;
Lock if_disc "discard interface". &a.rwatson; 9 March 2004 &status.done;
Lock if_faith "IPv6-to-IPv4 TCP relay interface. " &a.sam;, &a.rwatson; 9 March 2004 &status.done;
Lock if_gif "generic tunnel interface". &a.rwatson; 9 March 2004 &status.done;
Review ECN tunnel support (ip_ecn.c). &a.rwatson; 9 March 2004 &status.done;
if_tap global and softc locking. &a.rwatson; 23 April 2004 &status.done;
if_tun global and softc locking. &a.rwatson; 23 April 2004 &status.done;
netatalk/aarp.c locking. &a.rwatson; 23 April 2004 &status.done;
Cache socket MAC label in inpcb label for IPv4 sockets so that the label can be used safely at the inet layer without socket locks. &a.rwatson; 23 April 2004 &status.done;
IP encapsulation subroutines (ip_encap.c). &a.rwatson; 23 April 2004 &status.done;
Lock globals in loopback interface (if_loop.c). &a.rwatson; 23 April 2004 &status.done;
Use m_tags in if_gif to limit looping configurations, rather than a non-MPSAFE static coutner. &a.ru; 23 April 2004 &status.done;
netatalk DDP PCB locking. &a.rwatson; 24 July 2004 &status.done;
Lock up syncache. &a.hsu;, &a.sam; 10 November 2003 &status.done;
Permit IP forwarding path to run Giant-free. &a.sam; 1 December 2003 &status.done;
Lock &unix; domain protocols, fifofs. &a.sam;, &a.rwatson; 24 July 2004 &status.done;
Giant lock over NFS server to protect against so_upcall() w/o Giant &a.rwatson; 24 July 2004 &status.done;
Lock interface cloning meta-data. &a.brooks; 24 July 2004 &status.done;
Apply combination of socket and socket buffer locks, label caching to MAC labels on sockets so that they can be used safely without Giant. &a.rwatson; 24 July 2004 &status.done;
Make routing socket message dispatch use a netisr to avoid re-entering the socket code from the routing code, resolving lock order issues. &a.rwatson; 24 July 2004 &status.done;
Introduce accept locking to protect accept incomplete and complete queues on listen sockets. &a.rwatson; 24 July 2004 &status.done;
Break out socket buffer wakeup, socket buffer append, socket state change, socket buffer reserve, flush, etc, calls into _locked() and unlocked versions, and avoid conditional locking. &a.rwatson; 24 July 2004 &status.done;
Lock down AARP, AppleTalk Address Resoluton Protocol. &a.rwatson; 24 July 2004 &status.done;
Fix pull/push cache data synchronization issues in sosend(), soreceive(), allowing them to run Giant-free. &a.rwatson; 24 July 2004 &status.done;
Protect socket global counters/limits and generation number with a mutex. &a.rwatson; 24 July 2004 &status.done;
Lock down unit allocation meta-data in interface related netgraph modules. &a.rwatson; 24 July 2004 &status.done;
Lock down socket buffer OOB fields across TCP/IP, IPX. &a.rwatson; 24 July 2004 &status.done;
Add MSG_NBIO so that fifofs can avoid frobbing SO_NBIO in a manner that risks races. &a.truckman; 24 July 2004 &status.done;
Protect all use of so_count with socket lock. &a.rwatson; 24 July 2004 &status.done;
Move socket buffer related state from so_state to sb_state so it can be properly locked by the socket buffer mutex. &a.rwatson; 24 July 2004 &status.done;
Introduce a temporary global lock to lock the if_label field used by the MAC Framework. &a.rwatson; 24 July 2004 &status.done;
Push VFS-specific behavior out of fdrop_locked() and acquire Giant in the fo_close per-object methods rather than fdrop_locked(), so that pipes and sockets can run fo_close() Giant-free. &a.rwatson; 24 July 2004 &status.done;
Push Giant acquisition into fo_stat() file descriptor stat operation, rather than acquiring it in fstat(), so that fstat() on sockets and pipes can run Giant-free. &a.rwatson; 24 July 2004 &status.done;
Don't hold socket locks over entry to protocol switch methods, allowing protocol methods to acquire socket locks after protocol locks in the lock order. &a.rwatson; 24 July 2004 &status.done;
Port inpcb mutex locking, assertions from IPv4 to IPv6. &a.rwatson; 8 August 2004 &status.done;
Add IFF_NEEDSGIANT to allow if_start to run with Giant for specific interfaces. Defer if_start to task queue. &a.rwatson; 8 August 2004 &status.done;
Push down Giant in stat(), fo_stat() to allow Giant-free stat of pipes, sockets. &a.rwatson; 8 August 2004 &status.done;
Add TCP lock assertions. &a.rwatson; 24 November 2004 &status.done;
Lock socket layer. &a.sam;, &a.rwatson; 24 November 2004 &status.done;
Review TCP timer code. &a.rwatson; 24 November 2004 &status.done;
Analyze and reduce cost of entropy gathering in network critical paths. &a.rwatson;, &a.markm; 24 November 2004 &status.done;
Allow code to declare NET_NEEDS_GIANT(), forcing Giant over the network stack if that code is compiled into the kernel. &a.rwatson; 28 August 2004 &status.done;
Disable Giant over the network stack in the default configuration. &a.rwatson; 28 August 2004 &status.done;
Additional KTR tracing for UMA, callouts, interrupts, etc. &a.rwatson; 07 November 2004 &status.done;
Move to using file descriptor reference counts instead of socket reference counts for socket system calls, avoiding extra reference couht operations. &a.rwatson; 24 October 2004 &status.done;
Lock IPv6. &a.sam;, &a.rwatson;, &a.ume;, &a.mlaier; 8 August 2004 &status.wip;
if_ppp global, per-softc locking. &a.rwatson;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org> 23 April 2004 &status.wip;
Lock struct ifnet. &a.mlaier;, &a.luigi;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org> 23 April 2004 &status.wip;
Lock IPv4, IPv6, atalk interface address lists. &a.mlaier;, &a.rwatson; 8 August 2004 &status.wip;
Lock consumers of BSD compress (bsd_comp.c) code to protect compression state. &a.rwatson; 23 April 2004 &status.wip;
Lock global and softc state for six-to-four converter (if_stf.c). &a.rwatson; 23 April 2004 &status.wip;
Lock down global and softc state for SLIP (if_sl.c). &a.rwatson; 23 April 2004 &status.wip;
Lock global and softc state for SPPP (if_sppsubr.c). &a.rik;, &a.rwatson; 23 April 2004 &status.wip;
IGMP locking. &a.rwatson; 23 April 2004 &status.wip;
IP ID locking. &a.ups; 24 June 2005 &status.wip;
Lock down netnatm. &a.rwatson; 01 July 2005 &status.wip;
Research and select options for inbound network stack parallelism. Direct dispatch is one option currently being considered. &a.rwatson; 19 October 2005 &status.wip;
Locking for polling(4). &a.pjd;, &a.glebius; 19 October 2005 &status.wip;
Reduce contention upon locking a socket buffer by replacing tsleep() and wakeup() with a condvar. &a.tanimura; 21 April 2002 &status.new;
Lock if_ef "ethernet frame" driver.   9 March 2004 &status.new;
Further cleanup of socket state machine in order to facilitate finishing socking locking of state transitions. &a.rwatson; 19 October 2005 &status.new;
Lock KAME IPSEC.   19 October 2005 &status.new;
Only one of our ATM stacks is MPSAFE, the other two should be deleted or fixed.   19 October 2005 &status.new;
Lock ND6 (IPv6 Neighbor Discovery). &a.gnn; 19 October 2005 &status.new;
Lock IPv6 multicast address lists.   19 October 2005 &status.new;
Lock IPv4 and IPv6 global address lists.   19 October 2005 &status.new;
Continued cleanup of the stack/device driver ownership and locking for struct ifnet needs to be done. Most fields are now either locked or assigned ownership. Some fields, such as if_flags, need a bit more cleanup due to device drivers modifying stack-owned fields. &a.rwatson; 19 October 2005 &status.new;
BPF locking needs some cleanup, there are some race conditions there relating to interface removal.   19 October 2005 &status.new;
When interfaces are torn down, there are a number of races (not all associated with SMPng) that need to be thought about.   19 October 2005 &status.new;
Lock if_vlan and inter-layer multicast address manipulation and synchronization in if_vlan. &a.glebius;, &a.yar; 19 October 2005 &status.new;
Further investigate locking in in_gif and in6_gif.   19 October 2005 &status.new;
FAST_IPSEC and KAME IPSEC's PF_KEY support likely needs an asynchronous dispatch to prevent socket lock ordering issues similar to what was done for PF_ROUTE. &a.rwatson; 19 October 2005 &status.new;
Investigate how to eliminate the use of ACCEPT_LOCK(), which currently prevents races in the tear-down of sockets. &a.rwatson; 19 October 2005 &status.new;
Fix SMP problems with netgraph restructuring. &a.glebius; 19 October 2005 &status.new;
Verify locking in netgraph nodes and improve where necessary. &a.glebius; 19 October 2005 &status.new;
More finely-grained locking for pf(4). &a.glebius; 19 October 2005 &status.new;

Known Issues

Issue Last updated Status
Idle processor time is not charged to the idle processes. 20 September 2000 &status.resolved;
microuptime creeps backwards. 4 October 2000 &status.resolved;
microuptime() went backwards 4 October 2000 &status.resolved;
Process accounting is not accurate (the more CPUs, the closer to correct it is). 5 October 2000 &status.resolved;
M_DEVBUF is probably the wrong memory pool for interrupt stuff and we should think about creating a new malloc pool for that stuff. 9 February 2001 &status.resolved;
PC card eject panics due to a race condition in the interrupt thread code. 15 March 2001 &status.resolved;
SMP x86 boxes are seeing NCPU * 100 clk interrupts and NCPU * 128 rtc interrupts. 15 May 2001 &status.resolved;
Witness will infinitely recurse when it acquires Giant after sleeping with a sleepable lock. 27 June 2001 &status.resolved;
Serial gdb does not work if boot_ddb and boot_gdb options are specified. 14 July 2002 &status.resolved;
Serial gdb does not work at 115200 baud. 14 July 2002 &status.resolved;
Serial gdb never regains control once 'cont' has been entered. 14 July 2002 &status.resolved;
Profiling is broken. 20 February 2001 &status.unresolved;

News

The remainder of this page is structured as a reverse-chronological log.

28 August 2004 30 October 2003 13 January 2002 15 May 2001 22 March 2001 5 March 2001 24 January 2001 12 January 2001 11 October 2000 6 September 2000 5 September 2000 1 September 2000 30 August 2000 12 August 2000 3 August 2000 6 July 2000 5 July 2000 3 July 2000 26 June 2000 25 June 2000 19 June 2000 &footer;