diff --git a/en/smp/index.sgml b/en/smp/index.sgml index 6352fed5ef..f5a2725a5d 100644 --- a/en/smp/index.sgml +++ b/en/smp/index.sgml @@ -1,1541 +1,1541 @@ - + %includes; Done"> In progress"> Stalled"> Not Started"> Resolved"> Unresolved"> %developers; ]> &header;

Contents

Project Goal

The FreeBSD SMP project, often referred to as SMPng (SMP next generation), is focused on implementing fine-grained SMP support for the FreeBSD 5.0 kernel (scheduled for November 2002). Due to FreeBSD's history, this is much like trying to fit a square peg into a round hole, and as such, the intermediate results aren't pretty in many ways. We are specifically not attempting to rewrite the kernel from scratch, nor are we on a crusade to fix all the architectural nits currently present in the kernel. In fact, we expect to leave a trail of architectural nits that will still be evident in many ways when FreeBSD 5.0 is released. This is a pragmatic project rather than a theoretical one; we need to have the kernel working and stable in under a year, so time restraints require that we be realistic about what to do when.

Project Plan

This web page contains information related to the effort to improve SMP support in FreeBSD. In general, this project uses what it can from the BSD/OS 5.0 development kernel, and re-implements what cannot be directly used due to divergence in the code bases.

As with any free software project, a detailed schedule is not possible. We expect to have significant performance and stability issues that need to be worked through over the first several months of the project, though every effort will be made to keep -current running as well as possible.

The task list below is not intended to be complete, but does represent a set of relevant and/or important components of the overall work. The "Responsible" field identifies a developer who has expressed willingness to be responsible for completing the identified task; this doesn't preclude others working on it, but suggests that coordination with the responsible party might be appropriate so as to avoid unnecessary duplication of work, and to maximize forward progress. If beginning work on a new area of substantial size, or one that appears unclaimed, it may be worth dropping an e-mail to the FreeBSD SMP mailing list to see if any progress has been made.

The definition of the date field varies depending on the status of a task. For completed tasks, it refers to the date completed or reported completed. For in-progress tasks, it refers to the date of the last update of the entry. For stalled tasks, it refers to the date that the task was declared stalled. For new tasks, it refers to the date the task was added to the list.

Tasks are sorted first by status, then by date.

Resources and Links

Status

Following is an incomplete list of general tasks.

- +
Task Responsible Last updated Status
Convert the giant lock from spinning to blocking, add the scheduler lock, add per-CPU idle processes. &a.dillon; 25 June 2000 &status.done;
Port the BSD/OS locking primitives (i386). &a.jake; 3 July 2000 &status.done;
Implement heavy-weight interrupt threads (i386). &a.grog; 3 August 2000 &status.done;
Rewrite the low level interrupt code (i386 UP). &a.grog; 3 August 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (i386 UP). -smp developers 12 August 2000 &status.done;
Port the BSD/OS locking primitives (alpha). &a.dfr; 24 August 2000 &status.done;
Stub out (disable) spl()s. &a.grog; 30 August 2000 &status.done;
Port the BSD/OS ktr code. &a.grog;, &a.jhb; 30 August 2000 &status.done;
Rewrite the low level interrupt code (i386 SMP). &a.jhb; 1 September 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (i386 SMP). -smp developers 6 September 2000 &status.done;
Demonstrated reasonable stability (self-hosted buildworld) (alpha). -smp developers 6 September 2000 &status.done;
Make malloc and friends thread-safe. &a.jasone; 10 September 2000 &status.done;
Implement msleep(), make tsleep() an msleep() wrapper. &a.jake; 11 September 2000 &status.done;
Make fxp driver thread-safe. &a.cp; 17 September 2000 &status.done;
Make mbuf's thread-safe. &a.bmilekic; 29 September 2000 &status.done;
Lock manager re-work. &a.jasone; 3 October 2000 &status.done;
Implement heavy-weight interrupt threads (alpha). &a.jhb;, &a.dfr; 5 October 2000 &status.done;
Rewrite the low level interrupt code (alpha). &a.dfr;, &a.jhb; 5 October 2000 &status.done;
Process accounting. Tor Egge, &a.jhb; 5 October 2000 &status.done;
Make ethernet drivers thread-safe. Bill Paul 15 October 2000 &status.done;
Make the mutex headers mostly machine-independent. &a.jhb; 20 October 2000 &status.done;
Rename SMP_DEBUG to MUTEX_DEBUG. &a.jhb; 20 October 2000 &status.done;
Give each soft interrupt its own thread. &a.cp; 25 October 2000 &status.done;
Make sf_bufs (sendfile(2)) thread-safe. &a.bmilekic; 5 November 2000 &status.done;
Make the witness code work correctly. &a.jhb; 18 November 2000 &status.done;
Split the ktr-specific code out of db_interface.c. &a.jhb; 15 December 2000 &status.done;
Convert the sio driver to using a spin mutex. &a.jhb; 18 December 2000 &status.done;
Implement condition variables. &a.jake;, &a.jasone; 15 January 2001 &status.done;
Add a flag to mtx_init() (MTX_RECURSE) that denotes whether a mutex is allowed to recurse. &a.bmilekic; 19 January 2001 &status.done;
Make the zone allocator thread-safe. Dag-Erling Smorgrav 21 January 2001 &status.done;
Convert simplelocks to mutexes. &a.jasone; 24 January 2001 &status.done;
Make kernel preemptive with respect to interrupts. &a.jake; 31 January 2001 &status.done;
Cleanup of mutex API. &a.bmilekic; 8 February 2001 &status.done;
Remove COM_LOCK. &a.markm; 11 February 2001 &status.done;
Merge various scheduling classes into one run queue. Modify scheduler to support preemptable kernel. &a.jake; 11 February 2001 &status.done;
Make priority propagation work correctly. &a.jake; 11 February 2001 &status.done;
Make most of the interrupt thread code MI and shared between hardware and software interrupts. &a.jhb; 18 February 2001 &status.done;
Add protection to struct jail and jail-related functionality. &a.rwatson; 20 February 2001 &status.done;
Implement sx (shared/exclusive) locks. &a.jasone; 5 March 2001 &status.done;
Generalize/improve witness to handle more complex locking primitives (mtx, sx). &a.jhb; 28 March 2001 &status.done;
Convert the allproc and proctree locks from lockmgr locks to sx locks. &a.jhb; 28 March 2001 &status.done;
Make mbuf system use condition variables instead of msleep()/wakeup(). &a.bmilekic; 2 April 2001 &status.done;
Remove <sys/mutex.h> includes from other kernel headers such as <vm/vm_zone.h>, <sys/resourcevar.h>, <sys/ucred.h>, and <sys/mbuf.h>. &a.markm; 15 May 2001 &status.done;
Cleanup the various mp_machdep.c's, unify various SMP API's such as IPI delivery, etc. &a.jhb; 15 May 2001 &status.done;
Make most of the forward_* and forwarded_* functions MI. &a.jhb; 15 May 2001 &status.done;
Complete the MD support for SMP on the Alpha platform. &a.gallatin;, &a.dfr;, &a.jhb; 15 May 2001 &status.done;
Convert select() to use condition variables. &a.tanimura; 15 May 2001 &status.done;
Add a "giant" lock around the VM subsystem. &a.alfred; 13 June 2001 &status.done;
Introduce a modified slab allocator for the mbuf subsystem. &a.bmilekic; 21 June 2001 &status.done;
Add a witness_assert() function to handle lock assertions. &a.jhb; 27 June 2001 &status.done;
Extend sx locks to support try lock operations. &a.jhb; 27 June 2001 &status.done;
Document KTR. &a.jhb; 28 June 2001 &status.done;
Make fork_return, fork_exit, ast, and userret MI. &a.jhb; 29 June 2001 &status.done;
Make sched_lock's savecrit a per-process property saved and restored in mi_switch and initialized in fork_exit. &a.jhb; 30 June 2001 &status.done;
Make ast() loop. &a.jhb; 10 August 2001 &status.done;
Add upgrade/downgrade sx lock operations. Alexander Kabaev, &a.jasone; 13 August 2001 &status.done;
Implement semaphores. &a.jasone; 14 August 2001 &status.done;
Add support for upgrade/downgrades in witness. &a.jhb; 23 August 2001 &status.done;
Make most of cpu_wait() and cpu_exit() MI. &a.peter; 9 September 2001 &status.done;
Split NFS into client and server. &a.peter; 18 Oct 2001 &status.done;
Lock taskqueues. Andrew Reiter, &a.jhb; 25 October 2001 &status.done;
Add a per-thread ucred reference. &a.jhb; 25 October 2001 &status.done;
Make most of the per-CPU stuff MI. &a.jhb; 11 December 2001 &status.done;
Make critical section saved state per-thread instead of per-lock so that interlocking spin locks work properly. &a.jhb; 17 December 2001 &status.done;
Replace the APIC-specific imen_mtx with a MI-named icu_lock to protect interrupt controllers and associated data within the kernel for both i386 and alpha. &a.jhb; 20 December 2001 &status.done;
Use the per-thread critical section nesting level in the mutex and interrupt thread code to automatically determine when to not preempt. This makes the MTX_NOSWITCH, SWI_SWITCH, and SWI_NOSWITCH flags obsolete as the kernel will be able to figure out the proper behavior on its own. &a.jhb; 5 January 2002 &status.done;
Lock struct filedesc and struct file. &a.tanimura;, &a.alfred; 12 January 2002 &status.done;
Lock struct pgrp, struct session, and struct sigio. &a.tanimura; 23 February 2002 &status.done;
Lock pipe implementation, but not sigio/fown, VM interactions &a.alfred; 27 February 2002 &status.done;
Move to explicit reference counting for soft vnode references. &a.phk; 8 March 2002 &status.done;
Initialize mutex pools early enough that sx locks can be used for VM. &a.green; 14 March 2002 &status.done;
Place a global lock (sellock) around selinfo structures to fix a variety of lock order reversals, and make select() MP-safe. &a.alfred;, &a.davidc; 14 March 2002 &status.done;
Push down Giant on read, write, pread, pwrite system calls, acquiring Giant in the per-subsystem fileop layer for sockets, VFS, etc. &a.alfred; 15 March 2002 &status.done;
Lock down kernel module structures. &a.arr; 18 March 2002 &status.done;
Rewrite kernel memory allocator to be a slab allocator that uses per-cpu caches. &a.jeff; 21 March 2002 &status.done;
Lock down access to the shared p_args "process arguments" structure through appropriate protection of that structure and references to it. Jonathan Mini 31 March 2002 &status.done;
Move from flags/tsleep lock to sx locks to protect sysctl tree from updates during sysctl operations. Jonathan Mini 1 April 2002 &status.done;
Create/port userland tool to manage KTR event dumps. &a.jake; 1 April 2002 &status.done;
Create MTX_SYSINIT and SX_SYSINIT macros that allow for initializing locks that are subsystem independent. &a.arr; 2 April 2002 &status.done;
Lock down the global securelevel variable. &a.arr; 2 April 2002 &status.done;
Make grow_stack() MI. Possibly even a macro or inline. &a.alc; 6 April 2002 &status.done;
Lock use of p_fd, which otherwise can result in corrupted p_fd panics during heavy operation. Start with a global, and move to per-proc locking. &a.alfred;, &a.tanimura; 8 April 2002 &status.done;
Lock struct proc. &a.jhb; 20 February 2001 &status.wip;
Make KTRACE write into tracefiles asynchronously. &a.jhb; 6 September 2001 &status.wip;
Make the kernel fully preemptive. &a.jhb; 7 September 2001 &status.wip;
Lock down the tty subsystem. Dick Garner, Jeremy Scofield, &a.tmm; 2 April 2002 &status.wip;
Fix clock locking to be the same on all platforms. &a.jhb; 16 November 2001 &status.wip;
Make sleep mutexes spin if the current lock holder is executing on another CPU. &a.jhb; 16 November 2001 &status.wip;
Add support for the IA32 pause instruction to spin loops in locks. &a.jhb; 16 November 2001 &status.wip;
Optimize UP support by changing spin locks to only perform critical section enter and exits. &a.jhb; 20 December 2001 &status.wip;
Implement lazy interrupt thread switching (context stealing) on i386. &a.bmilekic; 4 January 2002 &status.wip;
Fix synchronization of TLB flushes and invlpg() on x86 SMP. &a.peter; 4 January 2002 &status.wip;
Lock pipe implementation: sigio/fown-related evil &a.alfred; 27 February 2002 &status.wip;
Broadly transition to td_ucred from p_ucred once KSE dependencies are in place. &a.jhb; 27 February 2002 &status.wip;
Update suser() and p_can*() APIs to accept threads instead of processes. &a.jhb; 1 April 2002 &status.wip;
Make use of process locking and process reference counting to protect debugging interfaces (and procfs). &a.jhb; 27 February 2002 &status.wip;
Make use of process locking to protect process monitoring sysctls, including those employed by 'ps' and related tools. &a.jhb; 27 February 2002 &status.wip;
Lock down TrustedBSD MAC implementation. &a.rwatson; 27 February 2002 &status.wip;
Lock down newbus infrastructure to support driver fine-graining. &a.imp; 28 February 2002 &status.wip;
Remove the MP safe syscall flag from the syscall table and add explicit mtx_lock/unlock's of Giant to all syscalls. &a.dillon;, &a.mux; 28 February 2002 &status.wip;
SMPng architecture document. &a.jhb; 28 February 2002 &status.wip;
Move to shared lock for VOP_GETATTR() to reduce blocking during frequent lightweight VFS operations. Modify namei() to provide a LOOKUP_SHARED flag to indicate when the lock required may be shared instead of exclusive. &a.jeff; 11 March 2002 &status.wip;
Lock down kernel linker structures. &a.arr; 18 March 2002 &status.wip;
Replace incorrect use of MD critical section API to disable interrupts with a specific interrupt disable API. &a.imp;, &a.dfr;, &a.benno;, &a.jhb; 21 March 2002 &status.wip;
Rewrite kernel memory allocator so that Giant is not required for malloc() or free(). &a.jeff; 21 March 2002 &status.wip;
Create mutex profiling tool for the kernel so as to measure contention and behavior of kernel mutexes. &a.eivind;, &a.des; 31 March 2002 &status.wip;
Lock eventhandlers. &a.msmith;, Jonathan Mini 8 April 2002 &status.wip;
Create mechanism in cdevsw structure to protect thread-unsafe drivers. &a.jhb; 15 May 2001 &status.stalled;
Make printf() safe to call in almost any situation to avoid deadlocks. &a.cp; 15 May 2001 &status.stalled;
Add locking to NFS.   15 May 2001 &status.new;
Use per-CPU buffers for ktr to reduce synchronization.   5 September 2000 &status.new;
Remove priority argument from tsleep(), msleep(), cv_*wait*().   12 January 2001 &status.new;
Reimplement kqueue using condition variables. &a.jlemon; 15 March 2001 &status.new;
Conditionalize atomic ops in the SMP code that are used for debugging statistics. &a.peter; 15 March 2001 &status.new;
Add a new witness check for exiting processes to verify that an exiting process holds no locks. &a.jhb; 13 June 2001 &status.new;
Make cpu_core MI. Make cpu_coredump MI.   13 June 2001 &status.new;
Lock struct pargs.   20 June 2001 &status.new;
Specify priorities for condition variables, semaphores, and sx locks.   7 September 2001 &status.new;
Fix SIGXPU and other #if 0'd things in mi_switch().   7 September 2001 &status.new;
Axe schedpu() in favor of event driven priority updates as much as possible.   7 September 2001 &status.new;
Fix PHOLD() so that it blocks to guarantee PS_INMEM.   7 September 2001 &status.new;
Fix *hold (e.g. crhold) to return reference to object.   7 September 2001 &status.new;
Fix various procfs_machdep.c to use PHOLD, not sched_lock.   7 September 2001 &status.new;
Add witness checking for lockmgr locks.   7 September 2001 &status.new;
Add ICU spin locks on ia64.   4 January 2002 &status.new;
Lock sysctl hierarchy and access methods.   25 February 2002 &status.new;
Fast-path push-down of Giant for VOP_READ() and VOP_WRITE().   25 February 2002 &status.new;
Lock contention measurement tool to measure heat of various locks, including Giant, and permit more directed performance and locking strategy optimization.   25 February 2002 &status.new;
Push the grabbing of Giant into Linux i386 ABI system calls.   25 February 2002 &status.new;
Push the grabbing of Giant into Linux AXP ABI system calls.   25 February 2002 &status.new;
Push the grabbing of Giant into SVR4 i386 ABI system calls.   25 February 2002 &status.new;
Push the grabbing of Giant into OSF/1 AXP ABI system calls.   25 February 2002 &status.new;
Push the grabbing of Giant into IBCS i386 ABI system calls.   25 February 2002 &status.new;
Lock pipe implementation: VM optimizations.   27 February 2002 &status.new;
Document in-vnode locking strategy, clean it up, remove interlock, switch to sx locks. &a.jeff; 27 February 2002 &status.new;
Review locking strategy and correctness of VFS operations and fix up various failure modes associated with enabling VFS locking assertions. &a.jeff; 27 February 2002 &status.new;
Push down Giant into readv/writev system calls in style of read/write/pread/pwrite once malloc no longer requires Giant in the handling of iovec structures for uio.   15 March 2002 &status.new;
Replace complex shared/exclusive locking scheme in the VM system with a purely exclusive lockmgr locking scheme, simplifying locking and removing potential livelock/deadlock scenarios. &a.green;, &a.alc; 18 March 2002 &status.new;
Switch from using lockmgr in VM to using a mutex or exclusive sxlock. Push down Giant on all VM except for vm_object/VFS and vm_page/pmap components. &a.green;, &a.alc; 18 March 2002 &status.new;
Expand mutex profiling tool to also profile sx locks. &a.eivind;, &a.des; 1 April 2002 &status.new;

This table lists the todo subtasks for multithreading the network stack.

Task Responsible Last updated Status
Protect network interface queues. &a.jlemon; 24 November 2000 &status.done;
Lock struct ifnet.   19 January 2001 &status.new;

Known Issues

Issue Last updated Status
Idle processor time is not charged to the idle processes. 20 September 2000 &status.resolved;
microuptime creeps backwards. 4 October 2000 &status.resolved;
microuptime() went backwards 4 October 2000 &status.resolved;
Process accounting is not accurate (the more CPUs, the closer to correct it is). 5 October 2000 &status.resolved;
M_DEVBUF is probably the wrong memory pool for interrupt stuff and we should think about creating a new malloc pool for that stuff. 9 February 2001 &status.resolved;
PC card eject panics due to a race condition in the interrupt thread code. 15 March 2001 &status.resolved;
SMP x86 boxes are seeing NCPU * 100 clk interrupts and NCPU * 128 rtc interrupts. 15 May 2001 &status.resolved;
Witness will infinitely recurse when it acquires Giant after sleeping with a sleepable lock. 27 June 2001 &status.resolved;
Serial gdb does not work if boot_ddb and boot_gdb options are specified. 5 September 2000 &status.unresolved;
Serial gdb does not work at 115200 baud. 5 September 2000 &status.unresolved;
Profiling is broken. 20 February 2001 &status.unresolved;
jail_sysvipc_allowed is checked in an unsafe manner in the SYSV IPC syscalls. 5 March 2002 &status.unresolved;
Serial gdb never regains control once 'cont' has been entered. 25 March 2002 &status.unresolved;

News

The remainder of this page is structured as a reverse-chronological log.

13 January 2002 15 May 2001 22 March 2001 5 March 2001 24 January 2001 12 January 2001 11 October 2000 8 September 2000 6 September 2000 5 September 2000 1 September 2000 30 August 2000 12 August 2000 3 August 2000 6 July 2000 5 July 2000 3 July 2000 26 June 2000 25 June 2000 19 June 2000 &footer;