One of the benefits of the FreeBSD development model is a focus on centralized design and implementation, in which the operating system is maintained in a central repository, and discussed on centrally maintained lists. This allows for a high level of coordination between authors of various components of the system, and allows policies to be enforced over the entire system, covering issues ranging from architecture to style. However, as the FreeBSD developer community has grown, and the rate of both mailing list traffic and tree modifications has increased, making it difficult even for the most dedicated developer to remain on top of all the work going on in the tree.
The FreeBSD Monthly Development Status Report attempts to address this problem by providing a vehicle that allows developers to make the broader community aware of their on-going work on FreeBSD, both in and out of the central source repository. This is the first issue, and as such is an experiment. For each project and sub-project, a one paragraph summary is included, indicating progress since the last summary (in this case, simply recent progress, as there have been no prior summaries).
This status report may be reproduced in whole or in part, as long as the source is clearly identified and appropriate credit given.
Assuming there is some positive feedback on this idea, and that future submissions get made such that there is content for future issues, the goal is to release a development status report once a month. As such, the next deadline will be July 31, 2001, with a scheduled publication date in the first week of August. This will put the status report on a schedule in line with the calendar, as well as providing a little over a month until the next deadline, which will include a number of pertinent events, including the Annual USENIX Technical Conference in Boston, MA. Submissions should be e-mailed to:
robert+freebsd.monthly@cyrus.watson.org
Many submitters will want to wait until the last week of July so as to provide the most up-to-date status report; however, submissions will be accepted at any time prior to that date.
-- Robert Watson < rwatson@FreeBSD.org >
The FreeBSD Binary Updater Project aims to provide a secure mechanism for the distribution of binary updates for FreeBSD. This project is complementary to the Open Packages and libh efforts and there should be very little overlap with those projects. The system uses a client / server mechanism that allows clients to install any known "profile" or release of FreeBSD over the network. Where a specific profile might contain a specific set of FreeBSD software to install, additional packages, and configuration actions that make it more ideal for a specific environment (ie FreeBSD 4.3 Secure Web Server Profile)
The system can currently be used to install a FreeBSD system or perform the most simple of upgrades but many features are absent. In particular, the client is in its infancy and much work remains to be done. We need additional developers so please get in touch with us at updater@osd.bsdi.com if you are interested in spending some cycles on this.
Poul-Henning Kamp kicked off a drive to get our GNATS PR database cleaned up so the wheat can be sorted from the chaff. Progress is good, but there is still a lot of work to do. Give a hand if you can. Remember: every unhandled PR is a pissed off contributor or user.
I'm in the process of rewriting the CVSROOT/scripts to make them more clean and configurable. A lot of other projects also use these and so it makes sense to make them as easy to use in other environments as possible.
Status: work in progress. There is now a configuration file, but not all the scripts use it yet.
Work is progressing on implementing true cloning devices in DEVFS. Brian Somers and Poul-Henning Kamp are working to make if_tun the first truly cloning driver in the system. Next will be the pty driver and the bpf driver.
From July 1st DEVFS will be standard in -current.
Added the digi driver. Initial work was done by John Prince <johnp@knight-trosoft.com>, but all the modular stuff was done by me and initial work on supporting Xe and Xi cards (ala dgb) was done by me. I'm now awaiting an Xe card being sent from joerg@ (almost a donation) so that I can get that side of things working properly.
Ben Smithurst has written a "diskcheckd" daemon which will read all sectors on the disks over a configured period. With recent increases in disksizes it is by no means a given that disk read errors will be discovered before they are fatal. This daemon will hopefully result in the drive firmware being able to relocate bad sectors before they become unreadable. This code is now committed to 5.0-CURRENT.
In the last month (May-June), the new fxp driver was brought into -stable. This new driver uses the common MII code, so support for new PHYs is easy to add. Support for the new Intel 82562 chips was added. The driver was updated to add VLAN support and a workaround for a bug affecting Intel 815-based boards.
The FreeBSD Java Project has continued its "behind the scenes" work over the last month. Progress was made both technically, with the help of Bill Huey (of Wind River), on a port of JDK 1.3.1 and legally, with Nate Williams continuing negotiations with Sun on a mutually acceptable license to release a binary Java 2 SDK under. The JDK 1.2.2 port has also seen some development, with a new patchset likely to be released soon which includes JPDA and NetBSD support (the latter courtesy of Scott Bartram).
The Kernel Graphics Interface project has worked for several years to provide a framework for graphic drivers under Linux receiving input from other groups like the UDI project. Currently the KGI core implementation is quite settled, as is the driver coding model as a whole. Work is being done to newbussify KGI and produce a kld, as part of a future redesign of the graphics subsystem in FreeBSD. KGI will be an alternative for graphic card producers that don't accept the XFree86 model of userland graphic adapters and will also provide accelerated support for any other graphic alternative.
The libh project is a next generation sysinstall. It is written in C++ using QT for its graphical frontend and tvision for its console support. The menus are scriptable via an embedded tcl interpreter. It has been growing functionality quite a bit lately, including a new disklabel editor. Current work is on installation scripts for CDROM, FTP, ... installs as well as a fully functional standalone disk-partition and label editor. The GUI API was extended a little and many bugs were fixed. There seems to be some interest in i18n work.
Maxime Henrion is working on implementing a new and more extensible mount(2) systemcall, mainly to overcome the 32 bits for mountoptions limit, secondary goal to make it possible to mount filesystems from inside the kernel.
In the last two months, the OLDCARD pccard implementation was rototilled to within an inch of its life. Many new pci cardbus bridges were added. Power handling was improved. PCI Card cardbus bridges are nearly supported and should be committed in early June to the tree. This will likely be the last major work done on OLDCARD. After pci cards are supported, work will shift to improving NEWCARD.
The PowerPC port is proceeding well. All seems to be working in pmap.c after a number of problems encountered where FreeBSD passes a vm_page_t to a NetBSD-derived function that expects a vm_offset_t. Then after debugging the atomic operations code, I'm now at the point where VM appears to be initialized and it's now hanging while in sys/kern/kern_malloc.c:kmeminit(). Progress continues. =)
Developing full MPPE support for Andre Opperman @ Monzoon in Switzerland. Work is now complete and will eventually be brought into -current, but no dates are yet known.
Pseudofs is a framework for pseudo-filesystems, like procfs and linprocfs. The goal of pseudofs is twofold:
Pseudofs has reached the point where it is sufficiently functional and stable that linprocfs has been almost fully reimplemented on top of it; the only bit that's missing is the proc/<pid>/mem file.
The primary to-do item for pseudofs right now is to add support for writeable files (which are required for procfs, and are quite a bit less trivial to handle than read-only files). In addition, pseudofs needs either generic support for raw (non-sbuf'ed, possibly mmap'able) files, or failing that, special-case code to handle proc/<pid>/mem.
RELNOTESng is the name I've given to the rewrite of the *.TXT files that typically accompany a FreeBSD release. The information from these files (which include, among other things, the release notes and the supported hardware list) have been reorganized and converted to SGML. This helps us produce the documentation in various formats, as well as facilitating the maintenance of documentation for multiple architectures. This work was recently committed to -CURRENT, and I intend to MFC it to 4-STABLE before 4.4-RELEASE.
The SMPng project aims to provide multithreaded support for the FreeBSD kernel. Currently the kernel still runs almost exclusively under the Giant kernel lock. Recently, progress has been made in locking the process group and session structures as well as file descriptors by Seigo Tanimura-san. Alfred Perlstein has also added in a giant lock around the entire virtual memory (VM) subsystem which will eventually be split up into several smaller locks. The locking of the VM subsystem has proved tricky, and some of the current effort is focused on finding and fixing a few remaining bugs in on the alpha architecture.
mb_alloc is a new specialized allocator for mbufs and mbuf clusters. Presently, it offers various important advantages over the old (status quo) mbuf allocator, particularly for MP machines. Additionally, it is designed with the possibility of future enhancements in mind.
Presently in initial review & testing stages, most of the code is already written.
Work has (re)started on a port of FreeBSD to the UltraSPARC architecture, specifically targeting PCI based workstations. Jake Burkholder will be porting the kernel, and Ade Lovett has expressed an interest in working on userland. Recent work on the project includes:
At this point the kernel can be net-booted and prints the FreeBSD copyright before calling code that is not yet implemented. I am currently working on a design for the pmap module and plan to begin implementation in the next few days.
The TrustedBSD Project seeks to improve the security of the FreeBSD operating system by adding new security features, many derived from common trusted operating system requirements. This includes Access Control Lists (ACLs), Fine-grained Event Logging (Audit), Fine-grained Privileges (Capabilities), Mandatory Access Control (MAC), and other architecture features, including file system extended attributes, and improved object labeling.
Individual feature status reports are documented separately below; in general, basic features (such as EAs, ACLs, and kernel support for Capabilities) will be initially available in 5.0-RELEASE, conditional on specific kernel options. A performance-enhanced version of EAs is currently being targeted at 6.0-RELEASE, along with an integrated capability-aware userland, and MAC support.
Patches are now available to add ACL support to cp(1) and mv(1) along with preliminary support for install(1). Ilmar's i18n patches for getfacl(1) and setfacl(1) need to be updated for the last set of changes and committed. Some other functional improvements are also in the pipeline.
The kernel part of the capability implementation is mostly finished; all uses of suser() and suser_xxx() and nearly all comparisons of uid's with 0 have been converted to use the newly introduced cap_check() call. Some details still need clarification. More documentation for this needs to be done.
POSIX.2c-compatible getfcap and setfcap programs have been written. Experimental capability support in su(1), login(1), install(1) and bsd.prog.mk is being tested.
Support for capabilities, ACL's, capabilities and MAC labels in tar(1) is being developed; only the capability part is tested right now. Generic support for extended attributes is planned, this will require extensions to the current EA interface, which are written and will probably be committed to -CURRENT in a few weeks. A port of these features to pax(1) is planned.
An initial prototype of a Mandatory Access Control implementation was completed earlier this year, supporting Multi-Level Security, Biba Integrity protection, and a more general jail-based access control model. Based on that implementation, I'm now in the process of improving the FreeBSD security abstractions to simplify both the implementation and integration of MAC support, as well as increase the number of kernel objects protected by both discretionary and mandatory protection schemes. Generic object labeling introduces a structure not dissimilar in properties to the kernel ucred structure, only it is intended to be associated with kernel objects, rather than kernel subjects, permitting the creation of generic security protection routines for objects. This would allow the easy extension of procfs and devfs to support ACLs and MAC, for example. A prototype is underway, with compiling and running code and simple protections now associated with sysctl's.
Last month's status report was apparently a great success: I received countless e-mails with comments, questions, and suggestions. I've tried to incorporate any suggestions and address any problems from these e-mails in this month's report, which captures a far more extensive snapshot of FreeBSD activity in the last month. Unlike last month's report, it does a better job of reflecting non-development activity, such as on-going conference planning, documentation, and so on. This is a trend I hope to see improve in future months as well.
On the topic of conferences, in the future I'd like to report more on publication activities relating to FreeBSD, including online journals with articles relating to FreeBSD, paper journals, conference papers, and so on. Likewise, I would be interested in including references to Call for Papers relating to FreeBSD. I'll take this opportunity to plug both registration and paper submission for BSDCon Europe in November, which has status included in this report, and for the general BSD Conference being hosted by USENIX in February. Your attendance and submissions make these conferences "happen", and promote FreeBSD as a platform for new research, feature development, and application products. Work of extremely high calibre is performed on FreeBSD, and we need to get the word out.
Next month, we're maintaining much the same submission requirements: reports should be one or two paragraphs long, sent by e-mail, and approximate the layout of the entries this month (Project, Contact, URL, and text). I'll send out reminders again over the week before the deadline, with more specific instructions. An area where I'd like to explore improvement lies in the coordination of related status reports for larger projects, such as new architectural work or platform ports. This might even have the effect of encouraging communication within these projects :-). I'd like to continue to focus on pulling in a broader range of groups and their activities, including the Security Officer, Release Engineer, and Core Team.
-- Robert Watson < rwatson@FreeBSD.org >
ACPI (Advanced Configuration and Power Interface) is an industry standard which obsoletes APM, Intel MPS, PnPBIOS, and other Intel PC firmware interface standards. It is also used on the IA64 platform. More information on ACPI is available at
http://developer.intel.com/technology/iapc/acpiThe FreeBSD ACPI subsystem project is based heavily on the Intel ACPI Component Architecture. This status report outlines the current state of the project; future updates will focus on changes as they occur.
The Intel ACPI interpreter is fully integrated, although bugs are still coming out of the woodwork occasionally.
Work is ongoing in the following areas:
The ARM port is currently going pretty well. The kernel is compiling and is able to boot to the point where it panics trying to initialize the network subsystem. The current reference platform is the Netwinder but this may change as many people expressed interest in a more broadly available platform. Things that need to be done before it can get further includes adding footbridge, timer and interrupt supports. The pmap module is not completed yet either.
Now that BIND 8.2.4 is finally imported the time has come to look at getting BIND 9 imported into CURRENT. The current idea is to have it imported alongside BIND 8 so that people can play with either one until all import problems have been taken care of and people have tested it a bit.
Although gaining a new name, the project has been at a standstill due to both resource availability during the move between BSDi and Wind River, and other commitments of the developers. The project should obtain an official mailing list, as well as return to an active state after the dust settles.
The conference will take place at the Thistle Hotel, Brighton, UK from 9-11 November 2001.
The aim of the conference is to provide a focal point for European users and developers of all the BSD derived operating systems. The format will be similar to other conferences, with 2 days of technical sessions over the Saturday and Sunday.
We'll be finalizing the schedule towards the end of the month and anybody who is interested in doing a talk should contact us ASAP. There are no restrictions on the use of talks; if it's been done before we may still be interested in having it presented to an European audience, and we make no claims to the talks so speakers are free to present the talks again at other conferences.
We're also still looking for sponsors.
We had 80 pre-registrations in the first week so we're expecting a good turnout.
The new CAM transport code is starting to get supported in more HBAs and to get refined so that it does the intended per-protocol support. No progress on doing any SMPng work for CAM has been made yet. This is a fairly high priority.
Thanks to various outstanding individual efforts, we are now down to just below 2300 open bug-reports. This means that we have fought our way back to the level we had around march 2000.
Work continues (in large part sponsored by WRS) on updating the Handbook ready for the second print edition. There has been a flurry of activity in this area recently, and the ToDo list can be seen at
http://www.FreeBSD.org/docproj/handbook.html
Dima and others are doing a stellar job of keeping up with the steady flow of incoming PRs relating to the documentation project.
The Developers' Handbook,
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/index.html
is a year old; it contains a wealth of useful content for developers developing on, or for, FreeBSD. As ever, more contributions are always required, not only for the developers' handbook, but for all of the FreeBSD documentation set.
The basic design hasn't changed and this project mainly is in the phase of continued hardening and test case development. The next major feature will be to fully integrate into the new CAM TRAN code and to fully support on the fly device addition and removal. The only HBA supported is QLogic at this time. Future support for the QLogic line is planned to have 2300 (2Gb) and IP support before October.
Hardware watchpoints are now available for kernel debugging on the IA32 (i386) architecture. One can now set hardware watchpoints using the new ddb command 'hwatch', which is analogous to the existing 'watch' command. Alternatively, if greater flexibility is required, direct access to the debug registers is available using the ddb 'set' command which allows complete control over the processor hardware debug facilities. Hardware watchpoints are very useful in tracking down those elusive memory overwrite bugs in the kernel. Hardware watchpoints can even be used to set a code breakpoint in ROM, which is commonly found in embedded systems.
Support for configuring IEEE 802.11 wireless devices via ifconfig has been committed to -current and -stable. It contains most of the functionality needed to configure an wireless device. Some missing features are being worked on including integrated support for DHCP so a single entry in /etc/rc.conf can be used to fully configure a wireless device on a DHCP lan and setting the CTS/RTS threshold. Currently the an(4) and wi(4) drivers are supported in -current and -stable with the awi(4) device supported in -current. Further work is needed to support Frequency Hopping devices such as ray(4).
jailNG is a from-scratch rewrite of the popular jail(8) service, focusing on improved management functions, as well as more fine-grained configurability. An initial prototype has been written, based on explicitly named and configured jails, and work is proceeding on userland integration. Currently, it's not clear if the timeline for this will be 5.0-RELEASE, or 5.1-RELEASE.
The main development in the FreeBSD Java Project over the last month was the release of an initial "Developers Only" patchset for the JDK 1.3.1. Since that release progress had been made towards a much more usable alpha quality patchset which is likely to be turned into a port, as per the current JDK 1.2.2 patchset. This new patchset will feature a number of bugfixes, which essentially get the JDK to a working state for early adopters, and an initial implementation of "native threads" based on FreeBSD's userland pthreads. Unfortunately this implementation isn't fully functional, but is included in the hope of getting more eyeballs on the code (particularly experienced pthread programmers). We'd also like to welcome Fuyuhiko Maruyama-san as a new committer, the usual punishment for too many good patches.
We have been working to provide Japanese version of FreeBSD online manuals, since 1996. Currently, RELENG_4 manuals are based. Translated versions are placed on doc/ja_JP.eucJP/man and provided to users using ports/japanese/man-doc. Also, we discuss about related commands (e.g. ports/japanese/man and ports/japanese/groff).
The first FreeBSD kernel summit meeting was held June 29-30, 2001 in Boston, MA at the Usenix 2001 Annual Technical Conference. Links to a variety of files are posted on the web site.
Note: I (jhb) am still working on writing up a general summary of the meeting. When that is completed it will be posted here and mailed to the -hackers mailing list.
I'm working on multithreading the kernel. So far I have over 400KB of diffs relative to todays -current (I'm keeping my tree updated with changes as they occur rather than get hit with a big update at the end).
I have split the proc structure and am changing most of the kernel to pass around a thread identifier instead of a proc structure.
The following interfaces have been changed so far:
I have still a lot of work to go with a lot of "dumb editing" (s/struct proc \*p/struct thread \*td/) usually I change a few items and then fix everything that breaks when I try compile it. I'd like to check it in on a branch so others can help the editing but haven't worked out the best way to do it yet.
I have implemented changes to the scheduler so that KSE's are scheduled instead of processes, and threads sleep, letting the KSE pick up a new thread. but it's not anywhere ready yet (heck it doesn't compile yet :-)
Note that I have not yet updated the document listed above.. everywhere it mentions "ksec" or "KSE-context", the code uses the word "thread". I will update it soon as Jason has sent me the source.
The FreeBSD Monthly Development Status Report aims to keep users and developers up-to-date on the latest goings-on in the FreeBSD project by providing summaries of each project and its status. At the time of this writing, the July 2001 status report is being prepared and is very near release. The FreeBSD Web site now has a Status Reports section, which, when the July 2001 report is released, will be updated to include a link to an HTML-ified version.
The NetBSD rc.d port aims to improve the FreeBSD startup process by porting Luke Mewburn's rc.d work from NetBSD to FreeBSD. This will score FreeBSD startup and shutdown dependencies without losing the traditional and much loved monolithic configuration filesystem.
Luke Mewburn's USENIX paper and slides on the system as implemented in NetBSD are available here:
http://groups.yahoo.com/group/FreeBSD-rc/message/3
Interested parties are urged to study this material before joining the discussion list.
The intention at this stage is to decide on an approach that will ensure that the differences between the NetBSD rc.d system and the system as ported to FreeBSD will be kept to a minimum. This will probably involve discussions with Luke around those areas of the system that are identified as areas for potential improvement.
The goal of this project is the implementation of ATM signalling and other ATM protocols by means of the netgraph(4) framework. This should provide an easily extensible architecture for using ATM on FreeBSD. Currently the full UNI4.0 stack (except for the LIJ capability) has been implemented, including ILMI and a first version of the ATM Forum API for UNI. An implementation of Classical IP over ATM is also available. Drivers have been implemented for the Fore PCA200E and Fore HE-155 cards.
Network device cloning support has been imported from NetBSD. This allows virtual devices to be allocated on demand rather then being statically allocated at compile time. Our implementation differs slightly from that of NetBSD's in that we allow both the creation of specific devices (i.e. gif0) and arbitrary devices instead of just allowing specific devices. Currently, the only device in the tree which has been converted is the gif(4) device which has been converted in both -current and -stable. Work is ongoing to convert all other virtual network devices with work in progress on faith, stf, and vlan interfaces. In general this conversion is accompanied by appropriate modifications to make these devices fully modular.
NGPT is an effort led by IBM engineers to implement MxN threads (also known as many user threads to one kernel thread mapping) on Linux. I have ported it to FreeBSD to use rfork(2).
The port is right here:
Funded by: Monzoon Networking, LLC
This month has been a month of conventration and consolidation. Much of the changes from current have been migrating into stable. I've improved power support, suspend/resume interactions, interrupt handling, and ability to work after windows/NEWCARD has run. Interrupt routing continues to be a locking issue for a complete MFC. Current patches are available at the above website. I'm racing to get this done before 4.4 is released.
Information on Intel ORP - a BSD licensed Java VM is right here:
http://www.intel.com/research/mrl/orp/
A FreeBSD patch has been tested to work with NGPT and submitted to the ORP project. The patch is available here:
http://www.sharma-home.net/~adsharma/projects/orp/orp-freebsd-1.0.5.patch.txt.gz
There are some issues to be ironed out to make it work with FreeBSD's default (user level) pthread implementation.
OpenPackages intends to create a software packaging system that will allow third-party programs to be installed, without operating system dependent changes, on as many platforms as are feasible. OpenPackages was originally based on code from the BSD ports systems, and has been improved and extended by developers of many heritages.
The OpenPackages Project is pleased to release the Milestone 2 codebase. This release contains a working package building system and a single test package. OP currently is known to build on certain instances of the following operating systems: FreeBSD, HP/UX, IRIX, Linux (Debian, Red Hat, Suse, Mandrake, TurboLinux, Caldera, etc.), NetBSD, OpenBSD, Solaris
(First report)
Large cleanup and extension of FreeBSD PAM modules. All modules are to be documented, consistent in style (style(9) used) and as complete as possible WRT functionality. Mostly done.
We now have the rudiments of device support. We have a nexus driver for OpenFirmware machines, along with support for the Apple UniNorth PCI/AGP host bridge. I'm currently trying to get the USB hardware working so that I can get closer to having a console driver independent of OpenFirmware, then I'll be trying to get the system to get to single-user mode using NFS.
Work has begun, but nothing has yet been committed. The NCP addresses used by ppp have been abstracted and initial support has been added to the filter set for ipv6 addresses. NCP negotiation hasn't yet been started.
Patches have been submitted to get ppp working under HURD, and mostly under Linux. There are GPL copyright problems that need to be addressed.
Making pppoed function in a production environment. Most of the work is complete and committed. Additional work includes adding a -l option where ``-l label'' is shorthand for ``-e exec ppp -direct label'' and discovering why rogue child processes are being left around.
PRFW is a set of hooks which I have integrated into the FreeBSD kernel. This allows modules to easily intercept system calls with less overhead. It also supports per-pid restrictions, which means, one process may not be able to use X function in Y manner, but another process may.
Progress: I was working on this in 4.3-RELEASE, but now I'm merging it into current. I will be submitting a patch to the mailing lists in about a week.
This driver is currently not working well under -current and is undergoing some work at this time. No major design or feature changes are planned. There was some notion of adding TapeAlert support, but HP supports that as a binary product via a user library and it was felt that it'd be more politically prudent to leave it alone.
In the 'smpng' p4 branch there is code to make the ast() function loop to close the race when an AST is triggered while we are handling previously triggered AST's.
In the 'jhb_preemption' p4 branch work is being done to make the kernel fully preemptive. It is reportedly stable on UP x86, but SMP x86 locks up, UP alpha has problems during shutdown and can recurse indefinitely until it exhausts its stack.
We are using a perforce repository for live development work, which can track multiple separate long-lived works-in-progress and collaborate between multiple developers at the same time on the same change set.
FreeBSD-current is being imported into p4 hourly, for easy tracking of the moving -current tree.
I haven't written up a good primer yet, but we're able to open this up to the general developer community. NEWCARD work looks like it will be done here too. Perforce is ideal for tracking this sort of long-lived project without having to resort to passing patches around.
KSE work is now being checked into a kse p4 branch - thanks Julian!
KSE work is focusing on getting the main API changes into the base tree well before 5.0.
mb_alloc is a specialized allocator for mbufs and mbuf clusters. It offers various important advantages over the old mbuf allocator, particularly for MP machines. Additionally, it is designed with the possibility of important future enhancements in mind.
The mb_alloc code has been committed to -CURRENT a month ago and appears to be holding up well. Prior to committing it, preliminary performance measurements were done merely to ensure that it is not significantly worse than the old allocator, even with Giant still in place. Results were promising [http://people.FreeBSD.org/~bmilekic/code/mb_alloc/results.html] - also see jlemon's results (link at the bottom of accompanying text). Since the commit, Matt Jacob has provided useful feedback and bugfixes. Work is now being done to re-enable mbtypes statistics and make appropriate changes to netstat(1) and systat(1).
The sparc64 port has been committed to the FreeBSD repository. As such further development will occur in cvs, rather than as a separately maintained patch set. Significant progress has been made since the last status report, including; support for kernel debugging with ddb, much more complete pmap support, support for context switching and process creation, and filling out of important machine dependent data structures. Thomas Moestl has shown a strong interest in working on the port and is in the process of implementing support for saving and restoring a process's floating point context. I look forward to working with him and any other developers that happen to fall out of the wood works.
The sparc64 loader is functional enough to boot an ELF binary from an UFS filesystem using the existent openfirmware library, which has been revised to work flawlessly on 32-bit and 64-bit architectures. Support for netbooting and modules will be implemented next, followed by a better openfirmware mapping strategy.
This project brings a SYN cache implementation to FreeBSD, in order to make it more robust to DoS attacks. A SYN cookie approach was considered, but ultimately rejected because it does not conform to the TCP protocol. The SYN cache will work with T/TCP, IPV6 and IPSEC, and the size of each cache element is currently is less than 1/5th the size of a normal TCP control block.
It's been a busy month, with a number of relevant news items. Not least important is that NAI Labs was awarded a $1.2M contract from the US Defense Advanced Research Projects Agency (DARPA) to work on a variety of components relevant to the TrustedBSD Project, including support for pluggable security models, and supporting features such as improving the extended attributes implementation, simple crypto support for swap and filesystems, documentation, and much more.
On the features side, progress continues on Mandatory Access Control, object labeling, and improving the consistency of kernel access control mechanisms--in particular, with regard to inter-process authorization and credential management. Work has begun on porting LOMAC, NAI Labs' Low-Watermark Mandatory Access Control scheme, from Linux to FreeBSD, and it has been re-licensed under a BSD license. We hope to have an initial port complete in time for 5.0-RELEASE later this year.
The FreeBSD Project made substantial progress in the month of August, 2001, both on continuing the development of the RELENG_4 line (4.x-STABLE and 4.x-RELEASE), and on 5.0-CURRENT, the main development branch. During this month, the decision was made to push the release of 5.0-CURRENT back so that KSE (support for fine-grained user threads) could be completed in time for the release, rather than postponing that support for 6.0. As such, the lifespan of the RELENG_4 line will be extended, with new features continuing to be backported to that branch. 4.4-RELEASE went into final beta during this month, and will also be available shortly.
This month's edition of the status report has been written with the assistance of Nik Clayton and Chris Costello.
For next month, the submission procedures remain the same: reports should be between one and two paragraphs long, sent by e-mail, and in a format approximately that of this month's submissions (Project, Contact, URL, and text). Reminders will be mailed to the hackers@FreeBSD.org and developers@FreeBSD.org mailing lists at least a week before the deadline; complete submission instructions may be found in those reminders.
-- Robert Watson
2 Gigabit support was integrated on 8/31/2001 (QLogic 2300/2312 cards). Because of the author's shrinking time commitment for FreeBSD, the previously planned "next step" which would have been more complete new CAM Transport integration is now probably just the addition of an FC-IP adjunct (as this can benefit many platforms simultaneously).
A major update to error handling was done on 8/28/2001 which should correct most of the EOM detection problems that have been around for a while. There are several things to fix. The principle thing to fix next is the establishment of a loader(8) mediated device quirks method.
No change since last status. Some discussion amongst all of us occurred, but lack of time and commitment to FreeBSD has meant little has actually been committed to the tree. SMPng work will be left to those who seem to have a notion about what needs to be done.
No new status to report. This driver will be worked on again soon and cleaned up to work better.
Work in adding supporting infrastructure to the kernel for KSE threading support has reached "milestone 2".
Milestone 2 is where the kernel source consistently refers to its resources in terms of per-thread and per-process resources, in the way that it will need to when there are > 1 threads per process, but the LOGICAL changes to such things as the scheduler, and fork and exit, have not yet been made to allow more than one thread to be created. (nor have new threading syscalls been added yet). This is an important milestone as it represents the last point where the kernel has only "mechanical" changes. To go further we must start adding new algorithms and functions.
The kernel for milestone 2 is reliable and has no noticeable performance degradations when compared to a matching -current kernel. (the differences are less than the margin of error, so that sometimes the new kernel actually fractionally beats the unaltered kernel).
We hope that by the time this is published, the KSE patches will have been committed. The Major effect for most developers will be only that the device driver interface requires a 'thread' pointer instead of a Proc pointer in the open, close and ioctl entrypoints.
I'm sure there will be small teething problems but we are not expecting great problems at the commit.
The position of Core Secretary was filled by Alan Clegg <abc@FreeBSD.org> The first core-secretary report should be available the second week in September and will cover the issues discussed by core during August 2001.
Development is continuing; pam_unix has gained the ability to change passwords, login(1) has had PAM made compulsory (and is going to have more PAM-capable features handed over to PAM).
The ATM stack has been tested with a number of FreeBSD machines and a Marconi ATM switch and seems to be quite stable running CLIP. Multi port support for the native ATM API has been implemented but needs some testing.
PRFW is a set of hooks for the FreeBSD kernel. It allows users to insert code into system calls, for such purposes as creating extended security features. Last week, PRFW reached 0.1.0, with many bugfixes and cleaning. I urge anyone who is interested to please visit the site, join the mailing list. Also take a peek at lsm.immunix.org, the Linux hooks. It will be a good contrast.
Work is still progressing to make all of the perl scripts run using perl's 'strict' mode, and to migrate all FreeBSD specific options into the configuration file (CVSROOT/cfg.pm). I'll be looking for help soon to write a guide on how to make use of these scripts for use in your own repository. Anyone interested in helping should contact me at the above email address.
The software has been committed to -current and seems functional. Outstanding issues include dealing with IPV6CP events (linkup & linkdown scripts) and allocating site-local and global addresses (currently, ``iface add'' is the only way to actually use the link).
Status is unchanged since last month. Patches have been submitted to get ppp working under HURD, and mostly under Linux. There are GPL copyright problems that need to be addressed. Many conflicts are expected after the commit of IPv6 support in ppp.
Making pppoed function in a production environment. All known problems have been fixed and committed.
I looked at bringing PPPoA into the base system, but could not because of an overly restrictive distribution license on the Alcatel Speedtouch modem firmware. It has been committed as a port instead and is running live at a FreeBSD Services client site.
The OLDCARD improvements have been completed, except for a few edge cases for older laptops with CL-PD6729/30 chips and some pci bios issues. Some minor work will continue, but after 4.4R is released, only a few remaining bugs will be fixed before the author moves on to greener fields of NEWCARD development.
Targeting 4.4-RELEASE, one team has been translating newly MFC'ed section [125678] manpages. The other team has been updating section 3 since May and one third (1/3) is finished. The port ja-groff is updated to be groff-1.17.2 based, and now it has the same functionality as base system does. The port ja-man is updated to have the search capability under an architecture subdirectory, as base system does. The doc/ja_JP.eucJP/man hierarchy update (adding architecture subdirectories) is planned after 4.4-RELEASE.
Basic footbridge support is now functional and the kernel is now able to probe the pci bus. Access primitives for the bus are still missing so I can't attach any drivers yet.
The syncache implementation is completed, and currently under testing and review. The code should be committed to -current in the near future, and a patchset for -stable made available.
State information for TCP connections is primarily kept in the TCP/IP control blocks in the kernel. Not all of the TCP states make use of the entire structure, and significant memory savings can be had by using a cut-down version of the state in some cases. The first phase of this project will address connections that are in the TIME_WAIT state by moving them into a smaller structure.
This project has completed the initial research and rough design phases, with actual code development starting immediately.
For 5.0, the goal is for the network stack to run without the Giant lock. Initial development in this area may focus on partitioning the code and data structures into distinct areas of responsibilities. A first pass of locking may involve using a several smaller mini-giant code locks in order to reduce the problem to a manageable size.
Progress for this month includes the creation of a perforce repository to officially track the locking changes, and the initial submission of locks for the &ifnet list. Some code cleanup has also been done to the main tree in order to better support future locking additions.
Currently, all network devices (fxp0, lo0, etc) exist in their own namespace, and are accessed through a socket interface. This project creates device nodes in /dev for network devices, and allows control and access in that fashion.
This is experimental work, and suggestions for APIs and functionality are strongly encouraged and welcomed. In is not clear whether it will be possible (or desirable) to provide the exact same set of operations that can be done through the socket interface.
Benefits of approach include the fact that a kqueue filter can be attached to a network device for monitoring purposes. Initial code exists to send a kq event whenever the network link status changes. Other benefits may include better access control by using filesystem ACLs to control access to the device.
RELNOTESng, the DocBook-ified set of release documentation files, has been merged to the RELENG_4 branch. 4.4-RELEASE will be the first release of FreeBSD with the new-style release notes, hardware list, etc. Some of these documents are being translated by the Japanese and Russian translation teams.
Snapshots of RELNOTESng for CURRENT and 4-STABLE in HTML, text, and PDF are available at the above URL and are updated irregularly but frequently. Dima Dorfman <dd@FreeBSD.org> and Nik Clayton <nik@FreeBSD.org> have been working to have automatically-generated snapshots on the main FreeBSD web site.
On my TODO list: 1) Resynchronize the FreeBSD installation document with the installation chapter in the Handbook. 2) Update the hardware lists (with particular emphasis on PCCARD and USB devices). 3) Update the infrastructure to allow the architecture-dependent parts of RELNOTESng to scale to more hardware platforms.
Sparc64 development is still continuing rapidly and we're making some excellent progress. Of note, some problems with the way the pmap module implements copy-on-write mappings have been fixed and fork() now works as expected, support for signals has been added, and the port has been updated for KSE in the perforce repository. Thomas Moestl has begun work on pci bus support, and a basic nexus bus for sparc64 has been written. The driver for the Sun `Psycho' and `Sabre' UPA-to-PCI bridges and associated code has been ported from NetBSD (the Sabre is the on-chip version found in the UltraSparc IIi and IIe). PCI configuration, I/O and memory space accesses do already work, as well as interrupt assignment and delivery for devices attached directly to the bridge, and the first PCI device drivers can attach and seem to work mostly. Interrupt routing and busdma support still need much work.
The Handbook has been the main focus of activity this month. Due to go to the printers on the 15th a vast amount of new content has been submitted and committed. This includes a complete rewrite of the "Installing FreeBSD", which massively expands the amount of information available to people new to FreeBSD. It even includes screenshots.
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/install.html
Comments, and contributions are, of course, welcome.
FreeBSD's IP Multicast Routing support was recently updated in several ways. One big change is that it's now able to be loaded as a KLD instead of statically compiled into the kernel; this is especially useful for experimentation or updating of an existing system. It also now coexists nicely with the kernel IP encapsulation infrastructure, so that multicast tunnels can better coexist with MobileIP, certain IPSec tunnels and generic IPv4-in-IPv4 tunnels.
The allocator appears to be stable. Mbtypes statistics have been re-activated thanks, in part, to Jiangyi Liu <jyliu@163.net> although the diff has not yet been committed (I'm just in the process of cleaning it up a little and final testing). More work to come: cleanups, follow TODO from the original commit, and perhaps an eventual generalization of the allocator for various network-related allocations (in a more distant future).
After two months of little progress, RAIDframe work is gearing up again. The port to -stable has some known bugs but is fairly stable. The port to -current was recently completed and patches will be released soon. RAIDframe is a multi-platform RAID subsystem designed at CMU. This is a port of the NetBSD version by Greg Oster.
The aac driver has been given a lot of attention lately and is now nearly feature complete. Changes include crashdump support, correct handling of controller initiated commands, and more complete management interface support. The Linux RAID management tool available from Dell and HP now fully works; a FreeBSD native version of the tool is also in the works. These changes have been checked into -current, and will appear in -stable once 4.4 has been released.
We are making some progress, we are now down to 2170 open PR's down from an all time high of 3270 just 3 months ago. The aim is still to get rid of all the dead-wood in the PR database so only relevant PRs in the database. A big thanks from me to the people who have made this happen!
Support for cloning vlan devices via ifconfig has been committed to -current and will be MFC'd after further testing. Additionally, Maksim Yevmenkin submitted code to allow cloning of tap and vmnet devices on devfs systems. Code for faith and stf should be committed shortly.
Current status is that the ia64 kernel builds and runs in a simulator environment up to single user mode and has been tested lightly in that environment. My current focus is on completing the ia64 loader so that I can start to get kernels working on the real hardware. The loader is coming along well and I expect to be able to load kernels (but not necessary execute them) soon.
I have access to the libh CVS repo again and am testing a new, OBJDIR capable build structure at the moment. Done that, I'm going to continue testing the package library and implement the missing functionality. Currently, import of libh into the base system is under discussion (arch mailinglist). Now that 5.0-RELEASE has been shifted, I want 5.0 ship with a libh installer and package system. We can really need people who are good in C++, are able to understand what the current implementation does and also feel that working on libh is fun and thus are willing to help.
Getting GNOME Fifth-Toe metaport ready for 4.4-RELEASE was the main focus of activity this month. In the process many components were updated, many bugs were tracked down and solved, which allowed to make this 97-component meta-package building and working properly.
Next month the project will be focused on organizing work of the FreeBSD GNOME Team as well as on attempts to increase amount of people participating in the team (anybody who is willing to participate is welcome to drop a note to gnome@FreeBSD with a short explanation of how he/she could help).
NVIDIA Corporation releases Linux drivers by using a combination of binary object files and source (under a constrictive license). The FreeBSD NVIDIA driver project aimed to completely replace the source component of the driver using code targeting FreeBSD 4.3 and released under the BSD license. The binary module provided is supposedly the same module used on Windows, BeOS, and OS/2, so it should be portable between different i80x86 based OS's.
The project is currently on indefinite hold. Our contact at NVIDIA seemed enthusiastic about the project, and was fairly quick about returning email, but when we discovered issues that prevented porting without changes to the binary component or error codes we needed deciphered, Nick (the contact) said he'd look into it and never got back. The first major problem was the ioctl interface, the NVIDIA driver passes a pointer and depends on the kernel side to copyout the right amount, where FreeBSD expect the parameters to be correct and the copyout is performed by the subsystem. This was worked around using Dave Rufinos "ioctl tunnel" idea. After that, we found that X refused to load and traced it down to an ioctl defined in the binary component erroring. We cannot tell what that ioctl is, were told that we could not sign an NDA for source to that component, and have been waiting a month for Nick to "look into it". Therefore progress is impossible (without breaking the license) and we believe that the flaws make the driver unportable to any *nix other than Linux.
The FreeBSD release engineering process for FreeBSD 4.4 started to ramp up around August 1st when the "code slush" took affect. During this time all commits to the RELENG_4 branch were reviewed by re@FreeBSD.org (over 250 code snippets had to be reviewed). After the first release candidate on August 15th, all submissions were scrutinized under a more strict potential risk vs benefit curve. The best way to help get involved with the release engineering process is to simply follow the low volume freebsd-qa mailing list, help out with the neverending supply of PRs related to our installation tools (sysinstall), or to work on a possible next-generation replacement for our installation technology, such as the libh or OpenPackages projects.
Many companies donated equipment, network access, or paychecks to finance these activities. Including Compaq, Yahoo!, Wind River Systems, and many more.
In mid March, 2001, Tim Newsham of Guardent identified an attack possible against the initial sequence number generation scheme of FreeBSD (and other OSes.) In order to guard against this threat, a randomized sequence number generation scheme was ported over from OpenBSD and included in 4.3-release. Unfortunately, non-monotonic generation was found to cause major problems with applications which initiate continuous, rapid connections to a single host.
In order to restore proper operation under such circumstances while still providing strong resistance against sequence number prediction, FreeBSD 4.4 uses the algorithm specified in RFC 1948. This algorithm hashes together host and port information with a piece of secret data to generate a unique sequence number space for each connection. As a result, outgoing initial sequence numbers are again monotonic, but also unguessable by an attacker.
The port of LOMAC to FreeBSD is progressing well, and already has a very high level of stability (no known outstanding bugs!). Aspects which have already been implemented include a stacking filesystem overlay with fully-functional access controls (for files and directories) based on path names, access controls for sending signals, and file-backed-memory revocation for processes.
Updates to things from last month:
New stuff since last month:
Most of the work this month has focused on development of the native JDK 1.3.1 patchset. The 3rd patchset is out and has been accompanied with the creation of a FreeBSD "port". This has allowed early adopters much easier access to the code and naturally resulted in a number of bugs being found. Development work has mostly focused on fixing these problems and the project is now set to release fourth patchset over the weekend, which should see the JDK in a reasonably usable state. One of the big challenges left is producing a working HotSpot JVM, which looks like it will require some heavy hacking.
We also welcome OpenBSD's Heikki Korpela to the porting team :)
As part of some ongoing development activity, the floppy driver (fdc(4)) enjoyed some overhaul in the past which is part of an ongoing process. Automatic density selection will come next, something i meant to implement for years now. As part of that, the entire density selection stuff has been rewritten. 2.88 MB floppies are on the wishlist as well, but I need a working 2.88 drive before attempting to implement that.
sppp(4) should be merged with the ISDN4BSD offspring variant. This will merge some features and bugfixes from the i4b branch (like VJ compression), and eventually end up in a single sppp(4) in the tree. While being at that, incorporating many changes and bugfixes from NetBSD is considered as well.
The KAME project (http://www.kame.net/) has merged its IPv6 and IPsec implementation as of July 2001 to FreeBSD CURRENT and STABLE, in cooperation with some contributors of the project. The latest code includes a number of bug fixes, has been fully tested in FreeBSD STABLE, and will appear in FreeBSD 4.4 RELEASE. Thus, the new RELEASE version will be quite stable in terms of IPv6 and IPsec.
The project has assigned a talented guy to be responsible for merge from KAME to FreeBSD, so future merge efforts will be smoother.
The TrustedBSD project continues to move ahead, with progress made in the ACL, Capability, and MAC implementations. In addition, support from DARPA is permitting new work to improve the extended attribute code, improve security abstractions, and work on security documentation. Due to the push-back of the FreeBSD 5.0 release, it should now be possible to include a complete MAC implementation in that release. Specific status reports appear for components where substantial progress is being made.
Capabilities support is currently being committed to the base FreeBSD tree--userland libraries are now fully committed, and kernel infrastructure is being integrated.
Planning for BSDCon Europe is going well. We're still accepting proposals for talks but the schedule is starting to fill up so we may not be for much longer.
An update of the site that includes accommodation information, a preliminary schedule, a list of speakers and an online payment page will be launched on Wednesday 19 September.
The fee will be £150 for individuals and £250 for corporations. The individual pricing is valid only until the end of September, the price will rise to £200 for October and late registrations in November will be £250.
The updated website will include a list of sponsorship options, we're still looking for more sponsorship.
In the month of September, the FreeBSD Project continued its investment in long-term projects, including continuing work on a fine-grained SMP implementation, support for Kernel Schedulable Entities (KSE) supporting highly efficient threading, and broadening support for modern hardware platforms, including Intel's new IA64 architecture, UltraSparc, and PowerPC. Additional focus was placed on the release process, including work on the release notes infrastructure, support for DVD releases, and work on a binary updating tool.
Due to the delay in getting the September report out the door, the November status report will also cover October. During the month of November, we look forward to BSDCon Europe, the first such event outside the continental United States. The USENIX conference paper submission deadlines are also in November, and FreeBSD users and developers are encouraged to submit to the general and FREENIX tracks. Please see www.usenix.org for more information.
PRFW provides hooks in the FreeBSD kernel, allowing users to insert their own checks in system calls and various kernel functions. PRFW is nearing 0.5, which will incorporate numerous structural changes such as, much faster per-process hooks, kernel function hooks, plus, a new way of adding hooks which would enable users to reference hooks by a string.
The build process is now creating four different versions of the libs, which include support for TVision, Qt, both or none. I created some first packages from existing ports and installed those libh packages on my system only using libh's tools, including registering all the files in the package database, recording their checksums etc. Patches to the disk editor have been submitted, which include functionality to write the changes in the fdisk part and initial support for a disk label editor. We'll soon have a new committer.
FreeBSD 4.4-RELEASE was the first release of FreeBSD with its new-style release documentation. Both English and Japanese versions of these documents were created. Regularly-built snapshots of -CURRENT and 4-STABLE release documentation are now available on the Web site, but they require a little HTML infrastructure to make them viewer-friendly. I intend to continue updating my snapshot site at the URL above, at least for a little while.
Call for help: The hardware compatibility lists need to be updated in the areas of the Alpha architecture, USB devices, and PCCARD devices. I'm looking for volunteers to help; interested parties should contact me at the email address above. DocBook experience is not required; familiarity with the hardware above would be very helpful.
Bug fixing and move to -STABLE of 2Gb support.
Quite a lot of cleanup of this driver. Bug fixes and some performance enhancements. However, this driver is likely to be removed shortly and replaced by one from Intel itself.
As you know, in march 2001 the version 2.3 of TIRPC has been committed together with many userland changes. Alfred Perlstein and Ian Dowse have helped me a lot with the porting effort and if I had problems with understanding the code.
Most bugs are now fixed, some remaining areas to fix are secure RPC (keyserv) and unix domain support. I've patches for these area available. Ian Dowse fixed a lot of outstanding bugs in the rpcbind binary itself. Thank you Ian !
The plan is now to migrate slowly towards TIRPC 2.8, which is threadsafe for the server- and clientside. One first patch I've made available on my URL. TIRPC 2.8 is licensed under the "Sun Standards License Version 1.0" and we have to add some license lines and the license itself to all modified files.
A example is timed_clnt_create.diff which can be found on the homepage.
The project has gained a mailing list, freebsd-binup@FreeBSD.org - and the source tree has been moved into the projects/ directory in the FreeBSD CVS repository. Current work is focusing on extending the FreeBSD package framework, and the client library should be rewritten and completed by the end of the year.
TODO: make the projects/ hierarchy into a cvsup distribution and add it to cvs-all. Then update distrib.self.
Status is unchanged since last month. Patches have been submitted to get ppp working under HURD, and mostly under Linux. There are GPL copyright problems that need to be addressed. Many conflicts are expected after the commit of IPv6 support in ppp.
The software has been committed to -current and seems functional. Outstanding issues include dealing with IPV6CP events (linkup & linkdown scripts) and allocating site-local and global addresses (currently, ``iface add'' is the only way to actually use the link). A bug exists in -stable (running the not-yet-MFC'd ppp code) whereby routing entries are disappearing after a time (around 12 or 24 hours). No further details are yet available.
A two disc set has been mastered and sent for pressing. There are a few surprises with this release - details will be given in the official announcement (at BSDConEurope).
ATM-Forum LAN-emulation version 2.0 without support for QoS has been implemented and tested. The ILMI daemon has been modularized into a general mini-SNMP daemon, an ILMI module and a not yet finished IPOA (IP over ATM) module.
We have finished updating section [125678] manpages to 4.4-RELEASE based, 1 week after 4.4-RELEASE is announced. To finish this update, OKAZAKI Tetsurou has imported Ex/Rv macro support on ja-groff-1.17.2_1. SUZUKI Koichi did most Ex/Rv changes on Japanese manpages. He also find some issues of these macro usage on some original manpages and filed a PR. For post-4.4-RELEASE, now we target 4.5-RELEASE. Section 3 update is also in progress.
We've made some good progress now, and the new nmount(2) syscall is nearly finished. There is still some work to do to have a working kernel_mount() and to convert all filesystems to use this new API for their VFS_MOUNT() functions.
I am pleased to announce that as of 1 AM Friday October 19th, the sparc64 port boots to single user mode. A few binaries from the base system have been built and verified to work properly. Much of this work is still in review for commit, but will be integrated into the cvs tree as soon as possible. EBus support has been ported from NetBSD, and ISA support has been written. The PCI host bridge code has stabilized, and busdma seems to work correctly now. The sio driver has had EBus support added, and the ATA driver has been modified so that it works on big-endian systems and uses the busdma API. With these changes, a root file system can now be successfully mounted from ATA disks on sparc64, even in DMA mode. The gem driver, which supports Sun GEM and ERI and Apple GMAC and GMAC2 ethernet adaptor, has been ported from NetBSD but has not yet had sufficient testing.
No new status to report, the code is still waiting to be committed. It is likely that this code will be expanded to include syn cookies as a further fallback mechanism.
Development on this project has been slowed, pending the commit of the syncache code, as this builds on part of that work.
Not much progress has been made this month, with other projects occupying most of my time. However, reviewing all the code and data structures had a side benefit; a hash table for inet addresses has been added. This will significantly speed up interface address lookups in the case where there are a larger number of interface aliases.
Currently, a single device may act as a console at any time, which requires the user to choose the console device at boot time. With the upcoming network console support, it is desirable to allow multiple console devices which behave identically, and to alter consoles while the kernel is running.
The code is completed, and needs some final polishing to clean up the rough edges. Console output can be sent to both syscons and sio, (as well as the network) and when in ddb, input can be taken from any input source. A small control program allows adding and removing consoles on the fly.
This project's goal is to add low level network functionality to FreeBSD. The initial target is to make a network console available for remote debugging with ddb or gdb. A secondary target is to utilize the code to perform network crash dumps. The design assumes that the network card and driver are working, but does not rely on other parts of the kernel.
Initial development has been fairly rapid, and a minimal TCP/IP stack has been written. It is currently possible to telnet to a machine which is at the ddb> prompt and interact with the debugger.
Network devices now support aliases in the form of /dev/netN, where N is the interface index. Devices may be wired down to a specific index number by entries in /boot/device.hints of either:
hint.net.<ifindex>.dev="devname" hint.net.<ifindex>.ether="ethernet address"
Additionally, ifconfig has been updated so that it will accept the alias name when configuring a device.
The gx driver has finally been committed to the tree. The driver provides support for the Intel PRO/1000 cards, both fiber and copper variants. The driver supports VLAN tagging and TCP/IP checksum offload.
In the last month, not a lot has happened other than settling in of the big August commit. Largely due to me having a sudden increased workload at work, and a need for increased time to be spent elsewhere. However some design work has proceeded. The API has firmed up somewhat and several people have been reading through what has been done already in order to be able to help in the next phase.
Milestone 3 will be to have the ability to generate and remove multiple threads/KSEs per process. Milestone 3 will NOT require that doing so will be safe. (especially in SMP systems), i.e. locking issues will not be fully addressed, so while some testing will be possible, it will not be possible to actually run in this mode with any load.
This will require allocators and destructors for the new structures. Creation of the syscalls. Generation of an accurate written API for the userland crew. Writing of the upcall launch code. Production of a userland test program (not a full thread scheduler). Resolution of some of the more glaring incompatibilities (e.g. the scheduler) in a backwards compatible manner. (i.e. if there are no multi threaded processes on a system it should behave the same as now (and be as reliable)).
Criteria for knowing when we have reached Milestone 3 is the ability for a simple process on an unloaded system to perform a series of blocking syscalls reliably. e.g. open 2 sockets, and send data on one, after having done a read on another, and then 'respond' in like manner..
There have been a few major successes in the PowerPC port this month. Mark Peek has succeeded in getting the FreeBSD/PowerPC kernel cross compiled on FreeBSD and booting under the PSIM simulator (now in /usr/ports/emulators/psim-freebsd). I have succeeded in getting the FreeBSD loader to load and execute kernels using the OpenFirmware found on Apple Macintosh hardware. Mark is now working on completing some of the startup and pmap code, while I am taking advantage of the simulator to work on some interrupt and device issues.
The project has moved forward on JDK 1.3.1 development this month, with the release of two more patchsets. The team is reasonably confident that the latest patchset is a stable release of the core JDK 1.3.1 tools and classes, when the default "green" threads subsystem is used. This is mostly thanks to hard work by Fuyuhiko Maruyama to stabilize and fix the code. Bill Huey has also been progressing with his work on the "native" threads subsystem, although this hasn't yet reached the stability of "green" threads. Another (arguably the) major highlight of the latest patchset was the integration of NetBSD support by Scott Bartram and Alistair Crooks (the latter of NetBSD packages fame). Hopefully OpenBSD support will follow, making it truly a united BSD Java Project.
This group is for discussion about the startup scripts in FreeBSD, primarily the scripts in /etc/rc*. Primary focus will be on improvements and importation of NetBSD's excellent work on this topic.
Alright folks, I finally got off my butt last night and put together a roadmap for the migration to the new rc.d init scripts that were imported from NetBSD a long time ago and just sat in the tree.
M1 (Patch included)
Setup infrastructure
Make rcorder compile
Hook rc.subr into the distribution (and mergemaster)
Hook rcorder into the world
Add toggle in rc.conf to switch between rc_ng and current boot
scripts
M2
Get FreeBSD to boot with the new boot scripts
Rewrite the /etc/rc.d scripts to work with FreeBSD
M3
Add some FreeBSD specific support into rc.subr
M4
Add true dependency checking to the infrastructure so that
starting nfsd will start mountd and rpcbind
add support into rc.subr
Add dependencies into rc.d scripts
I'd like a couple of people to take a look at this and then I'll submit a pr for it if there aren't too many objections. I'm expecting M2 to run into quite a bikeshed, but hey, I got my nice shiny asbestos back from the cleaners.
The FreeBSD C99/POSIX Conformance Project aims to implement all requirements of the C99 Standard and the latest 1003.1-200x POSIX draft (currently Draft 7). In cases where aspects of the standard cannot be followed, those aspects will be documented in the c99(7) or posix(7) manuals. It is also an aim of this project to implement regression tests to ensure correctness whenever possible.
Patches that implement the <stdint.h> and <inttypes.h> headers, and modifications to printf(3) have been developed and will be committed shortly. They will allow us to use some of the new types C99 introduces, such as intmax_t and the printf(3) conversion specifier "%j".
Some progress has been made on the proc locking this month. Also, a new LOCK_DEBUG macro was defined to allow some locking infrastructure to be more efficient. Kernels now only include the filenames of files calling mutex, sx, or semaphore lock operations if the filenames are needed. Also, mutex operations are no longer inlined if any debugging options are turned on. The ucred API was also overhauled to be more locking friendly. A group has also started investigating the tty subsystem to design and possibly implement a locking strategy.
This months report covers activity during the second half of October, and the month of November. During these months, substantial work was performed to improve system performance and stability, in particular addressing concerns regarding regressions in network performance for the TCP protocol, and via the introduction of polled network device driver support. Work continues on long-term architectural projects for 5.0, including KSEs, NEWCARD, and TrustedBSD, as well as the cleaning up of long-standing problems in FreeBSD, such as PAM integration. Administrative changes are also documented, including work to redefine and formalize the release engineering process, and the approval of a new portmgr group which will administer the ports collection.
FreeBSD users and developers are strongly encouraged to attend the USENIX BSD Conference in February of next year; it is expected that this will be a useful forum both for learning about FreeBSD and on-going work, as well as providing an opportunity for developers to work more closely and act as a vehicle for discussion and round-the-clock hacking. More information is available at the USENIX web site.
Robert Watson
A number of serious TCP bugs effecting throughput snuck into the system over the last few releases and have finally been fixed. TCP performance should be greatly improved for a number of cases, including TCP/NFS.
The wx driver is desupported and removed from -current. No further support for wx in -stable is planned. Newer and better drivers are now in the tree.
Ongoing bug fixes. Work is underway, to be integrated shortly, that makes the cross platform endian support easier and will prepare the FreeBSD version for eventual sparc64 and PowerPC usage.
Currently, we are exploring a variety of strategies to learn about the implementation and performance issues in order to have a solid design. One of our main goals will be to use a standardized interface to the system, whether it be POSIX.1e, or another of the other standards, because as they say "Standards are great because you have so many to choose from." Hopefully within the next month or so, we will populate the perforce TrustedBSD tree with an agreed upon framework that is ready for serious final work.
On the code side, a number of libpam bugs have been fixed; a new PAM module, pam_self(8) , has been written; and preparations have been made for the transition from /etc/pam.conf to /etc/pam.d .
On the documentation side, new manual pages have been written for pam_ssh(8) , pam_get_item(3) and pam_set_item(3) , and work has started on a longer article about PAM which is expected to be finished by the end of the year.
A lot of work still remains to be done to integrate PAM more tightly with the FreeBSD base system—particularly the passwd(1) , chpass(1) etc. utilities—and ports collection.
Presently re-style(9)ing mbuf code with the help of Bruce (bde). The next larger step is approaching: to better performance, as initially planned, not have reference counters for clusters allocated separately via malloc(9). Rather, use some of the [unused] space at the end of each cluster as a counter; since this space is totally unused and since ref. counter <--> mbuf cluster is a one-to-one relationship, this is most convenient.
Release engineering activities for FreeBSD 4.5 have begun. An overview of the entire process has been added to the FreeBSD web site, along with a specific schedule for 4.5. The code freeze is scheduled to start on December 20. The team responsible for responding to MFC requests sent to re@FreeBSD.org for this release is: Murray Stokely, Robert Watson, and John Baldwin. Some of our many goals for this release include closing more installation-related problem reports, being more conservative with our approval of changes during the code freeze, and continuing to document the entire process. For suggestions or questions about FreeBSD 4.5 release activities, please subscribe to the public freebsd-qa@FreeBSD.org mailing list.
Work is (slowly) progressing on converting the web site to use pages marked up in a simple XML schema, and then generating HTML and other output formats using XSLT style sheets. The work so far can be tested by doing "cvs checkout -r XML_XSL_XP www" and then "cd www/en; make index.html". Take a look at index.page in the same directory to see the source XML. The CVS logs for index.page contain detailed instructions explaining how index.page was generated from its earlier form.
The FreeBSD in Bulgarian project aims to bring a more comfortable working environment to Bulgarian users of the FreeBSD OS. This includes, but is not limited to, font, keymap and locale support, translation of the FreeBSD documentation into Bulgarian, local user groups and various forms of on-line help channels and discussion forums to help Bulgarians adopt and use FreeBSD.
Bulgarian locale support has been committed to FreeBSD 5.0-CURRENT (and later merged into 4.x-STABLE on December 10th). A local CVS repository for the translation of the FreeBSD documentation into Bulgarian has been created.
There is now some code ready for the new mount API, which has to be reviewed and tested. If it is adopted, we will probably start converting all the filesystems, as well as other code in the kernel, to make them use it. If you want to play with it, the patch is available at the above URL.
Support for VLAN cloning has been merged from current and will ship with 4.5-RELEASE. Additionally, new rc.conf support for cloning interfaces at boot has been MFD'd. Work is ongoing to MFC stf and faith cloning as well as adding cloning for ppp devices and enhancing VLAN modularity.
This work uses a mixed interrupt-polling architecture to handle network device drivers, giving the system substantial improvements in terms of stability and robustness to overloads, as well as the ability to control the sharing of CPU between network-related kernel processing and other user/kernel tasks. Last not least, you might even see a moderate (up to 20-30%, machine dependent) performance improvement.
I've been working on making the Hardware Notes less i386-centric. This will be especially important for -CURRENT as the ia64 and sparc ports reach maturity; most of this work should be completed in time to be MFC-ed for FreeBSD 4.5-RELEASE. I encourage any interested parties to review the release documentation and send me comments or patches.
The port of the driver is around 90% feature complete. AGP support and "Registry" support via sysctl need to be finished/implemented. The NVIDIA guys are working on a build of the X11 libs and extensions for FreeBSD; once this is done hardware accelerated direct rendering should work. The previous version this driver is no longer available. I'm planning on making a snapshot of my code once I chase out a few more bugs.
Please note that development is taking place under -CURRENT right now; a port to -STABLE will be available at some later time.
jp.FreeBSD.org daily SNAPSHOTs project is yet another snapshots server that provides latest 4-stable and 5-current distribution. You also find installable ISO image, live filesystem, HTMLed source code with search engine, and more; please check project webpage for more details.
Modest gains have been made on the UDF filesystem since the last report. Reading of files from DVD-ROM now works (and is fast, according to some reports), and there is preliminary support for reading from CD-RW media. The CD-RW support has only been tested against CD's created with Adaptec/ Roxio DirectCD, and much, much more testing is needed. Once this support is solid, I plan to check it into the tree and start work on making the filesystem writable.
Not much to report. A number of minor bugs in OLDCARD have been corrected. A larger number of machines now work. Additional work on ToPIC support has been committed, but continued lack of a suitable ToPIC machine has left the author unable to do much work. A few stubborn machines still need to be supported (the author has an example of one such machine, so there is hope for it being fixed. Some pci related issues remain for both OLDCARD and NEWCARD.
NEWCARD work is ramping up, while OLDCARD work is ramping down. A number of things remain to be done for NEWCARD, including suspend/ resume support, generic device arrival/removal daemon and hopefully automatic loading of drivers. A number of current pccard drivers still need to be converted to NEWBUS. Several Chipset issues remain, as does the merging of isa pccard bridge code with the pccbb code.
This project is now finally underway, thanks to DARPA and NAI getting a sponsorship lined up. The infrastructure code and data structures are currently taking form inside a userland simulation harness.
Targeting 4.5-RELEASE, we continued to revising doc/ja_JP.eucJP/man/man[1256789] to catch up with RELENG_4. Section 3 updating has 45% finished.
A FreeBSD -CURRENT snapshot with LOMAC is currently being prepared, with aid of Perforce on the "green_lomac" branch. Very soon there should be a working demonstration installation CD of FreeBSD with LOMAC, including the ability to enable LOMAC in rc.conf with sysinstall, being a legitimate "out-of-the-box" FreeBSD experience. Actual release build is pending debugging issues with program start-up (especially xdm).
Work is underways to support failing mirror disks better and handle hotswapping in a new replacement disk and have it rebuild automagically.
Support for the Promise TX4 is now working in my lab, seems they did the PCI-PCI bridging in the not so obvious way.
Plans are in the works to backport the -current ATA driver to -stable with hotswap and the works. Now that -current is delayed I'm working on ways to give me time to get this done, since I've had lots of requests lately and we really can't let down our customers :).
SMART support is being worked on, but no timelines yet.
Although not strictly ATA, Promise has equipped me with a couple SuperTrak sx6000 RAID controllers, they take 6 ATA disks and does RAID0-5 in hardware. I have done a driver (its an I2O device) for both -current and -stable and it works beautifully with hotswap the works. It will enter the tree when it is more mature, and I have an agreement with Promise on how we handle userland control util etc. BTW it seems it can also be used as a normal 6 channel PCI ATA controller, a bit on the expensive side maybe...
Extending camcontrol's page definition file format to include both modepage and logpage definitions; adding support to camcontrol to query and reset log page parameters. Consideration is being made to possibly include support for diagnostic and vital product data pages, but that is outside the current project scope. New page definition file format includes capability to conditionally include page definitions based on SCSI INQUIRY results allowing vendor-specific pages to be described also. Approximately 80% complete.
Work on the FreeBSD C99 & POSIX Conformance Project is progressing nicely. Since the last status report, two new headers have been added [<stdint.h> and <inttypes.h>], several new functions implemented [atoll(3), imaxabs(3), imaxdiv(3), llabs(3), lldiv(3), strerror_r(3), strtoimax(3), and strtoumax(3)], and changes to assert(3) and printf(3) were made to support C99. More printf(3) changes are in the works to support the remaining C99 and POSIX requirements. Additionally, research was done into our POSIX Utility conformance and a list of tasks was derived from that research.
Several other interesting events occurred during November and the beginning of December. The project mailing list was moved to the FreeBSD.org domain, and is now available at standards@FreeBSD.org. On December 6, 2001, the IEEE Standards Board approved the Austin Group Specification as IEEE Std 1003.1-2001, thus making the work we're doing ever more important.
This group is for discussion about the startup scripts in FreeBSD, primarily the scripts in /etc/rc*. Primary focus will be on improvements and importation of NetBSD's excellent work on this topic.
<-- from Gordon Tetlow's ranting -->Due to personal commitments by the folks working on this project we have been unable to spend much time porting the rc.d infrastructure into the FreeBSD boot framework.
Currently, the system will boot (with a little fudging) just before network utilization. There are patches floating around for this (see the -arch list from September).
I have been working behind the scenes on design rather than programming for this last month. I have been working however in the p4 tree to make the system run with the thread structure NOT a part of the proc structure (a prerequisite for threading)
After a discussion with the Core Team about our status regarding the ports collection, we heard from them that they'd decided to recognize us as the final authority for approving ports committers. We've spent the last few weeks working on our ports build cluster (see the link) and trying to find ways to improve it for the ports development community. We've also handled a few minor issues in the ports collection.
The TrustedBSD Project continued focusing development efforts on fine-grained Capabilities and Mandatory Access Control this month. Kernel support for capabilities is essentially complete, and efforts are underway to adapt userland applications to use Capabilities. The login process has been updated to allow users to run with additional privilege based on /etc/capabilities. The MAC implementation work has also been active, with improved support for the labeling of IPC objects, including better integration into the network stack. Both development trees have been updated to work with recent KSE-related developments, as well as exist more happily in a fine-grained SMP kernel. Initial audit-related work appears in a separate entry.
Development of TrustedBSD source code was moved to the FreeBSD Perforce repository, permitting better source code management. As such, the TrustedBSD development trees will now be available via cvsup.
October ended up being a bit busier than November for SMPng. During October, Peter Wemm finally finished the ambitious task of unwinding all the macros in NFS and splitting it up into two halves: client and server. Andrew Reiter also submitted some code to add locks to taskqueues, and the folks working on the TTY subsystem designed the locking strategy they will be using. Per-thread ucred references were also added for user traps and syscalls. Once the necessary locking on the process ucred references is committed, this will allow kernel code to access the credentials of the current thread without needing locks while also ensuring that a thread has constant credentials for the lifetime of a syscall. November only saw a few small bug fixes unfortunately, but December is already shaping up to be a very active month, so next month's report should be a bit more interesting.
In non-coding news, the website for the SMPng project has moved from its old location to the new location above. Also, I have completed a paper I am presenting for BSDCon regarding the SMPng project. The paper will be available in the conference proceedings and will be available online after the conference as well.
This bi-monthly report covers development activities on the FreeBSD Project for December 2001 and January 2002. A variety of accomplishments have been made over the last couple of months, including strong progress relating to the KSE project, which brings Scheduler Activations to the FreeBSD kernel, as well as less visible infrastructure projects such as improvements to the mount interface, PAM integration work, and translation efforts. Shortly following the deadline for this status report, the BSD Conference and FreeBSD Developer Summit were held, and will be covered in the next bi-monthly report at the end of March. Plans are already under way for the USENIX Annual Technical Conference in Monterey, CA, later this year, and all and sundry are encouraged to attend to get further insight in FreeBSD development.
Robert Watson
I've been working to integrate recent improvements in the NetBSD usb stack to FreeBSD -current. Both NetBSD and OpenBSD currently share the same source, as FreeBSD did too at once point before it diverged. The goal is to get back to that state, but there are many improvements on both sides that need to be merged before this is complete.
I'm currently looking for someone to help maintain usb in -stable. Please let me know if you're interested.
Patches for cp(1), ls(1), and mv(1) to bring in POSIX.1e-compliant Access Control List support have been updated to patch against builds of -CURRENT. Other system utilities are currently being evaluated for ACL support including install(1) (patch available) and mtree(8). Work is in progress to verify the native getfacl(1), setfacl(1), and other utilities build and work correctly on other ACL-enabled systems (e.g. Linux w/ACL patches) and to help verify POSIX-compliance of the continuing TrustedBSD work along with other systems. Finally, experimental Perl and PHP modules are available allowing limited access to native ACLs for languages other than C.
The project is making progress. The goal is to design and implement Host Controller Interface (HCI) and Link Layer Control and Adaptation Protocol (L2CAP) layers using Netgraph framework. More distant goal is to write support for Service Discovery Protocol (SDP) and RFCOMM protocol (Serial port emulation over Bluetooth link) . All information was obtained from Bluetooth Specification Book v1.1.
Project status: In progress. 1) Design: mostly complete, there are some minor issues to be resolved. 2) Implementation: Kernel - HCI and L2CAP Netgraph nodes have been implemented; 3) User space (API, library, utilities) - in progress. 4) Testing: In progress. I do not have real Bluetooth hardware at this point, so i wrote some tools that allow me to test the code. Some of them will be used as foundation for future user space utilities.
Issues: 1) Bluetooth hardware; I do not have real Bluetooth hardware, so if people can donate hardware/specs it would be great. I promise to write all required drivers and make them available. I also promise to return hardware/specs on first request. 2) Project name; I would like to see the name that reflects the following: it is a Bluetooth stack, implementation is for FreeBSD and implementation is based on Netgraph framework
This project is now finally underway, thanks to DARPA and NAI getting a sponsorship lined up. The infrastructure code and data structures are currently taking form inside a userland simulation harness. Basic MBR and BSD methods have been written and device attach/taste/dettach algorithms been implemented and validated.
I've update OS of buildboxes to the latest FreeBSD 5-current and 4-stable. Everything goes fine. From January 2002, I've started a webzine, SNAPSHOTS Notes (only Japanese version is available). SNAPSHOTs Notes pickups tips and information especially for the people living with FreeBSD 5-current/4-stable. Article or idea for SNAPSHOTs notes are always welcome (you don't need to write in Japanese :-).
Robert Watson created the TrustedBSD audit perforce tree, which is a branch from the TrustedBSD base tree, in order to start pushing development efforts towards using a revision control system. Andrew Reiter started to merge in some framework related code for generation of audit records, enqueueing writes, and handling data writing. There is a great deal of work to be done with updates and discussion on the trustedbsd-discuss@TrustedBSD.org mailing list.
The KSE project (an attempt to support scalable thread in FreeBSD using kernel support), has reached What I call "milestone 3". At this milestone it is possible to run a multithreaded program on a single CPU but with full concurrency of threads on that CPU. In other words the kernel supports the fact that one thread can block by allowing another thread to run in its place. A test program that demonstrates this is available at the above website.
Milestone 4 will be to allow threads from the same program to run on multiple CPUs but may require more input from the SMPng project. I am at the moment (Feb 6) getting ready to commit a first set of changes for milestone 3, that have no real effect but serve to drastically reduce the complexity of the remaining diff so that others can read it more easily. After changes to libkvm to support this diff have been added it should be possible to run 'ps' and look at multiple threads in a treaded process. I will be demonstrating KSE/M3 at BSDcon.
The Netgraph ATM package has been split into a number of smaller packages: bsnmp is a general-purpose SNMP daemon with support for loadable modules. Two modules come with it: one implementing the standard network-interface and IP related parts of MIB-2 and one for interfacing other modules to the NetGraph sub-system. ngatmbase contains the drivers for the ATM hardware, the ng_atm netgraph type and a few test tools. This package allows one to use ATM PVCs. It should be possible, for example, to do PPP over ATM with this package. Both bsnmp and ngatmbase are available in version 1.0 under the link above. Two other modules will be released in February: ngatmsig containing the UNI-4.0 signalling stack as netgraph nodes and ngatmip containing CLIP and LANE-2.0.
A significant amount of progress was made in December and January, particularly in the area of utility conformance. Several utilities were updated to conform to SUSv3, they include: at(1), mailx(1), pwd(1), split(1), and uudecode(1). Several patches have been submitted to increase conformance in other utilities, they include: fold(1), patch(1), m4(1), nice(1), pr(1), renice(1), wc(1), and xargs(1). These are in the process of being reviewed and committed. Two new utilities have been written, specifically pathchk(1) and tabs(1). These are also being reviewed and will be committed shortly.
A patch which implements most of the requirements of scanf(3) is being reviewed and is expected to be committed shortly. This will allow us to MFC a number of new functions and headers. Additionally, work has started on wide string and complex number support.
For 4.5-RELEASE, port ja-man-doc-4.5.tgz is in sync with base system except for OpenSSH pages (OpenSSH 2.3 based instead of 2.9) and perl5 pages (jpman project do not maintain). Section 3 updating has 55% finished.
OKAZAKI Tetsurou has incorporated changes on base system's groff into port japanese/groff. MORI Kouji has fixed two bugs of port japanese/man.
The KAME project is currently focusing on the scoped addressing architecture, the advanced API implementation, NATPT and the mobile ipv6 implementation. Though these stuffs are not stable enough to be merge into the FreeBSD tree, you can get and try them from the above URL.
The FreeBSD in Bulgarian project aims to bring a more comfortable working environment to Bulgarian users of the FreeBSD OS. This includes, but is not limited to, font, keymap and locale support, translation of the FreeBSD documentation into Bulgarian, local user groups and various forms of on-line help channels and discussion forums to help Bulgarians adopt and use FreeBSD.
A guide for using FreeBSD with Bulgarian settings has been put up on the project's website. The CVS repository will be made public shortly, linked to on the URL's above.
An independent project for making FreeBSD easier to use by Bulgarians has appeared, http://www.FreeBSD-bg.org/. It also hosts a mailing list for discussions of FreeBSD in Bulgarian, stable@FreeBSD-bg.org. For more information about the mailing list, send an e-mail with "help" in the message body to majordomo@FreeBSD-bg.org.
The past two months have been an exciting time in the FreeBSD Java Project with the signing of a license between the FreeBSD Foundation and Sun allowing us access to updated JDK source code and the Java Compatibility Kit (JCK). This license will also allow the project to release a binary version of both the JDK and JRE once JCK testing is complete. Work on this testing is under way with the project hopeful of being able to make a binary release in the not too distant future.
In lieu of the binary release which was hoped for with FreeBSD 4.5 the project will release an updated source patchset this weekend. This patchset will feature further work on the FreeBSD "native" threads subsystem from Bill Huey. Also, thanks to hard work by Joe Kelsey and Fuyuhiko Maruyama, the patchset will for the first time feature a working Java browser plugin!
Extending camcontrol's page definition file format to include both modepage and logpage definitions; adding support to camcontrol to query and reset log page parameters. Consideration is being made to possibly include support for diagnostic and vital product data pages, but that is outside the current project scope. New page definition file format includes capability to conditionally include page definitions based on SCSI INQUIRY results allowing vendor-specific pages to be described also. Approximately 90% complete.
OpenPAM, a new library intended to replace Linux-PAM in FreeBSD, has been written and is undergoing integration testing. It is available for download from the URL listed above.
In addition to this, a couple of new modules have been written (pam_lastlog(8), pam_login_access(8)), and the pam_unix(8) module has been extended to perform most of the tasks normally performed by login(1), which is now fully PAMified.
The PAM FDP article has been put on hold until OpenPAM replaces Linux-PAM in CVS, to avoid wasting effort on soon-to-be obsolete documentation.
Substantial progress has been made towards a working MAC implementation. The focus over the last two months has been moving from a hard-coded series of MAC policies to a more flexible implementation. A pluggable policy framework has been created (and is still under development), supporting Biba, MLS, TE, a "BSD Extended" model, and a sample mac_none module. Some modules must be compiled in or loaded prior to boot; others may be introduced at run-time. Support for networking has improved, with improved handling of IP fragmentation in IPv4, support for various pseudo-interfaces such as if_tun and if_tap, improved integration into userland, NFS-related fixes, moving the VFS enforcement out of individual filesystems, support for a 'multilevel' mount flag, support for explicit labeling in procfs and devfs, addition of an 'extattrctl lsattr' argument to list EAs on a filesystem, support for label ranges in the Biba and MAC policies, and much more.
Targets for the next two months include more universal enforcement of VFS-related calls, improved support for alternative ABIs, improved flexibility of in-kernel subject and object labels, support for IPv6 and IPsec, and improved support for NFS serving.
Development continues in the FreeBSD Perforce repository, which may be accessed using cvsup.
Now that the patch has been mailed to the freebsd-arch@FreeBSD.org mailing list, and that there were no objections, the commit will happen soon. Poul is currently testing it in his own tree. After it has been committed, it will be time to modify the filesystems in the tree to use VFS_NMOUNT instead of VFS_MOUNT. Mount(8) will also need some modifications. Some new manpages -- nmount(2) and kernel_vmount(9) -- are being created in the meantime.
Alfred Perlstein committed file descriptor locking code which was definitely a good push towards trying to lock down some important pieces of global data. Peter Wemm has made progress on pmap cleanups for x86 SMP TLB shootdowns. Matt Dillon and John Baldwin have made progress on getting patches done for moving accesses to ucred's out from under Giant's protection. John Baldwin has also made some commits in order to get the alpha port's SMP working. Matt Dillon has plans for hunting down fileops locking issues in order to continue his previous Giant pushdown work.
This report covers FreeBSD development activities from February, 2002 through April, 2002. It's been a busy few months -- BSDCon in San Francisco, the FreeBSD Developer Summit, a first development preview of 5.0-CURRENT, not to mention lots of progress on the 5.0 feature set (SMPng, sparc64, GEOM, ... the list goes on).
In the next two months, the USENIX ATC occurs (highly recommended event for both developers and users), and a number of new software components will hit the tree, including UFS2 and the TrustedBSD MAC framework. We'll also complete the elections for the FreeBSD Core Team, and should have the next Core Team online by the time the next report rolls around. Stay tuned for more!
Robert Watson
Packages are built from the FreeBSD Ports Collection on a cluster of i386 and alpha machines using scripts available in /usr/ports/Tools/portbuild/. Over the past few months I have been cleaning up and extending these scripts to improve efficiency and allow for greater flexibility in how package builds are performed. Major improvements so far have been: cleaning up and modularizing the scripts to avoid code duplication and reduce the need for ongoing maintenance; optimizing the build process and making it much more robust against client machine failure; and allowing package builds to be restarted if they are interrupted. The i386 package cluster is currently running FreeBSD 5.0-CURRENT, and it has proven to be a useful testing ground for exposing kernel bugs, especially those which only manifest under system load.
Future plans include the ability to perform incremental package rebuilds which only build packages that have changed since the last run. This will allow packages to be made available on the FTP site within an hour or two of the CVS commit to the ports collection. We also hope to set up a sparc64 package cluster in the near future, but this is contingent on suitable hardware.
FreeBSD's new kernel memory allocator has been committed to 5.0. UMA is a slabs derived allocator that supports memory reclaiming, object caching, type stable storage, and per CPU free lists for optimal SMP performance. It has both a malloc(9) interface and a zone style interface for specific object types. uma(9) will be available shortly.
Read-only support for UDF filesystems was checked into the 5-CURRENT branch in April. Backporting for 4-STABLE is being conducted by Jeroen. The next phase is to write a newfs_udf, then move on to adding write support to the filesystem. I'm still looking for a volunteer to handle read and write support for write-once media (e.g. CD-R).
I have released a new zero copy sockets snapshot, the first since November, 2000. The code has been ported up to the latest -current, and the jumbo code now has mutex protection. Also, zero copy send and receive can be selectively turned on and off via sysctl to make it easier to compare performance with and without zero copy. Reviews and comments are welcome.
I'm slowly making progress. The second engineering release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20020506.tar.gz
This release includes support for H4 UART transport layer, Host Controller Interface (HCI), Link Layer Control and Adaptation Protocol (L2CAP) and Bluetooth sockets layer. It also comes with several user space utilities that can be used to configure and test Bluetooth devices.
I'm currently working on RFCOMM protocol implementation (Serial port emulation over Bluetooth link). My next goal is to port Service Discovery Protocol (SDP) implementation from BlueZ (http://bluez.sf.net). I'm also thinking about adding USB device support (as soon as i find/buy hardware).
Issues: 1) Bluetooth hardware; I have couple PC-CARDs that i use for development and testing purposes, but i'd love to have more. 2) Time; My regular day job kicked in, so i will be spending more time doing stuff i'm getting paid for.
Since the last status report, two developers working on utility conformance were given commit access to the FreeBSD CVS repository to help expedite development. As a result, the following utilities have been brought up to conformance, they include: csplit(1), env(1), expr(1), fold(1), join(1), m4(1), mesg(1), paste(1), patch(1), pr(1), uuencode(1), uuexpand(1), and xargs(1). The printf(1) utility was brought up to conformance with the 1992 edition of POSIX.2, with further development planned.
On the header front, much progress has been made. Specifically, infrastructure to control visibility of components of a header, based on the standard requested by an application, has been added to <sys/cdefs.h>. Some work has been completed on renovating the way types are defined. This has lead to the creation of <sys/_types.h>. Further improvements such as the merger of <machine/ansi.h> and <machine/types.h> are planned. Additionally, the headers: <strings.h>, <string.h>, and <sys/un.h> have been made to conform to POSIX.1-2001.
On the API front, scanf(3) has received support for 5 new length modifiers (hh, j, ll, t, and z). A patch to implement two additional conversion specifiers (j and z) has been developed for printf(9) and is expected to be committed soon.
In other news, the project's web site has been moved to the main FreeBSD site. It is now available at the URL at the top of this status report. Please update your bookmarks.
Version 1.1 for FreeBSD-current is now available. It includes the SNMP-daemon package bsnmp, the driver package ngatmbase, the UNI4.0 signaling package ngatmsig and the network emulation package ngatmnet. NgAtm allows both to build applications running directly on top of ATM and to use ATM-Forum LAN emulation to use IP over ATM. Currently we are working on a simple switch module, that implements the network side signaling and ILMI as well as simple routing and call admission control.
The GNOME project has seen quite a few changes lately. For one, the author of this update has recently been given "The Bit." Joe Marcus Clarke now has CVS access, and is working primarily on the GNOME project. Joe has been closing a good deal of GNOME PRs, as well as patching some of the existing GNOME 1.4 components.
The GNOME 2 porting effort continues on. We have completed porting of the GNOME 2.0 API, and are 75% complete on porting the full GNOME 2.0 desktop. When complete, GNOME 1.4 and GNOME 2.0 will be co-resident in the ports tree. Both APIs can be installed concurrently in the same PREFIX, but the respective desktops will remain mutually independent. Maxim Sobolev is working on adapting bsd.gnome.mk to handle both versions of the desktop in an elegant fashion.
Not to be left out, the existing GNOME 1.4 components have received numerous updates to keep them in sync with the stable distfiles on gnome.org. We have seen many "1.0" milestone releases including the most recent AbiWord 1.0.0. In the next few weeks, we will be making sure all the GNOME 1.4 components build correct packages on bento so that GNOME 1.4 will be on the 4.6-RELEASE CD.
FreeBSD/KGI started last year after the port of GGI to VGL. KGI (Kernel Graphic Interface) is a kernel infrastructure providing user applications with access to hardware graphic resources (dma, irqs, mmio). KGI is already available under Linux as a separate project. The FreeBSD/KGI project aims at integrating KGI in the FreeBSD kernel. Mostly a port for now, but optimized for FreeBSD in the future. Currently FreeBSD/KGI is under development and the code is only available for reading, compiling but not running. More interesting are design hints found at the project URL.
We now have a loadable mfsroot floppy. It contains just the diskeditor (which is really a disk partitioner) which has been enhanced and is probably in its final form. It's been geared towards making the newfs(1) and mount(1) steps separate dialogs, so it reduceed its complexity. A basic fstab class has been implemented to manipulate /etc/fstab and mountpoint. This might find a use outside libh, by the way. Libh package format is still incomplete and somehow buggy, so it's my next target.
There is a API documentation effort underway with the help of doxygen(1), so there's now more documentation for people that want to get started with libh.
All this lead me to prepare the release of another alpha preview of libh that will shortly be available in the ports collection (0.2.2). Also, a new committer (okumoto) has joined the project (as well as I) and he is currently working on cleaning up the build system. It's been a few months without news, so this probably seemed a bit long, but don't worry, we still need your help to really get this going!
There are several new topics, including: Source Code Tour is now separated into kernel part and userland part, yet another snapshots from RELENG_4_x branch (currently 4.5-RELEASE-p4), add several packages including XFree86 4.x to installation CD-ROM, new cdboot-only ISO image, fix breakage of duplex.iso, etc. See also the project webpage for more detail. Also, I have a plan to add FreeBSD/alpha distribution to this project -- stay tuned.
KAME Project has been extended until March 2004, and we decided the project roadmap for these two years. The first one year is for implementation, and the remaining year is for feedback of our results into other BSD projects (please refer to the above URL for further detail). Great change is lack of NAT-PT support due to a lack of human resource, although KAME snap still contains it as it is.
SUZUKI Shinsuke (suz@kame.net) has begun working for KAME and FreeBSD merge task in cooperation with Umemoto-san (ume@FreeBSD.org). Some of KAME stuff (critical bug fix, newest ports for pim6sd and racoon, etc) has been merged into 4-stable in this April.
Over the past couple of months, progress has pretty much stopped until very recently. The past few changes to the audit code were update the usage of zones to UMA zones, cleanup some old cruft, and start toying with the idea of having an audit write thread implemented as an ithd. The next step is to decide two realistic approaches to the where the records will be dumped -- whether that is to a local disk or fed up to userland and then dealt with. After that, the goal will be to expand the number of events that are being audited, while also working in some performance testing procedures. I will be posting to trustedbsd-audit about the recent changes shortly.
Over the last three months, there has been a lot of activity in the TrustedBSD MAC tree. An initial commit of the SEBSD code (NSA FLASK and SELinux implementation) was made; many MAC policies previously linked directly to the kernel via kernel options were moved to kernel modules; the flexibility of the framework was improved relating to the life cycle of object labels; additional labeling and access control hooks were introduced; new policies were introduced to demonstrate the flexibility of the framework (including a cleanup of inter-process authorization, additional VFS hooks, improved support for multilabel filesystems, network booting, IPv6, IPsec, support for "peer" labels on stream sockets). Current modules include Biba integrity policy, MLS confidentiality policy, Type Enforcement, "BSD Extended" (permitting firewall-like rulesets for filesystem protection), "ifoff" (limit interface communication by policy), mac_seeotheruids (limit visibility of processes/etc of other users), "babyaudit" (a simple audit implementation), and SEBSD (FLASK/SELinux port).
Over the next month, a final move to completely dynamic labeling will be made, permitting policies to introduce new state relating to process credentials, vnodes, sockets, mounts, interfaces, and mbufs at run-time, allowing a broad range of flexible label-driven policies to be developed. In addition, application APIs will be re-designed and re-implemented so as to better support a fully dynamic policy framework. We plan to make an initial prototype patchset available for review in June, with the intent of committing that patchset in mid-June.
Updated prototype code may be retrieved from the TrustedBSD CVS trees on cvsup10.FreeBSD.org.
The painful parts are now completed, with all authentication- related utilities converted to PAM (except for those cases where it doesn't make sense, like Kerberos- or OPIE-specific commands). OpenPAM is complete (except for a few missing man pages) and seems to work well.
For more details, see the activity reports linked to above.
OpenSSH has been upgraded to 3.1, and the kinks seem to have been worked out by now. OpenSSH will now use PAM for both ssh1 and ssh2 authentication.
The KSE project had floundered due to lack of development time for awhile, but has been picked up recently by Jonathan Mini. Currently, the main focus is to prepare the "milestone 3" code for inclusion into -CURRENT.
The project is still working towards "milestone 4" (allowing threads from the same process to run on multiple CPUs), which should be significantly easier now due to work done by the SMPng project over the past several months.
Help could be used in several areas of the project, especially with porting the libc_r (pthreads) library to KSE's threading model.
NEWCARD support tried to merge CardBus functions with PCI functions, but that failed to properly route interrupts. A branch for the merge was created and will be merged into the main line at a later date. Too many other things going on in my life to make much progress.
Work on the host access point support for the Prism2 and Prism2.5 based wireless cards has been integrated into the kernel. This work is largely based on Thomas Skibo's initial implementation.
Continued bug fixing and hardening for this last few months.
Future work will include making target mode work correctly and fast.
The LSI-Logic chipset's MPT Fusion driver is also being evaluated.
The FreeBSD MTRR code has been made more robust against unexpected values sometimes found in the Athlon's Memory Type Range Registers. Problems with these values had prevented XFree 4.2 running on some motherboards. Experimentation indicates that these undocumented values may control the mapping of BIOS/ROMs or have something to do with SMM. If anyone can provide details of what these values mean, can they please let me know, so the MTRR code can be completed.
IPMI Tools for FreeBSD is a collection of C and Python applications and modules for exploring the information available via the Intelligent Platform Management Interface (IPMI), as implemented on server motherboards by Intel and HP. IPMI is an open standard with patent protection for adopters which defines standard interfaces to on-board management hardware. The management hardware consists of a CPU, sensors such as temperature probes and fan speeds, and repositories such as the System Event Log and Field-Replaceable Unit (FRU) inventory, and other system information.
A basic set of tools was recently made available which uses the KCS and SMIC system interfaces to retrieve the System Event Log, FRU repository, and system sensors. Additional features are currently under research. Suggestions for additional features and programs are greatly appreciated.
The PowerPC port is moving ahead. It can now mount a root file system and exec init, but fails when trying to map init's text segment in. I'm hoping to have it starting my fake "Hello, world!" init soon, after which I plan to try and get some libc bits in place so that I can build /bin and /sbin and try to get to actual single-user.
4.5-RELEASE Japanese manpage package, ja-man-doc-4.5.tgz, once published with OpenSSH 2.3 (as reported by previous status report) on January 31, is replaced with new package with OpenSSH 2.9 based manpages on March 3. Since then, we have been updating Japanese manpages for 4.6-RELEASE. For new translation and massive update, we have been making a lot of effort.
Continuing section 3 updating has 73% finished.
The GEOM code has gotten so far that it beats our current code in some areas while still lacking in others. Work continues on a generalized interface for "magic data" (boot blocks, disklabels MBR's etc) manipulation from userland.
With GEOM enabled in the kernel any FreeBSD platform will now recognize PC style MBR's, i386 disklabels, alpha disklabels, PC98 extended MBRs and SUN/Solaris style disklabels.
Since the last progress report, the initialization code was much cleaned (thanks to NetBSD's acort32 port) and partial DDB support as been added. I'm now struggling to put the pmap module into a working state. The latest patch set only includes the initialization changes. I did some tries to get what I had so far working on my iPAQ without much successes (downloading a kernel over a serial link is way too painful). If anyone has had success in getting any iPAQ to work as a USB storage device under *BSD please contact me.
I've been mentoring someone on locking up the protocol control blocks in the networking stack. She has already finished TCP and UDP and I'm currently reviewing the patch with her and going over some networking lock order issues. Locking up raw protocol interface control blocks follows next.
Support for stf(4), faith(4), and loopback interfaces has been committed to current. The stf and faith support has been MFC'd. In current the API has changed to move unit allocation into the generic cloning code reducing the amount of support code required in each driver. Code improvements to increase our API compatibility with NetBSD will be committed soon along with cloning support for discard interfaces and ppp(4) interfaces.
Thanks to
IA64 has had a busy few months. Aside from gcc, we are now fully self hosting on IA64. Doug Rabson has performed his magic and implemented the execution of 32 bit i386 application binaries although more work remains to be done to make ld-elf.so.1 happy with the different underlying page size. We have been using the i386 perforce binary to do actual development work and submit from the ia64 systems themselves. Marcel Moolenaar has been working on SMP and machine-check support. We have been running SMP kernels amazingly reliably on our development boxes for quite some time now. syscons is now functional. We have produced a self-booting run-root-on-cdrom ISO image (idea taken from the sparc64 folks) that has been used to manually self install an IA64 system from a blank disk. Aside from a few minor loose ends we now have complete 'make world' functionality. sysinstall works on ia64. We plan on producing a semi-respectable boot/install cdrom image shortly.
As of Thur May 9th, 2002 FreeBSD 5-CURRENT is now using a GCC 3.1 prerelease snapshot as the system C compiler. At this time of cutting over, the compiler is working well on i386, Alpha, Sparc64, and IA-64 for building world. There is a known problem with our atomic ops on Alpha that prevents a GCC 3.1 built kernel from booting.
Currently the C++ support libraries (libstdc++, et.al.) does not build and thus prevents the system C++ compiler from being used.
The release engineering team released FreeBSD 5.0-DP1 on 8 April 2002. This Developer Preview gives developers and other interested parties a chance to help test some of the new features to appear in 5.0-RELEASE. This distribution has known bugs and areas of instability, and should only be used for (non-production) testing and development.
The next releases of FreeBSD will be 4.6-RELEASE (scheduled for 1 June 2002) and 5.0-DP2 (scheduled for 25 June 2002). Information on the release schedules and more can be found on the team's new area on the FreeBSD Web site (see the URL above).
Finally, the team has gained two new members: Brian Somers and Bruce A. Mah.
libradius now supports RADIUS vendor attribute extensions and user-ppp is now capable of doing MS-CHAP authentication via a RADIUS server. A new net/freeradius port has been created for support of MS-CHAP in a RADIUS server.
MS-CHAPv2 support will be added soon.
The work is sponsored by Monzoon.
Mike Makonnen has done quite a bit of excellent work on porting the scripts from FreeBSD into the NetBSD framework. The next step seems to be to try to reduce the amount of diffs between our implementation and the original set from NetBSD.
The SMPng project has been picking up steam in the last few months thankfully. In February, Seigo Tanimura-san committed the first round of process group and session locking. Alfred Perlstein also added locking to most of the pipe implementation. In March, Alfred fixed several problems with the locking for select() and pushed down Giant some in several system calls. Andrew Reiter added locking for kernel module metadata, and Jeff Roberson wrote a new SMP-friendly slab allocator to replace both the zone allocator and the in-kernel malloc(). The use of the critical section API was cleaned up to not be abused as replacements for disabling and enabling interrupts. Also, Matt Dillon optimized the MD portion of the critical section code on the i386 architecture. Several other subsystems were also locked in April as well. See the SMPng website and todo list for more details.
Some of the current works in progress include locking for the kernel linker by Andrew Reiter and light-weight interrupt threads for the i386 by Bosko Milekic. Seigo Tanimura-san, Alfred Perlstein, and Jeffrey Hsu are also working on locking down various pieces of the networking stack. Alan Cox has started working on fixing the existing locking in the VM subsystem and moving bits of it out from under Giant. John Baldwin has written an implementation of turnstiles as well as adaptive mutexes in the jhb_lock Perforce branch. The adaptive mutexes appear to be stable on i386, alpha, and sparc64, but the turnstile code still contains several tricky lock order reversals. John also plans to commit the p_canfoo() API change to use td_ucred in the very near future and then finish the task of making ktrace(4) use a worker thread.
The patch for the new mount API has now been committed to the tree. Several filesystems also have been converted to this new mount API, namely procfs, linprocfs, fdescfs and devfs. I'm working on converting more filesystems to nmount, and actually already have UFS done. It has not been committed yet to avoid conflicting with the UFS2 work, but it should hit the tree soon. Manpages are still missing at the moment because I had to modify the API slightly. I hope to have them done soon now.
The second FreeBSD Developer Summit, held following the BSD Conference in San Francisco in February, was a great success. Around 40 developers attended in person, another five by phone, and many others by webcast. During a marathon-esque eight hour session, a variety of development topics were discussed, including adding inheritance to the KOBJ system, ports to new architectures, adaptations of the toolchain for new architectures, the GEOM extensible storage device framework, upcoming changes to the network stack, TrustedBSD features, KSE, SMPng, and the release engineering schedule. This event was sponsored by DARPA and NAI Labs, with webcasting provided by Joe Karthauser, bandwidth provided by Yahoo!. Planning for future such events is now underway; a summary/transcript of discussion may be found at the URL above.
May and June were remarkably busy months for the FreeBSD Project-- FreeBSD developers met in Monterey, CA in June for FreeBSD Developer Summit III to discuss strategy for the FreeBSD 5.0 release later this year, for the USENIX Annual Technical conference and for the FreeBSD BoF. Substantial technical progress was made on FreeBSD 5.0, and FreeBSD 4.6-RELEASE was cut on the RELENG_4 branch in June.
The remainder of the summer will continue to be busy. Final components and features for 5.0-RELEASE will go into the tree, and the development direction will change from new features to stability, performance, and production-readiness. With additional 5.0 development previews late in the summer, we hope to broaden the tester base for the -CURRENT branch, and start to get early adopters digging out any potential problems in their test environments. I encourage both FreeBSD Developers and FreeBSD Users to give 5.0-DP2 a spin (on a machine without critical data!) and let us know how it goes. The more testing that happens before the release, the less fixing we have to do afterwards!
Robert Watson
The current cache for the TCP metrics is embedded directly into the routing table route objects. This is highly inefficient as every route has an empty 56 Byte large metrics structure in it. TCP is the only consumer (except the MTU and Expiry field) of the structure. A full view of the Internet routes (110k routes) has more than 6 Mbyte of unused overhead due to it. The hit rate today is at only approx. 10% in webserver applications. The TCP hostcache will move this entire metrics structure from the routing table to the TCP stack. Every entry is a host entry so a simple hash table is sufficient to keep the entries. Its implementation is much like the TCP Syncache.
The hostcache is going through testing on our servers and will be ready for committing in September. The results of the TCP metrics measurement will be used to tune the cache.
The current Patricia Trie routing table in BSD UNIX is not very efficient and wastes an enormous amount of space for every node (more than 256 bytes) (A full Internet view of 110k routes takes 33 MByte of KVM). Another problem are pointers from and to everywhere in the routing table. This makes replacing the table very hard and also significantly increases the table maintenance burden (for example for some kinds of updates the entire PCB has to be searched linearly). Also this is a heavy burden for SMP locking. The rewrite focuses on untangling the pointer mess, making the routing table replaceable and providing a more IP optimized table (5 MByte for 110k routes). Other new options include policy routing and some structural alignments in the network stack for clarity, simplicity and flexibility.
The rewritten IP routing table will be ready for committing in October.
These students will analyse the tcpdumps of five major Swiss newspaper websites which give a representative overview of the user structure in Switzerland. The nice thing about Switzerland is that is has a very good mix of Modem/ISDN, leased line, Cable, ADSL and 3G/GSM/GPRS users. Every Internet access technology is represented. The goal is to analyze the behavior of all TCP sessions to the monitored sites. Parameters to be analyzed include TCP session RTT, RTT variance, in/outbound BDP, MSS changes, flow control behavior, packet loss, packet retransmit and timing of HTTP traffic to find optimal TCP parameter caching method.
If you have any other metrics you think is useful please contact me so I can put that into the job description for the Students. The study will be made in September and October.
The current natd is pretty powerful in translating different kinds of traffic but not very powerful in configuration. This project rewrites natd and parts of libalias to give it a configuration set as powerful and expressive as the ones in ipf (ipnat) and pf. In addition it'll use kqueue and will support aliasing to multiple IP addresses.
The rewritten natd will be ready for committing in early September.
IA64 has been progressing slowly. We have access to a prototype 4-way Itaninum2 system from Intel and have managed to get it up and running to the point of being able to access disk and network with SMP enabled. We have a big problem with ACPI2.0 and PCI routing table entries behind pci-pci bridges with no short-term solution in sight. Various WIP items have been committed to CVS, namely more complete support for executing 32bit i386 binaries as well as Marcel Moolenaar's prototype EFI GPT tools.
Max has been busy cleaning up the user interface dark side, and has come up with a plan to improve the build system (using an automated Makefile dependency generator); the UI design and the TCL glue magic (using Swig). A development page has been created on usw4, publishing a lot of information about the current project status, a Changelog, screenshots, documentation, etc. A new listbox widget has been implemented, making diskeditor look nicer and more usable. The package system backend is being inspected and redesigned to conform to a standard that is itself being re-thought. Indeed, the old sysinstall2.txt text has been SGML-ized and enhanced and now provides a good (although rough) overview of libh package system. This allowed the document to be enhanced with diagrams of how different procedures work. We are therefore getting closer to a real pkgAPI specification document. The package management tools have been slightly enhanced and should be a bit more usable, and we started committing regression test suites in the tree, mostly to test and maintain pkg API conformance.
So work continues on libh. I plan to take a look at the rhtvision port to see if it would be better to use it for the tvision backend. I'll keep on working on the package system to make it really trustworthy, while Max is continuing his great work on the UI subsystem. I hope to make a new libh alpha release soon. Note that from now on, libh progress will be published on the development page.
A major power bug was fixed in oldcard. This caused many problems for people using PCI interrupts having their machines hang on boot. This fix has made it into 4.6.1.
Cardbus power is now used on all cardbus bridges that support it. This means that we now support 3.3V cards on all cardbus bridges. Before, we only supported them on some of the bridges because every bridge uses different 3.3V power control when programmed through the ExCA registers. Now that we're going through the CardBus bridge's power control register, 3.3V cards work. In fact, for CardBus bridges, the so called X.XV and Y.YV cards will work in those bridges that support them. However, X.XV and Y.YV haven't been defined yet, and no bridges support them (but the bridge interface define it). Obviously this latter part is untested.
CL-PD6722 support has been augmented slightly. Now it is possible to instruct the driver which type of 3.3V card detection strategy to use. There are three choices: none, do it like the CL-PD6710 does it and do it like the CL-PD6722 does it.
Preliminary support for the CL-PD6729 on a PCI card using PCI interrupts has been committed. However, it fails for at least one of the cards like this the author has.
Client drivers can now ask for the manufacturer and model number of the card without parsing the CIS directly.
Except for fixing bugs and updating pccard.conf entries, no additional work is planned on the OLDCARD system.
A devd daemon, to replace pccardd and usbd, has been designed. A few minor bugs have been fixed in NEWCARD. NEWCARD is now the default in -current. There is an experimental pci/cardbus bus code merge available as a branch which will be merged into current as soon as it is stable.
Status: The ed driver, for non-ne2000 clones, is broken and won't probe. The ata driver won't attach. The sio driver hangs on the first character. The wi driver is known to work well. Cardbus cards are generally known to work well, except for some de based cards, which unfortunately includes the popular Xircom cards. Many systems fail to work because acpi fails to route interrupts correctly for non-root pci bridges.
Things are going well with the FreeBSD GNOME Project. We have just finished porting the GNOME 2.0 Final development platform and desktop to FreeBSD! We hope to be able to make GNOME 2.0 the default for 5.0-DP2 and 4.7-RELEASE. In the meantime, we're working to port more GNOME 2.0 applications.
In order to allow GNOME 1.4.1 applications to work with GNOME 2.0, we are revamping the GNOME porting infrastructure. GNOME 1.4.1 based ports are being converted to use the new GNOMENG porting structure. The specifics of this new system will be written up in the GNOME porting guide found on the FreeBSD GNOME project homepage.
The BSD Java Porting Team has been making slow but steady progress on a number of fronts in the last few months. Unfortunately most of this has occurred behind the scenes, meaning this is a good opportunity to bring the community up to date.
I'm afraid KAME Project does not work actively with regard to FreeBSD in these two month, since we are too busy with the demonstration of our IPv6 implementation at Networld+Interop 2002 Tokyo. (Thanks to a great effort, the demonstration was quite successful)
We are aware of netinet6-related bug reports regarding socket handling, fine-grain locking, ip6fw etc. Regret to say, we could not answer them right now due to the above situation, however we'll discus these issues internally and determine what to do.
The BSDCon 2003 Program Committee invites you to contribute original and innovative papers on topics related to BSD-derived systems and the Open Source world. Topics of interest include but are not limited to:
Submissions in the form of extended abstracts are due by April 1, 2003. Be sure to review the extended abstract expectations before submitting. Selection will be based on the quality of the written submission and whether the work is of interest to the community.
We look forward to receiving your submissions!
Over the past few months the FreeBSD Release Engineering Team oversaw a release process that culminated in the release of FreeBSD 4.6 for the i386 and Alpha architectures on June 15. The RE team is currently working concurrently on FreeBSD 4.6.1 and 5.0 DP2. 4.6.1 is a minor point release with an updated SSH and BIND, fixes for some of the reported ata(4) problems, and assorted security enhancements that will be detailed in the release notes. The release engineering activities for 4.6.1 are taking place on the RELENG_4_6 branch in CVS, while the work on 5.0 DP2 is taking place in Perforce so as not to disturb ongoing -CURRENT development. We are still committed to FreeBSD 5.0 on or around November 15, 2002. For more information about upcoming release schedules, please see our website above. The RE team would like to thank Sentex Communications for providing the release builders with access to a fast i386 build machine. Compaq also donated a couple of fast Alpha build machines to the project.
The main goal of this project is to modify the IPSEC protocols to use the kernel-level crypto subsystem imported from OpenBSD (see elsewhere). A secondary goal is to do general performance tuning of the IPSEC protocols.
Basic functionality is operational for IPv4 protocols. IPv6 support is coded but not yet tested. Hardware assisted cryptographic operations are working with good performance improvements. Operation with software-based cryptographic calculations appears to be at least as good as the existing implementation. Numerous opportunities for performance improvements have been identified.
This work is currently being done in the -stable tree. A port to the -current tree is about to start.
Since the last status report, the following utilities have been brought up to conformance (at least to some degree) with POSIX.1-2001, they include: asa(1), cd(1), compress(1), ctags(1), ls(1), newgrp(1), nice(1), od(1), pathchk(1), renice(1), tabs(1), tr(1), uniq(1), wc(1), and who(1). In addition, development is taking place on bringing the BSD SCCS suite up to date with newer standards.
On the API front, printf(9) has been given support for the `j' and 'n' flags, waitpid(2) now supports the WCONTINUED option, and an implementation of fstatvfs() and statvfs() has been committed. An implementation of utmpx is in progress, which has an aim to address some of the major problems with the current utmp. Several headers have been brought up to conformance with POSIX.1-2001, they include: <netinet/in.h>, <pwd.h>, <sys/statvfs.h>, and <sys/wait.h>.
The goal of this project is to import the OpenBSD kernel-level crypto subsystem. This facility provides kernel- and user-level access to hardware crypto devices for the calculation of cryptographic hashes, ciphers, and public key operations. The main clients of this facility are the kernel RNG (/dev/random), network protocols (e.g. IPSEC), and OpenSSL (through the /dev/crypto device).
The software has been available as a patch against the -stable tree for about six months. The core crypto support is tested, including device drivers for the Hifn 7951, and Broadcom 5805, 5820, and 5821 parts. Recent work has concentrated on fixing device driver bugs, fixing support for Hifn 7811 parts, adding support for public key operations, and adding flow-control between the crypto layer and device drivers. Future work includes porting this facility to the -current tree.
The project took a major step at the beginning of July when Milestone-III was committed. Milestone-III allows a simple test program (available at /usr/src/tools/KSE/ksetest/) to run multiple threads, using kernel support. It does not yet allow the ability to allow these threads to run on different CPUs simultaneously. Milestone IV will be to allow this, however Milestone-III should allow Dan to start (with any interested parties) to start prototyping the userland part of the system. Milestone-III is only currently usable on x86, and does not include some of the requirements for full thread-control, suspension etc. that will be required later.
Before M-IV is started some small tweaking is likely in the central sources on M-III as we discover issues as we try to get the userland jumpstarted. These will have no effect on non-KSE processes, (i.e. all of them :-) and should not be an issue for other developers.
A tex/fig->html guru is needed to help maintain the KSE web page (not mentioned above as it is broken).
The SMPng project has continued to make steady progress in the past two months. Jeff Roberson completed the switch over to UMA for the general kernel malloc() and free() pushing down Giant appropriately so that callers of malloc() and free() are no longer required to hold Giant. Alan Cox continues to clean up the locking in the VM system pushing down Giant in several of the VM related system calls. Jeffrey Hsu committed locking for TCP/IP protocol control blocks in the network stack. John Baldwin committed the changes to the p_canfoo() API to use thread credentials for subject threads and added appropriate locking for the targer process credentials. Support for adaptive mutexes on SMP systems as well as the new IA32 PAUSE instruction were also committed in May. The kernel tracing facility KTRACE also received an overhaul such that the majority of its work was pushed out into a worker thread allowing trace points to no longer require Giant. Andrew Reiter has also been pushing down Giant in several system calls.
Bosko continues to work on light-weight interrupt threads for i386. Most of the bugs in the turnstile code have been found and fixed; however, the turnstile and preemption patches have temporarily been put on hold so that more emphasis can be placed on fixing bugs and making -current more stable in preparation for 5.0 release in November. Alan Cox and Andrew Reiter are continuing the work mentioned above. Jeff Roberson is also working on fixing the current vnode locking in VFS. Peter Wemm has also started to tackle TLB issues on SMP in the i386 pmap again as well.
After an outstanding job serving the project as Security Officer for over a year, Kris stepped down in January in order to focus more of his time pursuing his PhD. I offered to attempt to fill the vacant role.
This is the first report by the SO Team. Notable events since the beginning of 2002 follow.
28 FreeBSD Security Advisories have been issued, 16 of which were regarding the base system. Of those sixteen, 8 affected only FreeBSD.
FreeBSD Security Notices were introduced, and four have been issued so far. The Security Notices cover issues that are not regarded as critical enough to warrant a Security Advisory. So far only Ports Collection issues (i.e. vulnerabilities in optional 3rd party packages) have been reported in Security Notices. The first four Security Notices covered 53 individual issues.
Issues reported to the SO team are now being tracked using a RequestTracker ticket database.
The SO team has undergone membership changes, as well as some changes in internal organization. The membership and organization has also been made publicly visible on the FreeBSD Security Officer web page.
For 4.6-RELEASE, we announced the package ja-man-doc-4.6.tgz which is in sync with 4.6-RELEASE base system manual pages except for perl5 pages (jpman project do not maintain them). Continuing section 3 updating has 88% finished.
Progression is slow, but the effort is maintained. Most of fb over KGI has been written in parallel with a KGI display driver based on fb. DDC/DDC2 is being discussed for Plug & Play monitor support. KGI aims at providing a generic OS independent interface which would take advantage of FreeBSD I2C (iic(4)) infrastructure.
UFS2 is an extension to the well-known UFS filesystem which using a new inode format adds support for "64bit everywhere" and later for extended attribute support, in addition to the current UFS features: soft-updates and snapshots.
The basic UFS2 code has been committed and work on the extended attribute interface and vnode operations will continue.
The GEOM code has gotten so far that it beats our current code in some areas while still lacking in others. The goal is for GEOM to be the default in 5.0-RELEASE.
Currently work on a cryptographic module which should be able to protect a diskpartition from practically any sort of attack is progressing.
The port of openoffice 1.0 has been finished. Most showstopper issues with rtld, libc and our toolchain have been fixed. There is one remaining deadlock in the web-browser code of OO.org. If anybody like to help us with fixing this bug (may be another libc_r bug as it looks like) just mail me! Unfortunately gcc2 support got broken again with the import of gcc2.95.4 in STABLE. Exceptions support seems to be broken again; we get internal compiler errors with c++ exceptions code. You'll have to use gcc31 again.
Since our package cluster is outdated and can not build OO.org packages anytime soon, I did my own little package cluster and can now offer packages for 4.6R for 16 different languages. They can be found on the project homepage.
Porting of OpenOffice1.0.1 is on it's way. A beta port and a package have been made available on the project homepage.
The lightweight interrupt scheduling code makes scheduling an interrupt on i386 without having to grab the sched_lock possible, and also avoids a full-blown context switch.
Currently, the code in the p4 branch works, although needs a little bit of cleanup and, most importantly, requires a merge to post-KSE III. Now that stuff seems to have stabilized a bit, I'm waiting to get a little time (and nerve) to do the merge. Also, looking forward for some KSE interface that will allow for "KSE borrowing," which would make this cleaner with regards to KSE and lightweight interrupts. This is a 5.0 feature.
A lot of remaining PR's and Bugs have been closed. All relevant rpc concerning patches have been committed. Thanks go to Alfred and Ian Dowese.
Jean-Luc Richier <Jean-Luc.Richier@imag.fr> has made a patch available which adds IPv6 support to all remaining rpc servers. See ftp://ftp.imag.fr/pub/ipv6/NFS/NFS_IPV6_FreeBSD5.0.gz and ftp://ftp.imag.fr/pub/ipv6/NFS/0README_NFS_IPV6_FreeBSD5.0 We will check his code and add it to CURRENT ASAP.
A first commit part from TIRPC99 has been done. I'm working now on porting the remaining parts so when FreeBSD 5.0 gets released, it will be TIRPC99 based. This will happen together with the NetBSD project, as they use the same codebase as we do.
mb_alloc is getting some updates and a couple of optimizations. A new allocator interface routine should already be committed by the time this report is "published:" m_getcl() allocates an mbuf and a cluster in one shot. This is the result of months (literally) of requests from Alfred and, recently, Luigi - who, coincidentally, is the author of the same [upcoming] routine in -STABLE.
Other than that, mb_alloc is being shown how to perform multi-mbuf or cluster allocations without dropping the cache lock in between (m_getcl() and m_getm() will use this). Finally, work is being done to optimize ext_buf ref. count allocations and to provide support for jumbo (> 9K) clusters.
We are making excellent progress. There is a fully functioning implementation imported to -current now. We need as many people as possible to rc_ng equal to YES in /etc/rc.conf.
The next step is to set the default to YES, which we plan to do before DP 2.
In summer 2002 the native FreeBSD firewall has been completely rewritten in a form that uses BPF-like instructions to perform packet matching in a more effective way. The external user interface is completely backward compatible, though you can make use of some newer match patterns (e.g. to handle sparse sets of IP addresses) which can dramatically simplify the writing of ruleset (and speed up their processing). The new firewall, called ipfw2, is much faster and easier to extend than the old one. It has been already included in FreeBSD-CURRENT, and patches for FreeBSD-STABLE are available from the author.
I spent busy days in last two months, many new topics are emerged from the project. We now support FreeBSD/alpha 5-current distribution by cross-compiling on the x86 PC. Anonymous ftp area is now exported to the yet another web server. Our release branch snapshots are relocated to daemon.jp.FreeBSD.org because of our CPU/network bandwidth problem.
I'm seriously considering to solve the lack of CPU and network resources for the project's future evolution. Maybe the bandwidth problem can be resolved (several bandwidth offers have been received!), but there is no answer about CPU problems (I have a plan to upgrade our PCs from P3-500MHz to P4 or better). If you have interested in donating PCs to the project, please email me for more detail.
Regression tests for many bugs fixed in text manipulation utilities have been added, as well as tests for various non-standard versions of functionality that FreeBSD users should expect. A library of m4 macros for creating the tests themselves has been added.
The final version of SCCS distributed by CSRG has been integrated into the projects CVS repository, and worked on extensively to the point where essential functionality works on FreeBSD (and other operating systems). Some standards-related functionality has been implemented
The zero copy sockets code was committed to FreeBSD-current on June 25th, 2002. I'm not planning on doing any more patches, although I will leave the web page up as it contains useful information.
Many thanks to the folks who have tested and reviewed the code over the years.
Jennifer Yang's patch was committed June 10 for the BSD Summit. After a few bugs which were reported initially and fixed that same week, networking in -current has been stable, including the parts that were not locked up, like IPv6. Work is on-going to lock up the rest of the stack.
Not much to report. Another engineering snapshot is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20020709.tar.gz. If anyone has Bluetooth hardware and spare time please join in and help me with testing.
This snapshot includes basic support for USB devices and manual pages. The HCI layer now has support for multiple control hooks. All HCI transport drivers (H4, BT3C and UBT) has been changed to provide consistent interface to the rest of the world. Some userspace utilities have been changed as well.
Still no support for RFCOMM (Serial port emulation over Bluetooth link) and SDP (Service Discovery Protocol). Several design flaws have been discovered and it might take some time to resolve these issues.
The TrustedBSD Project has been busy in May and June, developing new features, presenting on the technology at the FreeBSD Developer Summit, and improving the readiness of the MAC branch for integration into the main FreeBSD tree. The migration to dynamic labeling in the TrustedBSD MAC framework is complete, with all policies now making use of dynamic labels in the kernel. This permits policies to associate arbitrary additional security data with a variety of kernel objects at run-time. Implement mac_test, a sanity checking module. Pass labels as well as objects to each policy entry point to reduce knowledge of label storage in the policies. Implement mac_partition, a simple jail-like policy. Adapt the MAC framework for process locking.
Improve support for sockets: provide a peerlabel maintained for stream sockets (unix domain, tcp), entry points for accept, bind, connect, listen. Improve support for IPv4 and IPv6 by labeling IP fragment reassembly queues, and providing entry points to instrument fragment matching, update, reassembly, etc. Locally disable KAME if_loop mbuf contiguity hack because it drops labels on mbufs: we need to make sure the label is propagated. Label pipes and provide access control for them. Improve vnode labeling: now handle labeling for devfs, pseudofs, procfs. Fix interactions between MAC and ACLs relating to the new VAPPEND flag.
SELinux policy tools now ported to SEBSD. SEBSD now labels subjects and file system objects. Provide ugidfw, a tool for managing rules for the mac_bsdextended policy.
Massive diff reduction. KSEIII merged. Main tree integration will begin shortly.
Updated prototype code may be retrieved from the TrustedBSD CVS trees on cvsup10.FreeBSD.org.
Throughout July and August, the FreeBSD Project has been working on pulling together the last few major pieces of new functionality for FreeBSD 5.0-RELEASE. At this point, the release appears to be on track for late November or early December. Work on fine-grained locking continues, especially in the VFS, as with improved support for threading through the KSE work; features such as GEOM, UFS2, and TrustedBSD MAC are maturing, and the new ia64 and sparc64 hardware ports are approaching production quality. In the next two months, we have a lot to look forward to: additional 5.0 developer preview snapshots, additional locking and threading improvements, and many cleanups on the new supported architectures. Firewire support has been imported into the main tree, and substantial cleanup of the ACPI/legacy PCI code is also in the works. Also, expect the import of new IPsec hardware acceleration support in the near future.
When new developer previews are posted, please give them a try! While we know that 5.0-RELEASE will be for "early adopters", the more testing we get out of the way now, the less we have to tidy up later. The new features are extremely exciting, and understanding when and how to deploy them properly will be important. In the next two months, among other things, the release engineering team will post updated release schedules, as well as guidance for FreeBSD consumers as to how to decide what releases of FreeBSD will be right for them. Keep an eye out for this, and provide us with feedback.
Also, for those of you in Europe -- we look forward to seeing you at BSDCon Europe in a couple of months!
Scott Long, Robert Watson
The BSDCon 2003 Program Committee invites you to contribute original and innovative papers on topics related to BSD-derived systems and the Open Source world. Topics of interest include but are not limited to:
Submissions in the form of extended abstracts are due by April 1, 2003. Be sure to review the extended abstract expectations before submitting. Selection will be based on the quality of the written submission and whether the work is of interest to the community.
We look forward to receiving your submissions!
Cloning support for ppp(4) and disc(4) interfaces has been committed. A man page for disc has been created and the disc devices now appear as disc# instead of ds#. Some work is still needed on pppd to make it understand cloning though it should work as long as the devices are created beforehand.
On the API front, management of mandatory interfaces (i.e. lo0) is handled by the generic cloning code so if_clone_destroy has the same API as NetBSD again and <if>_modevent doesn't need to create the necessary devices manually.
At this point, all pseudo interfaces have been converted to the cloning API or already did their own cloning (sl(4) for example uses it's own mechanism). Some devices such as tun(4) and tap/vmware should probably be converted to use the cloning API instead of their current ad-hoc, devfs based cloning system. This would be a good junior kernel hacker task. Also, the handbook and FAQ could use some general cloning documentation prior to 5.0 release.
We have been updating RELENG_4 targeting for 4.7-RELEASE. When port ja-man-1.1j_5 was broken around the end of July, Kumano-san and Mori-san tried to update the port to be based on a newer FreeBSD base system's man commands. But, we decided only to fix the port ja-man-1.1j_5 to be buildable, as the new one was not complete at that time.
The GEOM code has gotten so far that it beats our current code in some areas while still lacking in others. The goal is for GEOM to be the default in 5.0-RELEASE.
Currently work on a cryptographic module which should be able to protect a diskpartition from practically any sort of attack is progressing.
The UFS2 filesystem approaches feature completion: Extended attribute functionality have been added, including a new compound modification API and basic testing has been passed.
We've got currently almost 50% of the new handbook translated (all the installation part is translated). Most of the articles are translated too.
The web site in on the way, see the Web Server. We need now to integrate it on the US CVS tree.
One of the big job now, is to translate the latest FAQ and the very big project will be the manual pages
I'm very pleased to announce that another engineering release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20020909.tar.gz
This release features several major changes and includes support for H4 UART and H2 USB transport layers, Host Controller Interface (HCI), Link Layer Control and Adaptation Protocol (L2CAP) and Bluetooth sockets layer. It also comes with several user space utilities that can be used to configure and test Bluetooth devices. Also there are several man pages.
Service Discovery Protocol (SDP) is now supported. This release includes SDP daemon, configuration tool and user space library (ported from BlueZ-sdp-0.7).
RFCOMM is now supported. This release includes rfcommd daemon that provides RFCOMM service via pseudo ttys. Not very useful for legacy application, but it is possible to run PPP over Bluetooth now. This was ported from old BlueZ-rfcommd-1.1 (no longer supported by BlueZ) and still has some bugs in it.
Next step is to fix current RFCOMM support and work on new in-kernel RFCOMM and BNEP (Bluetooth Network Encapsulation Protocol) implementation. Also user space need more work (better tools, libraries, documentation etc.).
Version 1.2 has been released recently. It should compile and work an any recent FreeBSD-current. Support to manipulate SUNI registers has been added to the ATM drivers (to switch between SONET and SDH modes, for example). The ngatmsig package now includes a small and simple call control module that may be used to build a simple ATM switch. The netgraph stuff has been patched to use the official netgraph locking.
On the API front, fmtmsg(3) was implemented, glob(3) was given support for new flags, ulimit(3) was implemented, and wide character/string support was significantly improved with the addition of 30 new functions (see the project status board for details). Work is progressing on adding the C99 restrict type-qualifier to functions throughout the system. This allows the compiler to make additional optimizations based on the knowledge that a restrict-qualified argument is the only reference to a given object (ie. it doesn't overlap with another argument).
Several headers have been brought up to conformance with POSIX.1-2001, they include: <fmtmsg.h>, <poll.h>, <sys/mman.h>, and <ulimit.h>. The header <cpio.h> was implemented. The headers <machine/ansi.h> and <machine/types.h> were merged into a single header to help simplify the way variable types are created.
The sh(1) built-in, command(1), was reimplemented to conform with POSIX. Additionally, several utilities which were previously brought up to conformance were merged into the 4-STABLE branch.
The GNOME 2 desktop port has reach version 2.0.2rc1 with an expected 2.0.2 release before 4.7-RELEASE. Mozilla 1.1 has been ported, and is resident in the tree with Mozilla 1.0.1. The GNOMENG porting effort is going well. A good deal of ports have been moved to the new infrastructure with the help of Edwin Groothuis. We are now working on smoothing out some of the rough edges, then, once all the work is done, make GNOMENG the default.
A long-standing annoyance in Nautilus has also been recently corrected. The desktop is no longer cluttered with volume icons, and removable media (such as CDs) should now be handled correctly.
The ATAPI/CAM module allows ATAPI devices (CD-ROM, CD-RW, DVD drives, floppy drives such as Iomega Zip, tape drives) to be accessed through the SCSI subsystem (CAM). ATAPI/CAM has been integrated in -CURRENT. The code should be fairly functional (it has been used by many testers as patches against -STABLE and -CURRENT over the past eight months), but there are pending issues on SMP machines. Testers most welcome.
A MFC of this feature will probably happen after the end of the 4.7 code freeze.
The goal of this project is to import the OpenBSD kernel-level crypto subsystem. This facility provides kernel- and user-level access to hardware crypto devices for the calculation of cryptographic hashes, ciphers, and public key operations. The main clients of this facility are the kernel RNG (/dev/random), network protocols (e.g. IPSEC), and OpenSSL (through the /dev/crypto device).
OpenSSL 0.9.7 beta 3 was imported and patched with fixes from OpenBSD's source tree. This permits any user-level application that use -lcrypto to automatically get hardware crypto acceleration. Otherwise the core crypto support is stable and has been in production use on -stable machines for several months.
Import of this work into the -current tree has started. A publicly available patch against 4.7 will be released once 4.7 ships. Integration of this work into the -stable source tree is planned for 4.8.
The main goal of this project is to modify the IPsec protocols to use the kernel-level crypto subsystem imported from OpenBSD (see elsewhere). A secondary goal is to do general performance tuning of the IPsec protocols.
Recent work focused on increasing performance. Support is still limited to IPv4 protocols, with IPv6 support coded but not yet tested.
Import of this work into the -current tree has started. A publicly available patch against 4.7 will be released once 4.7 ships.
Work is in progress to MFC a number of bug fixes related to vm_map corruption into -stable. This work is probably too involved to make it into the 4.7 release but is expected to be committed just after the freeze is lifted. The corruption in question typically occurs in large-memory systems under heavy loads and typically panics or KPFs (kernel-page-fault's) the machine in a vm_map related function.
The existing SCSI target code has been rewritten. The kernel driver is much simpler, deferring all functionality to usermode and simply passing CCBs to and from the SIM. The supplied usermode emulates a disk (RBC) with IO going to a backing file. It replaces /sys/cam/scsi/scsi_target* and /usr/share/examples/scsi_target.
The code is definitely alpha quality and has known problems on -current although it appears to work ok on -stable. See the included README for how to install and test. Feedback is welcome!
Yet another implementation of Lottery Scheduling devised by Carl Waldspurger et. al. is being developed against FreeBSD -STABLE branch. It is being developed as part of a graduation project in Computer Science at Universidade de Brasília in Brazil. Therefore, other implementations have not yet been verified to avoid plagiarization but will be checked in a later stage of this project searching for better implementation ideas. Currently, part of the necessary scheduling kernel structure has been mapped and work has progressed despite the general lack of kernel documentation. Further outcomes of this project will be a simple documentation of the kernel scheduler structure of -STABLE branch, a port of the Lottery Scheduler to -CURRENT branch and additional implementations of other scheduling disciplines from Carl Waldspurger et. al. Members of the FreeBSD community have been and will continue to be instrumental in both testing and providing feedback for ideas implemented here.
The FreeBSD Brazilian Portuguese Documentation Project is merging with a translation group formed by members of the FUG-BR FreeBSD Brazilian user group. The Brazilian Project decided to become an official group under FUG-BR after receiving continued excellent contributions from them. They have managed to complete the translation of the FreeBSD FAQ which is currently undergoing both proofing and SGML"fication" stages. Work is progressing fast: the Handbook has been half translated and articles are under way. The previous Brazilian Project is proud to become part of such a dedicate group. The contacts above represent the current official contacts for the new translation group. We hope to have at least part of this work ready for the FreeBSD 4.7 Release.
David Xu and I have been working on cleaning up some of the work done in KSE-III and Jonathon and Dan have been working on the userland interface. The userland library will be committed soon in a prototypical state and a working test program using that interface will hopefully accompany it. I have just committed a rework of the run states for kernel threads that simplifies or solves some problems that were being seen recently.
Hopefully in the next few weeks we will be able to run threads on separate processors. The basics of Signal support are presently evolving. Archie Cobbs will also be assisting with some of this work. I have a mail alias for all the developers at kse@elischer.org. It is managed by hand at the moment.
The Release Engineering (RE) Team completed and released FreeBSD 4.6.2. This ``point release'' fixes several important bugs in the ATA subsystem, as well as addressing a number of security issues in the base system that surfaced shortly after FreeBSD 4.6 was released. The release documentation distributed with FreeBSD 4.6.2 contains more details. (Note: Some earlier documents and reports referred to this release as version 4.6.1.) The next release in the 4.X series will be FreeBSD 4.7, which has a scheduled release date of 1 October 2002.
Concurrently, work is continuing on the 5.0-DP2 developer preview snapshot, an important milestone along the release path of FreeBSD 5.0, which is scheduled for release on 20 November. As 5.0 draws closer, we are focusing more on getting the system stabilized, as opposed to adding new functionality. To help us with this effort, developers should discuss with us any new features planned for -CURRENT, beginning 1 October.
The project runs as it should be. New security-branch snapshots are available for both 4.5 and 4.6(.2). I've update buildboxes OS to the latest 5-current/4-stable without any errors. Also current problem, less CPU power for the future, is not solved yet -- but situation is not so bad, I hope I'll show a good news in the next report.
The Donations team started rolling in the last couple of months. Offers of equipment are coming in, and we are allocating them to FreeBSD committers as quickly as possible. We now have a "Committer Want List" available in our section of the Web site. Several small items, such as network cards, have been routed to people who are willing to write the code to support them. We have a few larger donations (i.e., actual servers) ready to go to developers, once shipping information is straightened out.
Work on RAIDFrame stalled for quite a bit, then it picked up in early summer, then it stalled, and now it's going again. A significant amount of work has been done to make the locking SMPng-friendly and to cut down on kernel stack abuse. I'm happy to say that it's starting to work reliably when used with file- backed 'md' disks. Even more exciting is that it's finally starting to work on real disks, too. A lot of cleanup is still needed, and a few gross hacks still exist, but it might actually be ready for the FreeBSD 5.0 release. Patches for FreeBSD 5-current and 4-stable are available from the website. The 4-stable patches are a year old but still apply and perform well.
The primary libh development box, where the CVS repo and development webpage was living, is dead. The server has crashed after a system upgrade and has never came back to life. We had to pull the drives out of it to make proper backups. We will setup another box in place of this one and hope for the best. So right now, the port is broken because the CVS is unaccessible, as the development web page. We're working on it, please bear with us.
On a brighter note, Max started implementing the changes he proposed to the build system and the TCL API; LibH is switching to SWIG for its TCL bindings, which should simplify the system a lot, and shorten build times. The Hui subsystem is therefore being completely re-written. On my side, I made a few tests in building and running LibH under rhtvision, and it didn't fulfill the promises I thought it would, so I just put aside that idea. Work on libh stalled during July because I completely lost network access for the whole month. So right now, LibH is in a bit of a mess, but we have high hopes of settling everything down to a new release pretty soon, which will make full use of the new SWIG bindings.
The Security Team continues to be very busy. The security-officer mailing list traffic for the months of June, July, and August consisted of 1,230 messages (over 13 messages a day). This is well over 50% of the freebsd-hackers traffic volume in the same period!
Since June (the time of our last report), 9 new Security Advisories were published, and one Security Notice was published covering 25 Ports Collection issues.
FreeBSD 4.6.2-RELEASE was released on August 15th. This marked the first time a point release was created from the security branch. The process went smoothly from the Security Team perspective, despite a schedule slippage due to newly discovered bugs, and a snafu which resulted in 4.6.1-RELEASE being skipped.
In September, the FreeBSD Security Officer published a new PGP key (ID 0xCA6CDFB2, found on the FTP site and in the Handbook). This aligned the set of those who possess the corresponding private key with the membership of the security-officer alias published on the FreeBSD Security web site. It also worked around an issue with the deprecated PGP key being found corrupted on some public key servers.
It's been a busy few months, with a variety of development, documentation, and public relations activities. The MAC Framework, our pluggable kernel access control mechanism for FreeBSD, has matured substantially, and large parts of it were merged to the main FreeBSD tree over July and August.
A variety of entry point changes were made, including: component names are now passed to VFS namespace VOPs; aggressive caching of MAC labels in vnodes; mmap memory access downgrades on subject relabel; check for access()/eaccess(); checks for vnode read, write, ioctl, pool, permitting revocation post-open() by aware policies; labeling and access control checks for pipe IPC objects, clean up of socket/visibility checks; checks for socket bind, connect, listen, ....; many locking improvements and assertions, especially for vnodes, processes; framework now supports partial label updates on subjects and objects; credential management in 'struct file' improved so that active_cred and file_cred are more carefully distinguished and passed to MAC framework explicitly; accounting system uses cached credentials for write operations now; socreate() can use cached credential to label sockets fixing deferred nfs socket connections and reconnections with TCP; kse interactions with proc1 fixed; IO_NOMACCHECK flag to vn_rdwr() for internal use to avoid redundant or incorrect MAC checks on aio vnode operations; mac_syscall() policy function demux; su no longer changes MAC labels by default; mac_get_pid() to support ps and getpmac -p pid; mmap revocation defaults to "fail stop"; MAC_DEBUG wraps atomic label counters; UFS2 extended attributes supported; initial port of LOMAC to the MAC framework; update all policies for all these changes; merge of KSE III; merge of nmount(); upgrade of ugidfw to speak user and group names; libugidfw; many namespace and naming consistency improvements; module dependencies on MAC framework; large scale merging of MAC functionality to the main FreeBSD tree. KDE interfaces to common management activities.
Wrote and taught full-day MAC framework tutorial at STOS BSD and Darwin Security Symposium; first draft of MAC framework architecture and API guide. This is now in the Developer's Handbook.
Next couple of months will bring continued maturity improvements, labeling and protection of more objects; VFS performance improvements; better support for UFS2 EAs and separate EA entries for each policy; improved support for LOMAC; MLS compartments; IPsec security association labeling; improved SEBSD FLASK/TE port; and much more.
Another busy pair of months at the FreeBSD Project have brought substantial maturity and feature completeness to the fledgeling 5.0-CURRENT branch. And just in time too, because by the time you read the next status report, we hope that you'll have FreeBSD 5.0 running on your desktop! Over the past two months, we've seen an upgrade of sparc64 to Tier 1 (Fully Supported) status, integration of a high quality storage encryption module, the commit of hardware-accelerated IPsec support, the addition of a general-purpose "Device Daemon" to process hardware attach/detach events to replace earlier single-purpose and bus-specific daemons, the commit of RAIDFrame, and the improved maturity of the TrustedBSD work. We've also seen another successful release of the 4.x branch, 4.7-RELEASE, which will continue to be the production supported platform as 5.X is brought in for landing.
Over the next two months, the FreeBSD Project will be focused almost entirely on making 5.0 a success: improving system stability and performance, as well as increasing the pool of applications that build and run on 5.0. The Release Engineering team will have announced the 5.0 code freeze, and released DP2 by the time you read this. Following DP2 will be a series of Release Candidates (RC's), and then the release itself. If you're interested in getting involved in the testing process, please lend a hand -- a spare box and a copy of the DP and RC ISOs burnt onto CD will make a difference. The normal caveats associated with pre-release versions of operating systems apply! You may also be interested in reading the Early Adopter's guide produced by the Release Engineering team to help determine when a transition from the 4.x branch to the 5.x branch will be appropriate for you and your organization.
Thanks,
Robert Watson, Scott Long
I'm very pleased to announce that another engineering release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20021104.tar.gz
This release features minor bug fixes and new OpenOBEX library port. The snapshot includes support for H4 UART and H2 USB transport layers, Host Controller Interface (HCI), Link Layer Control and Adaptation Protocol (L2CAP) and Bluetooth sockets layer. It also comes with several user space utilities that can be used to configure and test Bluetooth devices. Also there are several man pages.
Service Discovery Protocol (SDP) port has been updated to version 0.8. (ported from BlueZ-sdp-0.8). Most of the RFCOMM issues have been resolved and now rfcommd works with Windows (3COM, Xircom and Widcomm) and Linux stacks.
New supported USB device - EPoX BT-DG02 dongle. Also I have received successful report about Mitsumi USB dongle and C413S Bluetooth enabled cell phone (L2CAP and SDP works, waiting on RFCOMM report).
I'm currently working on OBEX server (Push and File Transfer profiles) which will be based on OpenOBEX library (included in the snapshot).
The BSDCon 2003 Program Committee invites you to contribute original and innovative papers on topics related to BSD-derived systems and the Open Source world. Topics of interest include but are not limited to:
Submissions in the form of extended abstracts are due by April 1, 2003. Be sure to review the extended abstract expectations before submitting. Selection will be based on the quality of the written submission and whether the work is of interest to the community.
We look forward to receiving your submissions!
October 10, 2002 marked the one year anniversary of our project. During that time we have made significant advances in FreeBSD's standards conformance. FreeBSD 5.0-RELEASE will be the showcase for most of our hard work. We hope that our tireless effort has had a positive effect on FreeBSD and software vendors that maintain or are considering porting their software to FreeBSD.
On the API front, _Exit(3) (an alias for _exit(2)) was added, sysconf(3) was update for POSIX.1-2001, and some of the glob(3) additions were MFC'd. The insque(), lsearch(), and remque() family of functions were reimplemented and moved to libc from libcompat. Several wide character functions were implemented, including all printf() and scanf() variants. Finally, support for wide character format types (%C, %S, %lc, %ls) were added to printf(3).
Work on utility conformance continued as getconf(1)'s compliance was updated, c99(1) (a new version of c89(1)) was implemented, and cd(1) and command(1) changes were MFC'd.
Almost 20 headers were brought up to conformance with applicable standards. Not much work remains to fix conformance issues in the remaining standard headers. Work in this area, as well as others, has slowed down in preparation for 5.0-RELEASE.
DEVD has been integrated into FreeBSD current. It was integrated in an incomplete state. However, it is useful in the state that it is in for doing simple things like running camcontrol rescan when a SCSI pcmcia card is inserted, or running /etc/pccard_ether with an ethernet card is inserted. The more sophisticated regular expression matching is not yet complete. Devd only does actions on device arrival and departure, but does not yet do anything with unknown devices. In addition to listening for device events, there is some desire to have /dev/devctl also allow for some direct control of the device tree.
The main goal of this project is to modify the IPsec protocols to use the kernel-level crypto subsystem imported from OpenBSD (see elsewhere). A secondary goal is to do general performance tuning of the IPsec protocols.
This work was committed to -current. To configure it for use specify options FAST_IPSEC in your system configuration file. At present support is limited to IPv4.
GBDE has been committed to -current.
The "Geom Based Disk Encryption" module provides a mechanism for very strong encryption of a GEOM "disk". The algorithm has passed informal review by a couple of seasoned crypto heavy-weights. Any GEOM device can be protected with GBDE, entire physical disks, MBR slices, BSD partitions etc etc. Booting from an encrypted partition is not possible, however.
The focus of GBDE is to protect a "cold" disk media. (FreeBSD is not equipped well for protecting key material on a running system from being compromised.) For a cold media, the only feasible attack on a GBDE protected media is guessing the pass-phrase.
Summary of the GBDE multilevel protection scheme: Up to four separate pass-phrases can unlock their own separate copies of the 2048 bit masterkey. The master-keys are protected using AES/256/CBC keyed with a SHA-2 hash derived from the pass-phrase. A salted MD5 hash over the sectoroffset "cherry-picks" which masterkey bytes participate in the MD5 hash which generates the "kkey" for each particular sector. The kkey AES/128/CBC encrypts the PRNG produced single-use key which AES/128/CBC encrypts the actual sector data.
GBDE has features for master-key destruction and pass-phrase invalidation.
See gbde(4) and gbde(8) for more details.
This software was developed for the FreeBSD Project by Poul-Henning Kamp and NAI Labs, the Security Research Division of Network Associates, Inc. under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA CHATS research program.
The GEOM code is now the default on most (if not all ?) architectures and the few remaining issues in libdisk/sysinstall is being hashed out.
Although we are far from finished developing GEOM, its current feature set is a significant step forward for FreeBSD, providing not only immediate relief for new architectures (sparc64, ia64 etc) but also because it is designed as SMPng code from the start.
This software was developed for the FreeBSD Project by Poul-Henning Kamp and NAI Labs, the Security Research Division of Network Associates, Inc. under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA CHATS research program.
These last two months have seen quite a lot of GNOME activity. GNOME has started releasing development snapshots of the upcoming GNOME 2.2 desktop. FreeBSD porting has begun outside of the main ports tree in the MarcusCom CVS repository. If you are interested in testing the new desktop, follow the instructions on the aforementioned cvsweb URL, and checkout the "ports" module.
Evolution 1.2 is also close at hand. Ximian has posted its first release candidate, 1.1.90, which has been ported to FreeBSD, and is available from the MarcusCom CVS repo listed above. As soon as Ximian officially releases Evolution 1.2, it will be placed in the FreeBSD ports tree.
The Mozilla ports have received numerous updates. We are now tracking all three released Mozilla versions. The mozilla-vendor port is tracking the 1.0.x branch, mozilla is tracking 1.1.x, and mozilla-devel is tracking 1.2.x. The mozilla-devel port now has support for anti-aliased fonts as well as a GTK+-2 interface
Finally, the GNOME team would like to welcome its newest team member, Adam Weinberger. Adam has been submitting patches for both GNOME ports as well as documentation. Currently, he has been active in the GNOME 2.2 porting effort. We are happy to have him.
The goal of this project is to import the OpenBSD kernel-level crypto subsystem. This facility provides kernel- and user-level access to hardware crypto devices for the calculation of cryptographic hashes, ciphers, and public key operations. The main clients of this facility are the kernel RNG (/dev/random), network protocols (e.g. IPsec), and OpenSSL (through the /dev/crypto device).
This work was committed to the -current tree. To configure it for use specify device crypto in your system configuration file or you can load the crypto module. The /dev/crypto device support is brought in with device cryptodev or by loading the cryptodev module. Two crypto device drivers exist: ubsec for Broadcom-based PCI hardware and hifn for Hifn-based PCI hardware.
Integration of this work into the -stable source tree should be completed by the time this report is published.
Since the last status report the BSD Java Porting Team has continued to make steady progress. The most exciting news we have is courtesy of our newest team member, Alexey Zelkin of FreeBSD committer fame.
For 4.7-RELEASE, we privately published package ja-man-doc-4.7.tgz which consists of man[1256789] entries 10 days after the 4.7-RELEASE release date. Man3 update god no progress, as updating other sections busied us. We decided to suspend man3 update officially, as we need to spend most of our time to catch up with the forthcoming 5.0-RELEASE.
The KDE/FreeBSD team has been working on two major goals during the last two months, Maintenance of the KDE 3.0.x ports and Preparing the upcoming KDE 3.1 Release.
Maintenance KDE 3.0 conducted by Alan Eldrige: September started with the Removal of the KDE 2.x Ports from the FreeBSD-Repository. Later Packages of KDE 3.0.4 were released and the FreeBSD Ports were updated.
Preparing for KDE 3.1 conducted by Will Andrews: A lot of effort was spent on Improving the Fruitsalad-Build-System. We are now able to create packages directly from the KDE CVS.
The KSE code has now all the basic kernel functionality to start being used by the userland. There are still things to be done for testing and familiarization.
General system utilities have not yet been changed. e.g. ps and top etc. need to know about threads.
There is quite a lot of code in the kernel that still assumes that there is one thread in a process. Signals are not yet handled in the final manner (though they are delivered to a random thread in the process :-/ ).
The system calls and datastructures are now however in place. The test program successfully starts several threads that can be scheduled on different processors, and closes them down again. The userland is probably going to be able to do simple scheduling of pthread threads using KSE by the time that this report is published.
I still need someone to take over the "official" web page since jason left. LaTeX sure isn't my thing.
Not much since the last status report, except that we now have the repo and development web page back online, thanks to the services of John De Boskey who freely provided the necessary hardware and bandwidth to host the project. We have also ported LibH to GCC 3.x, so that it can compile on -CURRENT correctly. This, however, broke tvision, which doesn't compile under GCC 3.x, so we moved to rhtvision but this caused linking problems so we're stuck with no console front end, for now.
Work on a Hui rewrite and SWIG bindings stalled. Alex was able to come up with a simple patch to make the ports system use LibH's pkg_create script to build libh packages, so we're getting closer to a real pkg_create(1) drop-in replacement. I rewrote the milestone list to show a bit more relevant and encouraging tasks that will be dealt with in order to really push LibH forward.
A mailing list was created, freebsd-mips, and a Perforce branch was created in //depot/projects/mips. Changes which will be necessary to allow multiple MIPS (and PowerPC) metaports to exist under one architecture port were made, and are being pushed back into the main FreeBSD tree. Some preliminary header work has been done, and porting the ARCBIOS interfaces to the kernel has begun. The toolchain in tree was updated and modified in places to support a FreeBSD/MIPS (Big Endian) target, in the Perforce branch. Some early boot code has proven the GDB MIPS simulator to work, for at least R3000 code, though whether R3000 will be supported has been under discussion. Some initial architectural decisions were also made, to steer current work.
Work on newcard continues. A number of bugs have been fixed in the last few months. You are now able to load and unload drivers (including the bridge) to test changes to pccard and/or cardbus bus code. It is now possible to load a driver that has a pccard attachment and have a previously inserted card probe and attach. This is also true for CardBus. A number of issues remain to be solved before 5.0. However, with the integration of devd into the tree nearly all of old functionality of OLDCARD is now present in NEWCARD (the biggest remaining parts are power control for the sockets, as well as pccardc dumpcis).
The PowerPC port has been running diskless on NewWorld G3/G4 machines for a while now. A GEOM module to support Apple Partition Maps is being written. There should be an installable ISO image available in the near future.
RAIDFrame was imported into FreeBSD-current in late October, a major milestone after 18 months. It is still very experimental and not suitable for production environments. The website contains a lengthy TODO list which I hope to start attending to soon. Still, I encourage everyone to try it out and report bugs back to me.
The Release Engineering (RE) team completed and released FreeBSD 4.7 on 10 October 2002. This release features updates for a number of contributed software programs in the base system, as well as all of the security and bug fixes from FreeBSD 4.6.2. The next release in the 4.X series will be FreeBSD 4.8, which has a scheduled release date of 1 February 2003.
Before that time, however, will be the release of FreeBSD 5.0. Thus far, we have not been able to release the 5.0-DP2 developer snapshot due to various stability issues. Thanks to much effort from many of our fellow developers, we believe that most of these have been resolved. The RE team wishes to emphasize that FreeBSD 5.0 will involve new code and features that have not seen widespread testing, and that more conservative users may wish to continue to track the 4.X series for the near-term future. To provide more information on these issues, we have added an Early Adopter's Guide to the release documentation for 5.0.
Brian Somers has resigned from the RE team due to increased time pressures. We thank him for all of his help with FreeBSD 4.5, 4.6, 4.6.2, and 4.7, and we hope to continue working with him as a fellow developer.
Scott Long has graciously offered to help improve the communication between the RE team and the rest of the developer community. We greatly appreciate his assistance.
Recent 5-current release procedure troubles prevent the project from releasing a new snapshots. But 5-current FreeBSD/i386 release is back again in late Oct/2002! I have a plan to build daily FreeBSD/sparc64 snapshots for 5-current. Stay tuned...
A lot has happened recently for the sparc64 port. Sysinstall and make release work and can be used to build installable snapshots. The gdb5.3 port now works, and, thanks to Thomas Moestl, kernel crash dumps are supported which can be analyzed by gdb. These 2 items are the last things considered necessary by the Core team for FreeBSD/sparc64 to be a Tier 1 architecture, which means that 5.0-RELEASE for sparc64 will be officially supported by the release engineering team and by the security officer team.
Recently Jake Burkholder has been working on alternate installation methods other than bootable iso, including a mini-root filesystem which can be written to the swap partition of an existing machine. Thomas Moestl has been putting some finishing touches on the release process, ensuring that the release documentation can be built properly, and that the port readme files can be generated by the release process.
An experimental iso built with make release is now available on the freebsd ftp site and mirrors in /pub/FreeBSD/development/sparc64/5.0-20021031-SNAP. It is expected that by the middle of November new 5.0-SNAP releases will be available every few days for download and for ftp install, cpu power and bandwidth permitting.
Most progress on TrustedBSD over the last two months related to improving the maturity of the ACL and MAC implementations, and merging new aspects of those features into the primary FreeBSD CVS Repository for inclusion in FreeBSD 5.0. This included fixes to run better on sparc64, improved tuning of what system objects are mediated, locking fixes and optimizations especially relating to the vnode and pipe implementations, improved support for MAC labeling on symlinks, support for asynchronous process label changes as required in some locking situations, remove use of "temporary labels" and prefer use of object type specific labels reducing redundant and/or confusing label management code in policies, improve avoidance of memory allocation in M_NOWAIT scenarios for socket allocation in the syncache, mediation of link operations, race condition fixes for devfs involving label creation, improve handling of VM events such as mmaping, improve mediation of socket send/receive events (as distinguished from socket transmit/deliver events), support for manipulating EAs on symlinks using new system calls, support for MNT_ACLS and MNT_MULTILABEL flags at mount time, as well as FS_ACLS and FS_MULTILABEL superblock flags to key useful defaults using tunefs, correction of a memory leak in the UFS ACL code, enable UFS ACL support by default in GENERIC, mediation points for file creation, deletion, and rename, support for a mac_execve() execution interface in the style of SELinux's execve_secure() permitting a label transition request as part of the exec operation for policies that support it, more consistent handling of NFS lookups, support for labeling of multicast encapsulated packets, ATM packet labeling, FDDI packet labeling, STF packet labeling, revised label interface that avoids userland parsing of per-policy elements, reducing us to a single instance of parsing and printing for each policy (and further abstracting policy implementation details from the library code).
Also, change to single-level sockets for Biba and MLS policies, support for partial label updates for Biba and MLS, addition of mac.9 man page, revised user API system calls, implementation of mac_get_pid(), and various other related bits, creation of mac.conf(5) to specify label defaults, checks for various system operations including swapon(), settime(), and sysctl(), reboot(), acct(), introduction of command line utilities for maintaining file and process labels, support for user labels tied to login class, su support for label changes, ifconfig support for interface labels, ps support for process labels, ls support for file labels, ftpd support for login labels, development of the Biba and MLS notions of privilege, and a move to C99 sparse structure initialization, restoring full type checking for policy entry points.
Universally Unique Identifiers (UUIDs) are 128 bit values that may be generated independently on separate nodes (hosts), which result in globally unique strings. UUIDs are also known as Globally Unique Identifiers (GUIDs). The UUID support for FreeBSD (libc) conforms to the DCE 1.1 RPC specification.
UUID support has been added to FreeBSD -CURRENT, and will be available in version 5.0. It is being extensively used in GPT partition handling for IA-64 platform. For now, a simple manual page has been provided, which outlines information about the provided uuid routines. Many documentation additions and enhancements to uuidgen(1) are in the pipeline.
The goal of this project is to improve the wireless networking support in the system. The initial work will incorporate the 802.11 link layer done by Atsushi Onoe for NetBSD. This core support code implements the basic 802.11 protocols required for Station and AP operation in BSS, IBSS, and Ad Hoc modes of operation. Wireless device drivers will then be revised to use this common code instead of their private implementations.
Following this initial stage the wireless networking support will be extended to support functionality needed for workgroup, enterprise, and metropolitan (e.g. mesh) networking environments. This will include full power management support, the 802.1D spanning tree protocol for running multiple AP's in a bridged configuration, QoS support, and enhanced security protocols (LEAP, AES, EAP). Support for new hardware devices is also planned.
At long last, FreeBSD 5.0 is here. Along with putting the final polish on the tree, FreeBSD developers somehow found the time to work on other things too. IA64 took some major steps towards working on the Itanium2 platform, an effort was started to convert all drivers to use busdma and ban vtophys(), hardware crypto support and DEVD hit the tree, NewReno was fixed and effort began on locking down the network layer of the kernel. Also high performance, modular scheduler started taking shape and will be a welcome addition to the kernel soon.
Looking forward, the focus will be on stabilizing and improving the performance of 5.0. The RELENG_5 (aka 5-STABLE) branch will be created once we've reached our goals in this area, so hopefully we will get there quickly. Meanwhile, preparations for the next release from the 4.x series, 4.8, will begin soon. Of course, the best way to get 5.x to stabilize os to install and run it!
Thanks,
Scott Long, Robert Watson
I'm very pleased to announce that all kernel modules and few userland tools made it to the FreeBSD source tree. Many thanks to Julian Elischer.
Unfortunately no big changes since the last report. Some minor problems have been discovered and patches are available on request. I will prepare all the patches and submit them to Julian for review.
OBEX server and client (based on OpenOBEX library) is almost complete. I'm currently doing interoperability testing. If anyone has hardware and time please contact me. The HCI security daemon has been implemented and tested with Sony Ericsson T68i cell phone and Windows stack. It is now possible to setup secure Bluetooth connections.
A few people have complained about RFCOMM daemon. These individuals want to use GPRS and Bluetooth enabled cell phone to access Internet. If you have this problem please contact me for possible workaround. My next goal is to get robust RFCOMM implementation to address all these issues.
Largely bug-fixing and userland application tweaks; new interfaces were added to manipulate ACLs on extended attributes; bugs were fixed in ls relating to ACL flagging. Patches to teach cp, mv, gzip, bzip, and other apps about ACL preservation are in testing and review. tunefs flags were added to ease configuration of ACLs, especially on UFS2 file systems.
Possible changes to make use of Linux/Solaris umask semantics are under consideration: right now we implement verbatim POSIX.1e/IRIX merging of the umask, ACL mask, and requested creation mode during file, device, fifo, and directory creation. Solaris and the most recent Linux patches ignore the umask in the context of a default ACL; this requires some rearrangement of umask handling in our VFS, although the results would be quite useful. We're exploring how to do this in a low impact way.
Framework changes:
Instrument KLD system calls (module and kld load, unload, stat) Instrument NFSd system call. Instrument swapoff(2). Instrument per-architecture privileged parts of sysarch(). Make use of condition variables to allow callers to wait for the framework to "unbusy" when loading/unloading policies, rather than returning EBUSY. Store mount pointer in devfs_mount structure for use by policies. Improve handling of labels in loopback interface "re-align" packet copy case. Provide full paths on devfs object creations to help policies label them properly (not merged). Experimentation with moving MAC labels into m_tags (not merged). NFS server now uses real ucreds, not hacked up ucreds, meaning we can start laying the groundwork for enforcement on NFS operations. (not merged)
Policy changes
LOMAC: mac_lomac replaces lomac (LOMAC now uses the MAC Framework), SEBSD: Improved support for devfs labeling based on SELinux genfs. Handling of hard link checks. Support export of process transition information for login and others using sysctl. Login now prompts for roles. Allow policy reload. TTY labeling. Locking adaptation from Linux. Many, many policy adaptations and fixes. We can now boot in enforcing mode! mac_bsdextended: fix a bug in which VAPPEND wasn't mapped to VWRITE, so opens with the O_APPEND bug failed improperly.
Userland changes
setfmac(8) now supports a setfsmac(8) execution mode, which accepts initial labeling specification files. Supports an SELinux compatibility mode so it can accept SELinux label specfiles using the SEBSD module. sendmail(8) now sets user labels as part of the context switch for mail delivery.
Documentation changes
Man page updates for MAC command line tools, modules, admin hints, etc. Updates to the FreeBSD Developer's Handbook chapter on MAC policies and entry points. MAC section in FreeBSD Handbook.
This project has been coming along pretty well. The amd(4) and xl(4) drivers have now been converted to use the busdma API, sparc64 got the bus_dmamap_load_mbuf() and bus_dmamap_load_uio() functions, and the gem(4) and hme(4) drivers have been updated to use bus_dmamap_load_mbuf() instead of bus_dmamap_load().
A lot more still needs to be done, as shown on the project's page. A fair number of conversions are on their way though, and we can expect a fair number of drivers to be converted soon, thanks to all the developers who are working on this project.
The POSIX Utility Conformance in FreeBSD list (link above) has been updated to reflect current reality. Not much work remains to complete base utility conformance.
On the API front, grantpt(), posix_openpt(), unlockpt(), wordexp(), and wordfree() were implemented. The header <wordexp.h> was added.
There are currently about 40 unassigned tasks on our project's status board ranging from documentation, utilities, to kernel hacking. We would encourage any developers looking for something to work on to check out the status board and see if anything interests them.
The goal of this project is to import the OpenBSD kernel-level crypto subsystem. This facility provides kernel- and user-level access to hardware crypto devices for the calculation of cryptographic hashes, ciphers, and public key operations. The main clients of this facility are the kernel RNG (/dev/random), network protocols (e.g. IPsec), and OpenSSL (through the /dev/crypto device).
This work will be part of the 5.0 release and has been committed to the -stable source tree for inclusion in the 4.8 release.
Recent work has focused on improving performance. System statistics are now maintained and an optional profiling facility was added for analyzing performance. Using this facility the overhead for using the crypto API has been significantly reduced.
The ubsec (Broadcom) driver was changed to significantly improve performance under load. In addition several memory leaks were fixed in the driver and the public key support was enabled for use.
Upcoming work will focus on load-balancing requests across multiple crypto devices and integrating OpenSSL 0.9.7 which will automatically enable application use of crypto hardware.
Devd has been integrated into FreeBSD 5.0-RELEASE. The integrated code supports a range of configuration options. The config files are fully parsed now and their actions are performed.
Future work in this area is likely to be limited to improving the devctl interface. /dev/devctl likely will be a cloneable device in future versions. Individual device control via devctl is also planned.
The Donations project expedited several dozen donations during 2002, and was able to place most of what was offered. We still are in dire need of SMP and Sparc systems. You can see information on our needs and donations that have been handled by the team on the donations web page.
We are relying increasingly upon the developer wantlist to place items offered to the Project, and using the commit statistics to help place items. As such, active committers who ask for what they want beforehand have a decent chance of getting it. Less active committers, and committers who do not ask for what they want, will be lower in our priorities but will not be excluded.
We are in the process of streamlining the tax deduction process for donations, and hope to have news on that shortly. We are also always working to accelerate and reduce our internal processes, to get the most equipment in the hands of the most people as quickly as possible.
I especially want to thank David O'Brien and Tom Rhodes for stepping up and making the team far more successful. Also, the FreeBSD Foundation has been quite helpful in handling tax-deductible contributions.
The main goal of this project is to modify the IPsec protocols to use the kernel-level crypto subsystem imported from OpenBSD (see elsewhere). A secondary goal is to do general performance tuning of the IPsec protocols.
This work will be part of the 5.0 release. Performance has been improved due to work on the crypto subsystem.
The goal of the project is to use a small amount of space in the FFS superblock to store a volume label of the user's choice. A GEOM module will then expose the volume labels into a namespace in devfs. The idea is to make it easier to manage filesystems across disk swaps and movement from system to system.
At this point, everything pretty much works. I've submitted parts of the patch to respective subsystem maintainers for review. There are some issues with namespace collision that I haven't addressed yet, but the basic functionality is there
Most of the articles are translated too. Marc is still translating the handbook, 60% is currently translated. Stéphane has began the integration of our French localization web site in the US CVS Tree. Sébastien is still maintaining the Release Notes.
We launched a new site, www.FreeBSD-fr.info, consisting in a French Daemon News like site. Netasq have donated our new server; we will install it in a new hosting provider in the few next weeks. One of the big job now is the translation of the FAQ, and the big project will be the manual pages.
Since the ports tree has been frozen for most of this reporting period, there have not been too many GNOME updates going into the official CVS tree. However, development has not stopped. GNOME 2.2 is nearing completion, and quite a few FreeBSD users have stepped up to test the GNOME 2.1 port sources from the MarcusCom CVS repository. If anyone else is interested, follow the instructions on the aforementioned cvsweb URL, and checkout the "ports" module.
The upcoming FreeBSD 5.0-RELEASE will be the first release to have the GNOME 2.0 desktop as the default GNOME desktop choice. During the previously mentioned ports freeze, all the GNOME 2 ports were fixed up so that they build and package on both i386 and Alpha platforms. Alas, the one port that will not make the cut for Alpha is Mozilla. There are still problems with the xpcom code, but work is ongoing to get a working Alpha port.
Finally, the FreeBSD Mono (an OpenSource C# runtime) port has also received some new life. Mono has been updated to 0.17 (the latest released version), and Juli Mallett has ported gtk-sharp (GTK+ bindings for C#).
The ia64 port is up and running on the new Itanium2 based hp machines thanks to a lot of hard work by Marcel Moolenaar. So far we are running on the hp rx2600 as these were the machines graciously donated by Hewlett-Packard and Intel. We had a prototype Intel Tiger4 system for a while, but we had to return the machine and we do not know if it currently runs. Most of the changes necessary to run these are sitting in the perforce tree and are not in the -current or RELENG_5 cvs tree. As a result, the cvs derived builds (-current and the 5.0-RC series and presumably 5.0-RELEASE) are only usable on obsolete Itanium1 systems.
Lots of other stability and functionality fixes have been made over the last few months, including initial libc_r support. The OS appears to be stable enough for sustained workloads - it is building packages now, for example. We still do not have gdb support, even for reading core files.
We have been updating our Japanese translated manual pages to RELENG_5 based. All existing entries have been updated, but 15 exceptions are not, most of which require massive update. We will also need to add translations which did not exist on RELENG_4.
KGI (Kernel Graphic Interface) is a kernel infrastructure providing user applications with means to access hardware graphic resources (dma, irqs, mmio). KGI is already available under Linux as a separate standalone project. The KGI/FreeBSD project aims at integrating KGI in the FreeBSD kernel.
KGI/FreeBSD has been recently donated 2 PCI graphic cards (Matrox Millenium II and a coming Mach64) and other have been proposed. Please see the FreeBSD web pages for details. Thanks to donation@ for organizing and promoting donations. Thanks to the donators for their contribution to KGI/FreeBSD.
KGI/FreeBSD progressed fine the last months. Most of the VM issues for mapping HW resources in user space have been addressed and a first attempt of coding was made. This prototyping raised some API compatibility problems with the current Linux implementation and was discussed heavily on the kgi devel lists. Ask if you're interested in such issues, I'll be pleased to share them.
Most of coding is now done. Let's start debugging!
Work is ongoing to continue to lock up the network stack. Recently, the focus has been on the IP stack. The plan there involves a series of inter-related pieces to lock up the ifaddr ref count, the inet list, the ifaddr uses, the ARP code, the routing tree, and the routing entries. We are over 3/5 of the way done down this path.
In addition to TCP and UDP, the other networking protocols such as raw IP, IPv6, AppleTalk, and XNS need to be locked up. Around 1/4 these remaining protocols have been locked and will be committed after the IP stack is locked.
The protocol independent socket layer needs to be locked and operating correctly with the protocol dependent locks. This part is mostly done save for much needed testing and code cleanup.
Finally, a pass will be need to be made to lock up the devices drivers and various statistics counters.
This effort fixes some outstanding problems in our TCP stack with regard to congestion control. The first item is to fix our NewReno implementation. Following that, the next urgent correction is to fix a problem involving window updates and dupack counts. When that stabilizes, we will then change the recovery code to make use of SACK information. Eventually, this project will update the BSD stack to add Limited Transmit and other new internet standards and standards-track improvements.
The 3 FreeBSD package clusters (i386, alpha, sparc64) have been unified to run from the same master machine, instead of using 3 separate masters. This has freed up some machine resources to use as additional client machine, as well as simplifying administrative overheads. Build logs for all 3 architectures can now be found on the http://bento.FreeBSD.org webpage. The sparc64 package cluster now has 3 build machines (an u5 and two u10s), and an ia64 cluster is about to be created.
Package builds now keep track of how many sequential times a port has failed to build (html summaries are available on the bento website). This allows tracking of ports which have suddenly become broken (e.g. due to a bad upgrade, or due to changes in the FreeBSD source tree), and in the future will be used to send out notifications to port maintainers when their port fails to build 5 times in a row. This feature is currently experimental, and further code changes will be needed to stabilize it.
The goal of this project is to improve the wireless networking support in the system. By the time of this report the 802.11 link layer code should be committed. A version of the wi driver that uses this code should be committed shortly. Conversion of other drivers is planned as are drivers for new devices.
Support for 802.1x/EAP is the next planned milestone (both as a supplicant and authenticator).
November and December were especially busy for the release engineering team. Scott Long joined the team to help with secretary and communications tasks while Brian Somers bowed out to focus on other projects.
FreeBSD 5.0-DP2 was released in November after much delay and anticipation, and marked the final milestone needed for 5.0 to become a reality. Shortly after that, we imposed a code freeze on the HEAD branch of CVS and released 5.0-RC1. Creation of the RELENG_5_0 branch came next, followed by the release of 5.0-RC2 from this branch. At this point, enough critical problems still existed that we scheduled an RC3 release for the new year, and pushed the final 5.0-RELEASE date to mid-January. By the time this is published, FreeBSD 5.0-RELEASE should be a reality.
For the time being, there will not be a RELENG_5 (aka 5-STABLE) branch. FreeBSD 4.x releases will continue, with 4.8 being scheduled for March 2003. Release in the 4.x series will be lead by Murray Stokely, and releases in the 5.x series will be lead by Scott Long. Once HEAD has reached acceptable performance and stability goals, the RELENG_5 branch will be created and HEAD will move towards 6.0 development. We hope to reach this with the 5.1 release this spring.
A new scheduler will be available as an optional component along side the current scheduler in the 5.1 release. It has been designed to work well with KSE and SMP. Some ideas have been borrowed from solaris and linux along with many novel approaches. It has O(1) performance with regard to the number of processes in the system. It also has cpu affinity which should provide a speed boost for many applications.
The scheduler has a few loose ends and lots of tuning before it is production quality although it is quite stable. Please see the post to arch and subsequent discussion for more details.
Another busy two months have passed in the FreeBSD project. With 5.0 released, attention is focusing on making it faster via more fine-grained locking, adding more high-end features like large memory (PAE) support for i386, and further progress on many other projects. FreeBSD 5.1 is expected to ship in late May or early June, with 5.2 following at the end of summer. A roadmap for the push to 5-STABLE is available at http://www.FreeBSD.org/doc/en/articles/5-roadmap. Although the 5.x series isn't expected to fully stabilize until the 5.2 release, 5.1 promises to be an exciting release and a significant improvement over 5.0 in terms of speed and stability.
Not to be forgotten, FreeBSD 4.8, the latest in the 4-STABLE series, is nearing release. Lots of last minute work is going into to it to deliver features like XFree86 4.3.0, Intel HyperThreading(tm) support, and of course many more bug fixes. Don't forget to support the FreeBSD vendors and developers by buying a copy of the CD set when it comes out!.
Thanks,
Scott Long, Robert Watson
Large portions of headers have been filled in, all have been stubbed out. Minimal functions and data elements have been stubbed out or filled in. Machinery added to support some requisite tunables for building real kernels. GCC fixed to generate correct local label prefixes making it possible to link real kernels. Work begun on providing enough to create and boot real kernels, on real hardware. Decision to only support MIPS-III and above made.
The BSDCon 2003 Program Committee invites you to contribute original and innovative papers on topics related to BSD-derived systems and the Open Source world. Topics of interest include but are not limited to:
Submissions in the form of extended abstracts are due by April 1, 2003. Be sure to review the extended abstract expectations before submitting. Selection will be based on the quality of the written submission and whether the work is of interest to the community.
We look forward to receiving your submissions!
I'm very pleased to announce that another release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20030305.tar.gz
This release features new in-kernel RFCOMM implementation that provides SOCK_STREAM sockets interface. This makes old user-space RFCOMM daemon obsolete. People should not use old user-space RFCOMM daemon any longer. The release features new RFCOMM PPP daemon that supports DUN and LAN profiles. Note: PPP patch (support for chat scripts in -direct mode) is required for DUN support. Look for it in the mailing list archive or contact me directly. People with Bluetooth enabled cell phones can now use them to access Internet.
The Bluetooth sockets layer has been cleaned up. People should not see any WITNESS complaints with new code. Locking issues have been revisited and code in much better shape now, although it probably is not 100% SMP ready just yet. The code should work on SMP system anyway because sockets layer is still under Giant.
The simple OBEX server and client (based on OpenOBEX library) is complete. OBEX File Push and OBEX File Transfer profiles work and have been tested with Sony Ericsson T68i cell phone and Bluetooth 3COM stack on Windows2K. It is now possible to send pictures, address book and calendar entries from the cell phone via Bluetooth. Minor bug in OpenOBEX library has been fixed and OPEX Put-Empty command now works.
Due to changes in API userland tools must be in sync with the kernel. People should install new include files, recompile and reinstall all userland tools as part of upgrade. I'm sorry about that.
The FreeBSD 4.8 Release Process is well underway. The RELENG_4 branch has been under code freeze since February 15, and the first release candidates were made available in early March. A testing guide has been put together and is available from http://www.FreeBSD.org/releases/4.8R/qa.html.
Developers should coordinate with re@FreeBSD.org about any changes they would like to include in this release, and users are encouraged to try out the release candidates and help find as many bugs as possible now, before the final release is made.
FreeBSD 4.8 represents the newest production release from the stable '4.X' branch. It does not include all of the features that were made available in the "new technology" 5.0 release in January.
The doceng@ team is a new body to handle some of the meta-project issues associated with the FreeBSD Documentation Project. The main responsibilities of this team are to grant approval of new doc committers, to manage the doc release process, to ensure the documentation toolchains are functional, to maintain the doc project primer, and to maintain the sanctity of the doc/ and www/ trees. The current members of this team are Nik Clayton, Ruslan Ermilov, Jun Kuriyama, Bruce A. Mah, and Murray Stokely.
The later months have been very busy on KGI. Most of the framework has been debugged for typical usage (fb, no accel). I got KII (the input interface) connected to syscons through atkbd. Opening /dev/graphic works and framebuffer resource access is permitted. Finally, the KGIM (KGI module) framework has a better building tree for board / monitor drivers and board drivers are now loading with resource allocation.
Most important on the TODO list: 5.0-RELEASE move (I currently work with a May-2002 5.0-current). Most of debug is now done. Let's validate!
Note that KGI project homepage has changed since the last report.
We have released Japanese translation of 5.0-RELEASE online manual pages on February 2nd. Most of entries which did not exist on RELENG_4 were not yet translated. I hope we can finish such entries soon.
We have the first disk device driver (aac) out from under Giant now, and in certain scenarios it gives improvements up to 20%. The device driver API was pruned to reflect that NO_GEOM compatibility is unnecessary, this resulted in approx 1000 lines less source code, the majority of which were removed from the device drivers. The new API for cdevsw is a lot simpler and hopefully less likely to confuse people. The ability to automatically allocate a device major number has been introduced and is already used by a handful of drivers. Checks introduced with this facility has shown that the uniqueness of manually allocated major numbers had already broken down.
Work continues on the statistics collection API and on a unified API for manual configuration of GEOM nodes.
Support for PAE is mostly complete, and has been checked into the jake_pae branch. The approach that is being taken to add support for PAE is to allow the pmap module to view the page table directory as 4 pages instead of 1, and to avoid using the 3rd level structure, the page directory pointer table, as much as possible. Due to its small size, 32 bytes, the PDPT cannot be uniformly recursively mapped, and as such does not provide a regular multi level structure like the page tables used by the alpha or x86-64 architectures. What remains to be done for PAE support is to develop an API for manipulating page table entries which will allow idempotent 64 bit loads and stores to be used where necessary.
Experimental support for >4G ram using PAE has been developed and checked into the jake_pae_test branch in Perforce. This involved adding a physical address type separate from virtual addresses, for use by the vm system and bus code which needs to use physical addresses directly. Initial testing has shown good results with device drivers that can dma to 64 bit physical addresses.
Funding for this project is being provided by DARPA and Network Associate Laboratories, and hardware support by FreeBSD Systems.
In the period from September 2002 through February 2003, the FreeBSD Security Team email aliases saw 1297 messages, a much smaller volume than over the summer (remember the Apache and OpenSSL worms? 4.6.1 oops I mean 4.6.2-RELEASE?).
Also during this period: 95 items were added to the SO issue-tracking database; 39 of these involved the FreeBSD base system while the rest involved ports. 9 new Security Advisories were published, 2 of which covered issues unique to FreeBSD.
In January, the SO published a new PGP key (ID 0xCA6CDFB2, found on the FTP site and in the Handbook). This aligned the set of those who possess the corresponding private key with the membership of the security-officer alias published on the FreeBSD Security web site. It also worked around an issue with the deprecated PGP key being found corrupted on some public key servers.
In February, Mike Tancsa of Sentex donated two machines to the Security Officer. These have been a great help already in testing the security branches, preparing patches, and generating updated binaries. Thank you very much, Mike!
FreeBSD 4.8-RELEASE will continue in the tradition of 5.0-RELEASE, and include GNOME 2 as the default GNOME desktop. This means that 4.8 will ship with GNOME 2.2.
Following on the heels of the recent GNOME 2.2 release, GNOME 2.3 snapshots are gearing up. The development schedule is available from http://www.gnome.org/start/2.3/. Ports will be made available the same way they were for the 2.1 development releases. Stay tuned to freebsd-gnome@ for more details.
We are currently in another ports freeze in preparation for 4.8-RELEASE. Following the freeze, a new bsd.gnome.mk will be committed that effectively removes the USE_GNOMENG macro. This new version will add support for GNOME 2 as well as setup backward compatibility for ports that have not yet been converted to the new GNOME infrastructure. People interested in testing this new Mk file, can check out the ``ports'' module following the instructions at http://www.marcuscom.com:8080/cgi-bin/cvsweb.cgi.
Work on PowerPC is progressing steadily. The system can now boot multi-user from the net and disk. ATA-DMA is being integrated with the ATAng code, and support for older G3 machines is being added.
January and February were quiet months that saw with them the addition of some C99 math functions and macros, which include: fpclassify(), isfinite(), isgreater(), isgreaterequal(), isinf(), isless(), islessequal(), islessgreater(), isnan(), isnormal(), and signbit(). Additional C99 math library support is in the works.
Most of the file system buffer cache has been reviewed and protected. The vnode interlock was extended to cover some buffer flag fields so that a separate interlock was not required. The global buffer queue data structures were locked and counters were converted to atomic ops. The BUF_*LOCK functions grew an interlock argument so that buffers could be safely removed from the vnode clean and dirty lists. The lockmgr lock is now required for all access to buf fields. This was not strictly followed before because splbio provided the needed protection.
There are a few areas of code that need to be protected and cleaned up before giant can be pushed down. Most notably the background write code is currently unsafe without giant. Also, many of the VM bits that the buffer cache relies on are not safe. This work has been done with the expectation that the VM and VFS subsystems will be giant free soon.
The ULE scheduler has been committed to the 5.0-CURRENT branch. Early adopters and experimenters are welcome to try it and submit bug reports. It has shown noticeable performance improvements over the old scheduler under some workloads. There are currently problems with nice fairness but otherwise the interactive performance is very good. More work to improve the load balancing algorithm is required as well. This should be ready for use by the general FreeBSD user base in the next month or so.
Some improvements have been made to the clustered read ahead code. They allow for many more outstanding IO requests when an application does sequential access. This has a larger impact on RAID systems than on single disk systems. The maximum number of file system blocks that we will read ahead is tunable via the 'vfs.read_max' sysctl. This optimization has shown a 20% improvement in simple tests.
Locking of the non-obj parts of newbus is nearing completion. A single lock is used for the device tree. Minimal changes to subr_bus have so far been necessary to make this work, however some lock order issues remain. After this work, it will no longer be necessary to hold Giant to call device_* routines safely. kobj work is being done by others and will likely require more extensive design work to make SMP friendly.
The objective of this effort is to improve the performance, stability, and correctness of the BSD networking stack by adding support for new standards and standards track proposals while maintaining compliance with existing specifications. The upcoming 4.8 and 5.1 releases will be the first ones using the new NewReno logic. Recently, we implemented the Limited Transmit algorithm (RFC 3042) which benefits connections with small congestion windows, as happens, for example, on many short web connections. We also recently added support for larger sized starting congestion windows as described in RFC 3390. This helps short TCP connections as well as those with large round-trip delays, such as those over satellite links.
The list of subsystems locked up include IP, UDP, TCP, ifaddr reference counting, syncache, the ifnet list, routing radix trees, and ARP. These have already been committed into the tree. In addition, SMP locking for raw IP, divert socket processing, and Unix domain sockets have also recently been completed and tested. Work is currently being done in some of the subsystems required to make parallel networking processing SMP-safe.
The FreeBSD Bi-monthly status reports are back! In this edition, we catch up on seven highly productive months and look forward to the end of 2003.
As always, the FreeBSD development crew has been hard at work. Support for the AMD64 platform quickly sprang up and is nearly complete. KSE has improved greatly since the 5.1 release and will soon become the default threading package in FreeBSD. Many other projects are in the works to improve performance, enhance the user experience, and expand FreeBSD into new areas. Take a look below at the impressive summary of work!
Scott Long, Robert Watson
Still in the planning stage. Working on creating an extensible interface that is usable for both userland and kernel implementations for device drivers. Deciding on how to interface userland implemented device drivers with applications.
KSE seems to be working well on x86, amd64, and ia64. The alpha userland bits are done, but a couple of functions are unimplemented in the kernel. For sparc64, the necessary functions are implemented in the kernel, but the userland context switching functions need more attention.
Since 5.1, efficient scope system threads (no upcalls when they block) have been implemented, and KSE based pthread library can have both POSIX scope process threads and scope system threads. It is also possible that KSE based pthread library can implement pthread both in 1:1 and M:N mode, I know Dan has such Makefile file patch for libkse not yet committed.
KSE program now can work under ULE scheduler, its efficient should be improved under the new scheduler in future. BSD scheduler is still the best scheduler for current KSE implement.
Much has happened since the last bi-monthly report, which was more than half a year ago. FreeBSD 5.0 and FreeBSD 5.1 have been released for example. With FreeBSD 5.2 approaching quickly, we're not going to look back too far when it comes to our achievements. There's too much ahead of us...
Two milestones have been reached after FreeBSD 5.1. The first is the ability to support both Intel and HP machines with sources in CVS. This due to a whole new driver for serial ports, or UARTs. Unfortunately this still implies that syscons is not configured. That's another task for another time, but keep an eye on KGI/FreeBSD... The second milestone is the completion of KSE support. Both M:N and 1:1 threading is functional on ia64 and the old libc_r library has been obsoleted. Testing has shown that KSE (i.e. M:N) may well become the default threading model. It's looking good.
The ABI hasn't changed after 5.1 and the expectation is that it won't change much. This means that we can think about becoming a tier 1 platform. This also means we need gdb(1) support. Work on it has been started but the road is bumpy and long. Kernel stability also has improved significantly and we typically have one kernel panic remaining: VM fault on no fault entry. This will be addressed with the long awaited PMAP overhaul (see below).
Most work for FreeBSD 5.2 will be "sharpening the saw". Get those loose ends tied. This is a slight change of plan made possible by a slip in the release schedule. The 5.2 release is not going to be the start of the -stable branch; it has been moved to 5.3. So, we use the extra time to prepare the ground for 5.3.
The planned PMAP overhaul will probably be finished after 5.2. This should address all known issues with SMP and fix those last panics. As a side-effect, major performance improvements can be expected. More news about this in the next status reports.
The following items are in progress in the Disk I/O area: Turn scsi_cd.c into a GEOM driver. (Patch out for review). Turn atapi-cd.c into a GEOM driver. Turn fd.c into a GEOM driver. Move softupdates and snapshot processing from SPECFS to UFS/FFS. Move userland access to device drivers out of vnodes.
Once these preliminaries are dealt with, scatter/gather and mapped/unmapped support will be added to struct bio/GEOM.
FreeBSD Update is a system for tracking the FreeBSD release (security) branches. In addition to being faster and more convenient than source updates, FreeBSD Update also requires less bandwidth and is more secure than source updates via CVSup. However, FreeBSD Update is limited; it can only update files which were installed from an official RELEASE image and not recompiled locally. Right now I'm publishing binary updates for 4.7-RELEASE and 4.8-RELEASE; since my only available box takes 3.5 hours to buildworld, I don't have enough resources to do any more than that.
In the near future, I'd like to: Find someone who is willing to donate a faster buildbox; start building updates for other releases (at a minimum, for all "supported" FreeBSD releases); add warnings if a file would have been updated but can't be updated because it was recompiled locally; add code to compare the local system against a list of "valid" MD5 hashes for intrusion detection purposes; and add support for cross-signing, whereby several machines could build updates independently to protect against buildbox compromise.
The project started this spring and released version 1.0 with a port installation (security/pf) in may 2003. Version 2.0 is on the doorstep as OpenBSD 3.4 will be released. Due to the porting efforts we were able to reveal some bugs in the OpenBSD code and provided locking for the PFIL_HOOKS, which we utilize. Tarball installation of a loadable kernel module for testing can be found on the project homepage, a patchset is in the making.
PF was started at OpenBSD as a substitute for ipfilter and provides the same function set. However, in the two years it exists now, it has gained many superior features that no other packet filter has. For a impression take a look at the pf FAQ.
We hope to be eventually integrated into the base system. Before that we have to resolve some issues with tcpdump and kame.
I'm very pleased to announce that another release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20030908.tar.gz. I have also prepared patch for the FreeBSD source tree. The patch was submitted for review to the committers.
Fixed few bugs in kernel modules. The ng_hci(4) and ng_l2cap(4) modules were changed to fix issue with Netgraph timeouts. The ng_ubt(4) module was changed to fix compilation issue on -current.
Improved user-space utilities. Implemented new libsdp(3). Added new sdpcontrol(8) utility. The rfcomm_sppd(1), rfcomm_pppd(8) and obexapp(1) were changed and now can obtain RFCOMM channel via SDP from the server. The hccontorol(8) utility now has four new commands. The hcsecd(8) daemon now saves link keys on the disk.
I've been recently contacted by few individuals who whould like to port current FreeBSD Bluetooth code to other BSD systems (OpenBSD and NetBSD). The work is slowly progressing towards un-Netgraph'ing current code. In the mean time Netgraph version will be the primary supported version of the code.
The rescue build infrastructure has been committed. There is one known issue with make using both the '-s' and '-j' flags that appears to be a bug in make. Anyone interested in tracking down should contact us.
Support for a dynamically linked /bin and /sbin has been committed, although it is not turned on by default. Adventurous users can try it out by building /bin and /sbin using the WITH_DYNAMICROOT make flag. More testing is needed to determine if this is going to be default for 5.2-RELEASE. If anyone would like to benchmark worldstones with and without dynamically linked /bin and /sbin, please feel free to do so and submit the results.
Work is continuing on updating ACPI with new features as well as bugfixing. A new embedded controller driver was written in July with support for the ACPI 2.0 ECDT as well as more robust polling support. Also, a buffer overflow in the ACPICA resource list handling that caused panics for some users was fixed. Marcel helped get acpidump(8) tested and basically working on ia64.
Upcoming work includes integrating ACPI notifies with devd(8), committing user-submitted drivers for ASUS and Toshiba hotkeys, Cx processor sleep states (so my laptop doesn't burn my lap), and power resource support for intelligently powering down unused or idle devices.
Users who have problems with ACPI are encouraged to submit a PR and email its number to acpi-jp@jp.FreeBSD.org. Bug reports of panics or crashes have first priority and non-working features or missing devices (except suspend/resume problems) second. Reports of failed suspend/resume should NOT be submitted as PRs at this time due to most of them being a result of incomplete device support that is being addressed. However, feel free to mail them to the list as any information is helpful.
The uart(4) project was born out of the need to have a working serial interface (i.e. an RS-232-C interface) in a legacy-free configuration and after an unsuccessful attempt to convert sio(4). The biggest problem with sio(4) is that it has been intertwined in many ugly ways into the kernel's core. Conversion could not happen without breaking something that invariably affects some group of people negatively. With sio(4) as a good bad example and a strong desire to solve multiple problems at once, the idea of an UART (Universal Asynchronuous Receiver/Transmitter) device that, given its generic name, could handle different flavors of UART hardware started to settle firmly in the authors mind.
The biggest challenge was of course solving the problem of the low-level console access prior to the initialization of the bus infrastructure and still have a driver that uses the bus access exclusively. Along the way the problem of having an UART function as the keyboard on sparc64 was solved with the introduction of system devices, which also encapsulated the console as a system device.
The uart(4) driver can be enhanced to support the various UART hardware on pc98 and this is currently being worked on. Keyboard support on sparc64 is underway as well. Plans exist for a rewrite of the remote gdb support that uses a generic interface to allow various drivers, including uart(4), to register itself as a communications channel. And since uart(4) does not support multi- port cards by itself, we likely need to either enhance puc(4) or otherwise introduce other umbrella drivers
Since I ported icc to FreeBSD I wanted to build FreeBSD with icc. Now with icc 7.1 (and some patches) it is possible. There are still some bugs, e.g. NFS doesn't work with an icc compiled kernel, IP seems to be fragile, and some advanced optimizations trigger an ICE (Intel is working on it). At the moment I'm waiting for our admins to install icc on the FreeBSD cluster (we got a commercial license from Intel, so we are allowed to distribute binaries which are compiled with icc), after that I will try to convince some people with more knowledge of the IP and NFS parts of the kernel to debug the remaining problems. When the icc compiled kernel seems to work mostly bugfree the userland will get the porting focus. Interested people may try to do a build of the ports tree with icc independently from the status of the porting of the userland... if this happens at the FreeBSD cluster, we would also be allowed to distribute the binaries.
Benefits include: another set of compiler errors (debugging help), more portable source, and code which is better optimized for a P4 (gcc has some drawbacks in this area)
The FreeBSD ports were updated to KDE 3.1.4, another bug- and security-fixes release. With this update, the QT port was updated to version 3.2. Both will be included in FreeBSD 4.9. Significant work was spent to fix KDE on FreeBSD-CURRENT after the removal of the gcc -pthread Option. Automatic package builds from KDE CVS continued to ensure and improve the quality of the upcoming KDE 3.2 release.
Future: Work is in progress to setup a new server for hosting the KDE-FreeBSD Website, Repository and another KDE CVS mirror. With help from Marcel Moolenaar the project will try to make KDE compile and working on the Intel IA64. And last but not least efforts are being made to fix the currently broken kdesu program.
WifiBSD is a miniture version of FreeBSD for wireless applications. Originally for the Soekris Net45xx line of main-boards, but is now capable of being targeted to any hardware/architecture FreeBSD itself supports. Although not feature complete, WifiBSD is expected to be ready for 5.2-RELEASE. The design goal is to meet, or exceed, the functionality of commercial/consumer 802.11 wireless gear. Features that need attention (to name just a few) are: http interface, consol menu interface, and installation. Volunters are welcome.
Work has restarted after a hiatus. Current focus is on getting loadable modules working, NEWBUSing the NetBSD dbdma code, and completing the BMAC ethernet driver.
There is a huge amount of work to do. Volunteers more than welcome!
The last known bug that prevented AMD64 machines completing a full release has been fixed - one single character error that caused ghostscript to crash during rendering diagrams. SMP work is nearing completion and should be committed within the next few days. The SMP code uses the ACPI MADT table based on John Baldwin's work-in-progress there for i386. We need to spend some time on low level optimization because there are several suboptimal places that have been ignored for simplicity, context switching in particular. MTRR support has been committed and XFree86 can use it. cvsup now works but the ezm3 port has not been updated yet. The default data segment size limit is 8GB instead of 512M, and the (primitive) i386 binary emulation support knows how to lower the rlimits for executing 32 bit binaries.
Notable things missing still: Hardware debug register support needs to be written; gdb is still being done as an external set of patches relative to the not-yet-released FSF gdb tree; DDB does not disassemble properly; DDB cannot do stack traces without -fno-omit-frame-pointer - a stack unwinder is needed; i386 and amd64 linux binary emulation is needed, and the i386 FreeBSD binary emulation still needs work - removing the stackgap code in particular.
The platform in general is very reliable although a couple of problems have been reported over the last week. One appears to be a stuck interrupt, but all that code has been redone for SMP support.
The FreeBSD Java community has started an effort to improve the current framework for Java-based ports. The main objective is the automation of JDK/JRE build and run dependency checking.
The original version was aimed to ease the life of porters. Although it has proved to be useful and reliable to a great extend, we are currently working on a new version. We intend to reach a high degree of flexibility to cope with the recent increase of available JDK/JRE flavors. Furthermore, the new version will be easier to maintain, which means improved reliability, and hopefully more frequent updates.
The BSD Java Porting Team has recently reached an exciting milestone with the release of the first "Diablo" JDK and JRE courtesy of the FreeBSD Foundation. The release of Diablo Caffe and Diablo Latte 1.3.1 was the first binary release of a native FreeBSD JDK since 1.1.8 and marks an important step forward in FreeBSD Java support.
The team is continuing development work, with a focus on achieving a compliant JDK 1.4 release in the near future.
With the introduction of ATAng, some users of ATAPI/CAM have experienced various problems. These have been mostly tracked down to issues in the new ATA code, as well as two long-standing problems in portions of the CAM layer that are rarely exercised with "real" SCSI SIMs. This has also been an occasion to cleanup ATAPI/CAM to make it more robust, and to enable DMA for devices accessed through it, resulting in improved performances.
We have released Japanese translation of 5.1-RELEASE online manual pages on June 10.
Several months ago, I took it upon myself to to try present the information contained on the bento build cluster to be presented in a more user-friendly fashion; that is, to be browsed by error type, by maintainer, and so forth. An early addition was code to attempt to classify ports PRs by either "existing port" (after assiging the most likely category and portname); "new port"; "framework" (e.g. bsd.port.mk changes); and "unknown". Various columns about the ports PRs were added to the reports.
The initial intent of this was to make life easier for ports maintainers; however, the "general" reports are also useful to anyone who just wants to, e.g., find out if a particular port is working on their particular architecture and OS combination before downloading it. Those with that general interest should start with the overview of one port.
A lot of work done since last report: site reworked completly (see new URL), console design with console message in text or graphic modes implemented, implementation of a compatibility layer to compile Linux fbdev drivers with more or less changes in the original driver (experimental).
Except some memory allocation bugs, X (XGGI based on XFree 3.3.6) is now working with the same driver as the console. A basic terminal has now to be implemented.
Volonteers are welcome to the project...
A number of races have been identified in locking device_t. Most of the races have been identified in making device_t have to do with how drivers are written. Efforts are underway to identify all the races, and to contact the authors of subsystems that can help the drivers. Of special concern is the need for the driver to ensure that all threads are completely out of the driver code before detach() finishes. Of additional concern is making sure that all sleepers are woken up before certain routines are called so that other subsystems can ensure the last condition and leave no dangling references. Locking device_t is relatively straight forward apart from these issues. Towards the end of proper locking, sample strawmen drivers are being used to work out what, exactly proper is. Once these issues are all known and documented in the code, efforts will be made to update relevant documentation in the tree. There are many problems with driver locking that has been done to date, but until we nail down how to write a driver in current, it will be premature to contact specific driver writers with specific concerns.
Support for several new crypto devices was added. The SafeNet 1141 is a medium performance part that is not yet available on retail products. The Hifn 7955 and 7956 parts are starting to appear on retail products that should be available by the end of the year. Both devices support AES encryption. Support for public key operations for the SafeNet devices was recently done for OpenBSD and will be backported. Public key support for the Hifn parts is planned.
A paper about the performance work done on the cryptographic subsystem was presented at the Usenix BSDCon 2003 conference and received the best paper award.
NetBSD recently imported the cryptographic subsystem.
The release of 4.9 is just around the corner and offers Physical Address Extensions (PAE) for x86 along with the same world-class stability and performance that is expected from the 4-STABLE series. As always, don't forget to purchase a copy of the CD set from your favorite FreeBSD vendor.
FreeBSD 5.1 was released in June and offered vastly improved stability over 5.0 along with a working implementation of Kernel Scheduled Entities, allowing for true multithreading of applications across multiple CPUs. FreeBSD 5.2 will be released by the end of 2003 and will focus on improved network and overall performance.
Numerous bugs have been fixed since the last status report (and of course a few new ones added). Progress on improved security has been slowed by other work. But new features and fixes are coming in from other groups that are now sharing the code. In particular NetBSD recently imported the revised 802.11 layer and the Linux-based MADWIFI project is using it too (albeit in an older form). The MADWIFI users have already contributed features such as fragmentation reassembly of 802.11 frames and improved signal monitoring. Power save polling and an improved rate control algorothm are expected to come in from the NetBSD folks. WPA support is still in the plans; the best estimate is that work on that will start in January.
The purpose of this project is to improve performance of the network subsystem. A major part of this work is to complete the locking of the networking subsystem so that it no longer depends on the "Giant lock" for proper operation. Removing the use of Giant will improve performance and permit multiple instances of the network stack to operate concurrently on multiprocessor systems.
This project started in August. The emphasis has been on locking the "lower half" of the networking code so that packet forwarding through the IPv4 path can operate without the Giant lock as part of the 5.2 release. To this end locking was added to several network interface drivers and much of the "middleware" code in the network was locked (e.g. ipfw, dummynet, then routing table, multicast routing support, etc). Work towards this goal is still ongoing but should be ready for 5.2. A variety of test systems have been running for several months without the Giant lock in the network drivers and IP layer.
Past the 5.2 release Giant will be removed from the "upper half" of the network subsystem and the socket layer. Once this is done the plan is to measure and improve performance (though some work of this sort is always happening). The ultimate goal is a system that performs at least as well as 4.x for normal use on uniprocessor systems. On multiprocessor systems we expect to see significantly better performance than 4.x due to greater concurrency and reduced latency.
The FreeBSD status reports are back again with the 2003 year-end edition. Many new projects are starting up and gaining momentum, including XFS, MIPS, PowerPC, and networking locking and multithreading. The end of 2003 also saw the release of FreeBSD 4.9, the first stable release to have greater than 4GB support for the ia32 platform. Work on FreeBSD 5.2 also finished up and was released early in January of 2004. Many thanks to all of the people who worked so hard on these releases and made them happen.
This is the largest status report ever, so read and enjoy!
Scott Long, Robert Watson
The libarchive library, which reads and writes tar and cpio archives, is about ready to commit to the tree. The bsdtar program, built on libarchive, is also nearing completion and should soon be a worthwhile successor to our aging GNU tar. I plan a gradual transition during which "bsdtar" and "gtar" will coexist in the tree.
Oddly enough, libarchive and bsdtar are the first fruits of a project to completely rewrite the pkg tools. I've started architecting a libpkg library for handling routine package management and have a prototype pkg_add that is three times faster than the current version.
I did a xml/xslt conversion of the html files to make maintaining of the page more comfortable. I removed the cdsets, which might be kept in CVS or some kind of archive for historical reasons. The books got an update, and were categorized in respect to the language they are written in. As soon as I get my access on the cvs repository I will commit the updates. People are encouraged to add local FreeBSD books, I missed, especially in the asian area. Feel free to send me links to books to add.
DVB ASI stands for Digital Video Broadcast - Asynchronous Serial Interface. It is the standard defined to send and receive DVB stream from Satellite (DVB-S), Terrestrial link (DVB-T), and TV Cable (DVB-C). This standard was developed in Europe to transport 188-byte MPEG cells and 204-byte MPEG cells. However it can be used to carry IP over DVB too.
The FreeBSD driver uses the newbus amd the bus-dma API. It means that it could be easily ported to all the BSD flavors (NetBSD, OpenBSD).
It uses the same API than the Linux DVB ASI support from ComputerModules that is based on the following devices:
Special thanks to Tom Thorsteinson from Computer Modules who helped 6WIND to port their driver. It is used by 6WIND in order to provide IPv4, IPv6, Ethernet and our network services over DVB.
Copyright 2003-2004, 6WIND
Enhancements continue to be made to the system. Several, including improvements to the PR classification algorithm, the ability to more correctly guess when a PR has been updated, and better handling of errors in both port Makefiles and the bento builds, are invisible to end-users. However, the addition of a "repocopy" classification is notable, as is the allowing the wildcard search in "overview of one port" (thanks to edwin@ for the shove in that direction.) Additionally, logic has been added to identify the proposed category/portname of new ports, with the goal being to quickly identify possible duplications of effort. (Some SQL performance was sacrificed to this goal, leading to some pages to load more slowly; this needs to be fixed.)
The other work has been on an email back-end to allow the occasional sending of email to maintainers. Two functions are currently available: "remind maintainers of their ports that are marked BROKEN", and "remind maintainers of PRs that they may not have seen." A recent run of the former got generally good response, especially as changing some cases of BROKEN to IGNORE (PR ports/61090) had removed almost all the annoying false positives. However, work remains to try to find out why a few allegedly broken ports only fail in certain environments (including the bento cluster).
The next plan is to use the proposed DEPRECATED Makevar (see ports/59362) to create a new report to allow querying of "ports currently slated to be removed". This report could also be posted to ports@ periodically with minimal work. The author believes that doing this would allow the port deprecation process to be much more visible to the general FreeBSD user community.
The FreeBSD kernel now builds and runs fine with icc v7 (only GENERIC and a custom kernel tested so far). A review on arch@ revealed no major concerns and some src committers are willing to commit the patches. As icc v8 is out and defines __GNUC__ I want to rework the patches before they get committed so an icc v8 compiled kernel DTRT too.
A complete build of the ports collection (as of start of December)
finished and is under review to determine the reason of build
failures. Current
The above mentioned build of the ports collection was run on a P4 with a icc compiled kernel (optimized for a P4). No kernel panics or other strange behavior was noticed. The ports collection was build with a CPUTYPE of p4 and CFLAGS set to "-Os -pipe -mfpmath=sse -msse2" in the gcc and "-O2" in the icc case. No package is tested for correct run-time behavior so far.
Much work has been invested into getting release 2.00 stable. It provides the complete OpenBSD 3.4 function set, as well as fine grained locking to work with a giant free network stack.
pf provides: IPv6 filtering and normalization, "syn-proxy" to protect (web)server against SYN-floods, passive OS detection, fast and modular address tables, source/policy routing, stateful filter and normalization engine, structured rulesets via anchors and many many more. Especially in connection with ALTQ, pf can help to harden against various flood attacks and improve user experience.
New features from OpenBSD-Current like: state synchronization over wire and enhanced support for cloned interfaces require patches to the kernel. We are trying to resolve this issue and start OpenBSD-Current tracking again as soon as possible.
Thanks to recent donations, I am now building binary security updates for FreeBSD {4.7, 4.8, 4.9, 5.0, 5.1, 5.2}-RELEASE. (Note that FreeBSD 4.7 and 5.0 are no longer officially supported; any advisories which are not reflected in the CVS tree will likewise not result in binary updates.)
The current version (1.5) of FreeBSD Update will warn about locally modified files and will, by default, leave them untouched; if a "distribution branch", (i.e. crypto, nocrypto, krb4, or krb5) is specified, FreeBSD Update can be forced to "update" files which have been compiled locally.
The only major issue remaining with FreeBSD Update is the single-point-of-failure of the update building process; I would like to resolve this in the future by having several machines cross-verify and cross-sign, but this will require a significant investment of time, and will probably have to wait until I've finished writing my DPhil thesis.
A project was started to revive a stalled effort to port SGI XFS journaling filesystem to FreeBSD. The project is based on Linux development sources from SGI and is currently being kept in a private Perforce repository. The work is progressing slowly due to lack of free time. At the moment we have XFS kernel module which is capable of mounting XFS filesystems read-only, with a panic or two happening infrequently, that need to be isolated and fixed. Semi-working metadata updates with full transaction support are there too, but will probably have to be rewritten to minimize the amount of custom kernel changes required.
We seek volunteers to help with userland part of the port. Namely, existing xfsprogs port needs to be cleaned up, incompletely ported utilities brought into a working shape. xfs_dump/xfs_restore and as much from xfstests suite as possible need to be ported too. We do not need testers for now, so please to not ask for module sources just yet.
Not much to report. Bluetooth code was integrated into the FreeBSD source tree. Bluetooth kernel modules appear to be stable. I have received few success stories from the users.
During last few months the efforts were to make Bluetooth code more user friendly. Bluetooth Service Discovery Procotol daemon sdpd was reimplemented under BSD-style license and committed. The next step is to integrate existing Bluetooth utilities with SDP.
Thanks to Matt Peterson <matt at peterson dot org> I now have Bluetooth keyboard and mouse for development. I'm currently working on Bluetooth HID profile implementation.
Dave Sainty <dave at dtsp dot co dot nz> from NetBSD project offered his help in porting Bluetooth stack to NetBSD.
At the end of October, the if_name and if_unit members of struct ifnet were replaced with if_xname from NetBSD and if_dname and if_dunit. These represent the name of the interface and the driver name and instance of the interface respectively. Other then breaking IPFilter for a few weeks due to the userland being on the vendor branch, this change went quite well. A few ports needed minor changes, but otherwise nothing changed from the user perspective.
The purpose of this change was the lay the groundwork for support for network interface renaming and to allow the implementation of more interesting pseudo interface cloning support. An example of interesting cloning support would be using "ifconfig fxp0.20 create" to create and configure a vlan interface on fxp0 that handled frames marked with the tag 20. Interface renaming is being worked on in Perforce at the moment with a working version expected for review soon. Support for enhanced device cloning is still in the planing stage.
FreeBSD has well over a few hundred tunables without documentation. This project aims at designing an automated process to rip all available tunables and generate a manual page based on the selected kernel options. The ideal implementation, however; would gather tunables from the LINT kernels as well. This would provide a default manual page for all supported architectures. A simple tool has been forged from the various off-list and on-list discussions and is waiting review from the -doc team. Anyone interesting in reviewing my current work is requested to get in contact with me.
We have been updating existing Japanese translations of manual pages to meet the 5.2-RELEASE schedule. Also, 22 new translations were complete during this period.
This project aims to update the current MIDI implementation. We are currently looking at removing the current code sometime in February and importing the new version soon after. I'm currently working on a kernel/timidity bridge for those without external hardware.
The FreeBSD Russian Documentation Project aims to provide FreeBSD Documentation translated to Russian. Already done: FAQ, Porters Handbook, WWW (partially synched with English version), some articles.
We working at Handbook (and more docs) translation and synchronization with English versions and need more translators (or financial aid to continue our work. If you can help, please, contact us at ru-cvs-committers@FreeBSD.org.ua (or andy@FreeBSD.org).
The libkse library will shortly be renamed to libpthread and be made the default thread library. This includes making the GCC -pthread option link to -lpthread instead of libc_r and changing PTHREAD_LIBS to -lpthread. David Xu has been working on GDB support and has it working with the GDB currently in our tree. The next step is to make a libpthread_db and get it working with GDB 6.0 which marcel has imported into the perforce tree.
2003 was quite successful for the Donations team. We shepherded over 200 items from donors into the hands of developers. Some high points include: a small cluster for the security team, assorted laptop hardware for our cardbus work, and documentation for our standards group. In the main FreeBSD.org cluster we were able to replace 8 DEC Miata machines with 6 Alpha DS10s (21264). Every committer doing SMP work now has multi-processor testing hardware.
We have smoothed out the tax deduction process with the FreeBSD Foundation, and can ship donated items directly to the recipients instead of tying up Foundation time handling shipping.
Current team membership is: Michael Lucas, David O'Brien, and Tom Rhodes. Wilko Bulte has replaced Robert Watson as the Core Team representative.
The updated acpi_cpu driver was committed in November. Work is ongoing to finish support for _CST re-evaluation, which makes it possible for laptops based on processors like the Centrino to use varying CPU idle states when on or off AC power. 5.2-RELEASE also went out with support for _CID packages, which fixed mouse probing for Compaq users. Control of CPU idle states and throttling can now be done through rc.conf(5) settings for the /etc/power_profile script, which switches between performance/economy levels when the AC status changes.
One huge task underway is the cpufreq project, a framework for detecting and controlling various frequency/voltage technologies (SpeedStep, LongRun, ACPI Performance states, etc.) The ACPI performance states driver is working and the framework is being implemented. It requires newbus attachments for CPUs so some ground work needs to go in before the driver can be committed.
ACPI-CA was updated to 20031203 in early December and with a few patches is reasonably stable. An ACPI debugging how-to has been written and is being DocBooked by trhodes@. Ongoing work on fixing interrupt storms due to various ways of setting up the SCI is being done by jhb@.
I'd like to welcome Philip Paeps (philip@) to the FreeBSD team. Philip has written an ACPI ASUS driver that will be committed soon and has been very helpful on the mailing lists. We've also had a lot of help from jhb@, marcel@, imp@, and peter@. We're hoping to see the return of takawata@ and iwasaki@, who have been very helpful in the past. If any developers are interested in assisting with ACPI, please see the ACPI TODO and send us an email.
Most of the console blocks are in place with nice results (see screenshots on the site). Boot console and virtual terminals are working with 8bit rendering and perfect integration of true graphic drivers in the kernel.
Now it is time to bring it to end user and a precompiled R5.2 GENERIC kernel is available for this (see the site news). In parallel, after providing a last tarball/patch for R5.2, everything will move to Perforce.
As always, volunteers are welcome. The task is huge but very exciting.
The direct objective is to make FreeBSD/powerpc work on Motorola MCP750 and similar (single board computer that is compliant with Compact PCI standard) Based on this work it would be easy to bring it to other embedded systems.
1. loader(8): it is based on the existing loader for FreeBSD/powerpc port but binding to OpenFirmware was removed and replaced with PPCBug firmware binding. It only supports netbooting for the moment, so disk (compact flash) support needs to be done one day. The loader is the only piece that relies onPPCBug system calls - once the kernel starts it doesn't need firmware support any longer.
2. kernel: it is now divorced from OpenFirmware dependencies; most of the groundwork finished includes: nexus stuff is sorted out (resources management is ok except interrupts assignment); host to PCI bridge low level routines are finished so configuration of and access to PCI devices works; the only important thing missing is the IRQ management (Raven MPIC part is done, but the board has the second PIC, 8259-compatible that needs to be set up, but here the existing code from x86 arch will be adopted).
Once the IRQ management is cleared out, most of the devices on board would work straight away since they are pretty standard chips with drivers already implemented in the tree (e.g. if_de).
At the moment work is on hold (don't have physical access to the device) but will resume when I'm back home (late Feb).
The TrustedBSD Mandatory Access Control (MAC) Framework permits the FreeBSD kernel and userspace access control policies to be adapted at compile-time, boot-time, or run-time. The MAC Framework provides common infrastructure components, such as policy-agnostic labeling, making it possible to easily development and distribute new access control policy modules. Sample modules include Biba, MLS, and Type Enforcement, as well as a variety of system hardening policies.
TrustedBSD MAC development branch in Perforce integrated to 5.2-RELEASE.
The TrustedBSD MAC Framework now enforces protections on System V IPC objects and methods. Shared memory, semaphores, and message queues are labeled, and most operations are controlled. The Biba, MLS, Test, and Stub policies have been updated for System V IPC. (Not yet merged)
The TrustedBSD MAC Framework now enforces protections on POSIX semaphore objects and methods. The Biba, MLS, Test, and Stub policies have been updated. (Not yet merged)
The TrustedBSD MAC Framework's central kernel implementation previously existed in one large file, src/sys/kern/kern_mac.c. It is now broken out into a series of by-service files in src/sys/security/mac. src/sys/security/mac/mac_internal.h specifies APIs, structures, and variables used internally across the different parts of the framework. System calls and registration still occur in kern_mac.c. This permits more easy maintenance of locally added object types. (Merged)
Break out mac_policy_list into two different lists, one to hold "static" policy modules -- ones loaded prior to kernel initialization, and that may not be loaded, and one for "dynamic" policy modules -- that are either loaded later in boot, or may be unloaded. Perform less synchronization when using static modules only, reducing overhead for entering the framework when not using dynamic modules. (Merged)
Introduced a kernel option, MAC_STATIC, which permits only statically registered policy modules to be loaded at boot or compiled into the kernel. When running with MAC_STATIC, no internal synchronization is required in the MAC Framework, lowering the cost of MAC Framework entry points. (Not yet merged)
Make mac.h userland API definition C++-happy. (Merged)
Created mac_support.4, a declaration of what kernel and userspace features are (and aren't) supported with MAC. (Not yet merged)
Stale SEBSD module deleted from MAC branch; SEBSD module will solely be developed in the SEBSD branch from now on. See the TrustedBSD SEBSD report for more detail.
Use only pointers to 'struct label' in various kernel objects outside the MAC Framework, and use a zone allocator to allocate label storage. This permits label structures to have their size changed more easily without changing the normal kernel ABI. This also lowers the non-MAC memory overhead for base kernel structures. This also simplifies handling and storage of labels in some of the edge cases where labels are exposed outside of the Framework, such as in execve(). Include files outside of the Framework are substantially simplified and now frequently no longer require _label.h. (Merged)
Giant pushed down into the MAC Framework in a number of MAC related system calls, as it is not required for almost all of the MAC Framework. The exceptions are areas where the Framework interacts with pieces of the kernel still covered by MAC and relies on Giant to protect label storage in those structures. However, even in those cases, we can push Giant in quite a bit past label internalization/externalization/ storage allocation/deallocation. This substantially simplifies file descriptor-based MAC label system calls. (Merged)
Remove unneeded mpo_destroy methods for Biba, LOMAC, and MLS since they cannot be unloaded. (Merged)
Biba and MLS now use UMA zones for label allocation, which improves storage efficiency and enhances performance. (Merged)
Bug fix for mac_prepare_type() to better support arbitrary object label definitions in /etc/mac.conf. (Merged)
Labels added to 'struct inpcb', which represents TCP and UDP connections at the network layer. These labels cache socket labels at the application layer so that the labels may be accessed without application layer socket locks. When a label is changed on the socket, it is pushed down to the network layer through additional entry points. Biba, MLS policies updated to reflect this change. (Merged)
SO_PEERLABEL socket option fixed so that peer socket labels may be retrieved. (Merged)
mac_get_fd() learns to retrieve local socket labels, providing a simpler API than SO_LABEL with getsockopt(). mac_set_fd() learns about local socket labels, providing a simpler API than SO_LABEL with setsockopt(). This also improves the ABI by not embedding a struct label in the socket option arguments, instead using the copyin/copyout routine for labels used for other object types. (Merged)
Some function names simplified relating to socket options. (Merged)
Library call mac_get_peer() implemented in terms of getsockopt() with SO_PEERLABEL to improve API/ABI for networked applications that speak MAC. (Merged)
mac_create_cred() renamed to mac_cred_copy(), similar to other label copying methods, allowing policies to implement all the label copying method with a single function, if desired. This also provides a better semantic match for the crdup() behavior. (Merged)
Support "id -M", similar to Trusted IRIX. (Not yet merged)
TCP now uses the inpcb label when responding in timed wait, avoiding reaching up to the socket layer for label information in otherwise network-centric code.
Numerous bug fixes, including assertion fixes in the MAC test policy relating to execution and relabeling. (Merged)
TrustedBSD Access Control Lists (ACLs) provide extended discretionary access control support for the UFS and UFS2 file systems on FreeBSD. They implement POSIX.1e ACLs with some extensions, and meet the Common Criteria CAPP requirements. Most ACL-related work is complete, with remaining tasks associated with userspace integration, third party applications, and compatibility
Prototyped Solaris/Linux semantics for combining ACLs and the umask: if an default ACL mask is defined, substitute that mask for the umask, permitting ACLs to override umasks. (Not merged)
TrustedBSD "Security-Enhanced BSD" (SEBSD) is a port of NSA's SELinux FLASK security architecture, Type Enforcement (TE) policy engine and language, and sample policy to FreeBSD using the TrustedBSD MAC Framework. SEBSD is available as a loadable policy module for the MAC Framework, along with a set of userspace extensions support security-extended labeling calls. In most cases, existing MAC Framework functions provide the necessary abstractions for SEBSD to plug in without SEBSD-specific changes, but some extensions to the MAC Framework have been required; these changes are developed in the SEBSD development branch, then merged to the MAC branch as they mature, and then to the FreeBSD development tree.
Unlike other MAC Framework policy modules, the SEBSD module falls under the GPL, as it is derived from NSA's implementation. However, the eventual goal is to support plugging SEBSD into a base FreeBSD install without any modifications to FreeBSD itself.
TrustedBSD SEBSD development branch in Perforce integrated to 5.2-RELEASE. Other changes in the MAC branch, including restructuring of MAC Framework files also integrated, and a move to zone allocation for labels. See the TrustedBSD MAC Framework report for more detail on these and other MAC changes that also affect the SEBSD work.
FreeBSD PTY code modified so that the MAC Framework and SEBSD module can create pty's with the label of the process trying to access them. Improves compatibility with the SELinux sample policy. (Not yet merged)
SEBSD now loads its initial policy in the boot loader rather than using a dummy policy until the root file system is mounted, and then loading it using VFS operations. This avoids initial labeling and access control conditions during the boot.
security_load_policy() now passes a memory buffer and length to the kernel, permitting the policy reload mechanisms to be shared between the early boot load and late reloads. The kernel SEBSD code now no longer needs to perform direct file I/O relating to reading the policy. checkpolicy now mmap's the policy before making the system call.
SEBSD now enforces protections on System V IPC objects and methods. Shared memory, semaphores, and message queues are labeled, and most operations are controlled. The sample policy has been updated.
The TrustedBSD MAC Framework now controls mount, umount, and remount operations. A new MAC system call, mac_get_fs() can be used to query the mountpoint label. lmount() system call allows a mount label to be explicitly specified at mount time. The SEBSD policy module has been updated to reflect this functionality, and sample TE policy has been updated. (Not yet merged)
SEBSD now enforces protections on POSIX semaphores; the sample policy has been updated to demonstrate how to label and control sempahores. This includes sample rules for PostgreSQL.
The SEBSD sample policy, policy syntax, and policy tools have been updated to the SELinux code drop from August. Bmake these pieces so we don't need gmake.
Provide file ioctl() MAC Framework entry point and SEBSD implementation.
A large number of sample policy tweaks and fixes. The policy has been updated to permit cron to operate properly. It has been updated for FreeBSD 5.2 changes, including dynamically linked root. Teach the sample policy about FreeBSD's sendmail wrapper.
Adapt sysinstall and install process for SEBSD pieces. Teach sysinstall, newfs, et al, about multilabel file systems, install SEBSD sample policy pieces, build policy. Automatically load the SEBSD module on first boot after install.
Allow "ls -Z" to print out labels without long format.
The TrustedBSD Project is producing an implementation of CAPP compliant Audit support for use with FreeBSD. Little progress was made on this implementation between October and December other than an update to the existing development tree. However, in January, work began on porting the Darwin Audit implementation to FreeBSD. Details on this work will appear in the next report; more information is available on the TrustedBSD audit discussion list. Perforce messages may be seen on the trustedbsd-cvs mailing list.
The TrustedBSD Project is implementing many new features for the FreeBSD Project. It also provides documentation for users, administrators, and developers.
mac_support.4 added -- documents TrustedBSD MAC Framework feature compatibility. See also the MAC Framework report.
FreeBSD security architecture updated and corrections/additions made.
A variety of documentation updates relating to API changes, including the socket-related API changes in libc/mac(3).
TLB support code and PMAP have come along nicely. GCC and related have been kept up to date with the main tree. An evaluation board from Broadcom was donated and initial work on that platform has been occurring. Much old and obsolete code brought from NetBSD for bootstrapping the effort has been cleaned up. The system has been seen to get to the point of trying to initialize filesystems, but there are still bugs even before that milestone.
Simple support AGP 3.0 including support for AGP 8x mode was added. The support is simple in that it still assumes only one master and one target. The main gain is the ability to use AGP 8x with drm modules that support it.
The purpose of this project is to improve performance of the network subsystem. A major part of this work is to complete the locking of the networking subsystem so that it no longer depends on the "Giant lock" for proper operation. Removing the use of Giant will improve performance and permit multiple instances of the network stack to operate concurrently on multiprocessor systems.
Locking of the network subsystem is largely complete. Network drivers, middleware layers (e.g. ipfw, dummynet, bridge, etc.), the routing tables, IPv4, NFS, and sockets are locked and operating without the use of Giant. Much of this work was included in the 5.2 release, but not enabled by default. The remaining work (mostly locking of the socket layer) will be committed to CVS as soon as we can resolve how to handle "legacy protocols" (i.e. those protocols that are not locked). The code can be obtained now from the Perforce database. A variety of test and production systems have been running this code for several months without any obvious issues.
Performance analysis and tuning is ongoing. Initial results indicate SMP performance is already better than 4.x systems but UP performance is still lagging (though improved over -current). The removal of Giant from the network subsystem has reduced contention on Giant and highlighted performance bottlenecks in other parts of the system.
This work was supported by the FreeBSD Foundation.
Work to merge the NetBSD and MADWIFI code bases is almost complete. This brings in new features and improves sharing which will enable future development. Support was added for 802.1x client authentication (using the open1x xsupplicant program) and for shared key authentication (both client and AP) which improves interopability with systems like OS X. The awi driver was updated to use the common 802.11 layer and the Atheros driver received extensive work to support hardware multi-rate retry. Kismet now works with the device-independent radiotap capture format. All of this work is still in Perforce but should be committed to CVS soon.
Work has begun on full 802.1x and WPA support.
Work is progressing on SMPng on several different fronts. Sam Leffler and several other folks have been working on locking the network stack as mentioned elsewhere in this update. Several infrastructure improvements have been made in the past few months as well.
The low-level interrupt code for the i386 architecture has been redesigned to allow for a runtime selection between different types of interrupt controllers. This work allows the Advanced Programmable Interrupt Controllers (APICs) to be used instead of the AT 8259A PIC without having to compile a separate kernel to do so. It also allows the APIC to be used in a UP kernel as well as on a UP box. Together, all these changes allow an SMP kernel to work on a UP box and thus allowed SMP to be enabled in GENERIC as it already is on all of the other supported architectures. This work also reworked the APIC support to correctly route PCI interrupts when using an APIC to service device interrupts. This work was also used to add SMP support to the amd64 port.
A turnstile implementation was committed that implemented a queue of threads blocked on a resource along with priority inheritance of blocked threads to the owner of the resource. Turnstiles were then used to replace the thread queue built into each mutex object which shrunk the size of each mutex as well as reduced the use of the sched_lock spin mutex.
2004 started with another exciting two months for the project. FreeBSD 5.2 was released in early January and then quickly followed in February with the 5.2.1 bug-fix release. Looking forward, we are expecting a late-April release date for FreeBSD 4.10, and mid-summer date for FreeBSD 5.3. And don't forget to support the FreeBSD vendors and developers by buying a copy of the latest CD or DVD sets.
Thanks,
Scott Long
In the overall area of disk and device I/O, a significant milestone was reached with the implementation of proper reference counting on dev_t. We are now able to properly allocate and free dev_t. Cloning device drivers also had the job made easier for them with the addition of the unit number management routines.
It is not quite decided which will be the next step in the quest for a truly SMPng I/O subsystem, but a leading candidate is to implement the device-access vnode bypass to get more concurrency in the system: Instead of taking the tour through the vnodes for each i/o operation on a device we will go directly from the file descriptor layer to DEVFS/SPECFS. In addition to Giant-less disk I/O, this should enable us to pull the entire tty subsystem and the PTY driver out from under Giant and we expect that to improve the "snappiness" of the system measurably.
The Dutch Documentation Project is a ongoing project in translating the handbook and other documentation to the dutch language. Currently there is 1 active person (me) translating the documentation. I am currently working on the handbook/basics section. But i can use some more hands, please drop me an email if you wish to help out so that the dutch translation will speed up and be ready in some time. Contact remko@elvandar.org for information.
I have been producing weekly summaries of commits and the surrounding discussions as reported on the cvs-src mailing list. These summaries are posted to -current on Sunday evenings and archived on the Web. The reception has been overwhelmingly good. As of the end of February, Polish translations are being produced by Lukasz Dudek and Szymon Roczniak; they are also planning to translate the older summaries.
libarchive, with complete documentation, has been committed to -CURRENT. bsdtar should follow soon. For a few months, gtar and bsdtar will both be available in the base system. Once bsdtar is in the tree, I hope to resume work on libpkg and my pkg_add rewrite.
Note that bsdtar is not an exact replacement for gtar: it does some things better (reads/writes standard formats, archive ACLs and file flags, detects format and compression automatically), some things worse (does not handle multi-volume archives or sparse files) and a few things just different (writes POSIX-format archives by default, not GNU-format). The command lines are sufficiently similar that most users should have no problems with the transition. However, people who rely on peculiar options or capabilities of gtar may have to look to ports.
The first actual feature related to the if_xname conversion was committed in early February. Network interfaces can now be renamed with "ifconfig <if> name <newname>".
Work is slowly progressing on a new network interface cloning API to enable interesting cloners like auto-configurating vlans. This work is taking place in the perforce repository under: //depot/user/brooks/xname/...
After a slow time at the end of last year due to a disk crash, the project is moving along rapidly. The loader is fully functional with Forth support. Syscons has been integrated. New Powerbook models are supported. Work is starting on a G5 port.
There's still lots to do, so as usual volunteers are most welcome.
The project is a joint effort of volunteers, which focus in the internationalization and localization of the FreeBSD Operating System and applications running on FreeBSD. All of the work resulted in this project will be contributed back to the FreeBSD project.
Thanks to many volunteers' help, by this time of writing, we have finished more than 60% of the translation of the FreeBSD Handbook. We plan to submit a preliminary translation of the FreeBSD website as well as the FreeBSD Handbook when most part of them were finished, which is expected to happen in a couple of months. The snapshot of the documentation translation effort could be accessed through the URL listed above.
The project also supported individual efforts on porting applications (especially software that supports Simplified and/or Traditional Chinese) to FreeBSD. We are also doing some research on making FreeBSD kernel and base system more i18n-aware.
The verify source reachability option for ipfw2 checks if the source IP address of a packet entering the machine is reachable at all. Thus if we can't send a packet back because we don't have a route back we don't have to forward it because two way communication isn't possible anyway. It is more than likely that such a packet is spoofed. This option is almost the same as what is known on Cisco IOS as "ip verify unicast source reachable-via [any|ifn]". Using this option only makes sense when you don't have a default route which naturally always matches. So this is useful for machines acting as routers with a default-free view of the entire Internet as common when running a BGP daemon (Zebra/Quagga or OpenBSD bgpd).
One useful way of enabling it globally on a router looks like this: ipfw add xxxx deny ip from any to any not versrcreach or for an individual interface only: ipfw add xxxx deny ip from any to any not versrcreach recv fxp0
The ARP IP address to MAC address mapping does not belong into the routing table (FIB) as it is currently done. This will move it to its own hash based structure which will be instantiated per each 802.1 broadcast domain. With this change it is possible to have more than one interface in the same IP subnet and layer 2 broadcast domain. The ARP handling and the routing table will be quite a bit simplified afterwards. As an additional benefit full MAC address based accosting will be provided. Work on this project is already in progress.
The current TCP send and receive buffers are static and set to a conservative value to preserve kernel memory. This is sub-optimal for connections with a high bandwidth*delay product because the size of the TCP send buffer determines how big the send window can get. For high bandwidth trans-continental links this seriously limits the maximum transfer speed per TCP connection. For example a 170ms RTT and a 32kB send buffer limit the speed to approximately 1.5Mbit per second even thought you might have a 10Mbit pipe.
This project makes the TCP send buffer to automatically adapt to the optimal buffer size for maximal link usage. In the case above this would be a buffer of approximately 220kB. The main challenge is to have a stable and reliable measurement of the link parameters and manage the kernel memory properly and in a fair way. We don't want to have a few connections to monopolize all available socket buffer space and many edge cases have to be considered. The first implementation will be tuned conservatively but even that will provide significantly better performance than the static buffers currently. Work on this project is already in progress.
The TCP performance test and qualification testbed is an automated environment that simulates various common and uncommon end-to-end network and link characteristics such as delay, bandwidth limitations, congestion, packet drops, packet corruption and out of order arrival. The testbed automatically steps through all link types and tests various TCP optimizations and parameter adjustments. In the end all data is graphically arranged and compared against standard behaviour and each other to judge the positive or negative effects of the modifications. Work on this project has just started and is based on FreeBSDs dummynet.
Thanks to the loan of a box by Will Andrews, the system has been moved into production. The previous installation at lonesome.com now refers you to the new system. As part of the installation, a preliminary FAQ was added.
The database is updated once per hour.
New reports available include ones about ports marked DEPRECATED, since that function has now been incorporated into bsd.port.mk. (The author hopes that this will allow the port deprecation process to be much more visible to the general FreeBSD user community.) In addition, a report for ports marked FORBIDDEN was added (the code was essentially the same).
The next topic of interest is to try to identify ports which are slave ports because the status of these ports is not currently being updated automatically. This problem also affects FreshPorts. PR ports/63683 is an attempt to address this problem. Also, preliminary work has been done on creating some graphs and charts for various statistics, and in creating a tool to browse port dependencies for the entire ports tree.
Some general observations about the trends in ports PRs can be made:
The FreeSBIE Project aims to develop a set of scripts that allow anyone to create their own FreeBSD Bootable Cdrom, with their own set of installed packages. The Project releases an ISO builded with FreeSBIE scripts, to show what they can do. On Sunday 29 February 2004, FreeSBIE 1.0 was released and it had a great success, as there were post on Slashdot.org, OSnews, DaemonNews and BSDForums. Thanks to the huge amount of feedback they got, FreeSBIE Developers are now developing new features such as support for archs different from i386. Website redesign is on the way too.
Move to Perforce is done. I spent some time on building a common compilation tree with Linux: until now drivers were build in a FreeBSD makefile tree, not compatible with Linux.
The next priorities are ANSI support and keymaps in the KGC Kernel Graphic Console system.
Work on the PMAP overhaul has been put into gear. A lot of issues will be addressed, including support for sparse physical memory and of course SMP. Performance will be addressed to the extend possible, but functionality has priority. The redesign will lay the foundation for NUMA support where possible. An example of this is limiting TLB shootdowns to processors that actually have or had TLBs belonging to the PMAP loaded. Of course, without NUMA hardware the implementation of NUMA support is quite limited.
Distributed package builds are currently done using a set of home-grown shell scripts for managing, scheduling and dispatching of package builds on the client machines. This has been sufficient for our needs in the past, but has a number of significant shortcomings that limit future growth. I am rewriting the package build scripts to work on top of Sun GridEngine (ports/sysutils/sge), as a client application of a "FreeBSD package grid". Some of the design goals for the new system are:
The "geomification" of vinum has made some progress. I now have all basic setups working (concatenated plexes, striped plexes, RAID5 plexes, and RAID1), but I still have to implement correct error handling and status change handling.
Still missing is a userland tool, so currently you still have to use "old-style" vinum to configure your setup.
NanoBSD, src/tools/tools/nanobsd, is a tool for stuffing FreeBSD onto small disk media (like CompactFlash) for embedded applications. The disk image is built with three partitions, two for software images and one for configuration files. Having two software partitions means that new software can be uploaded to the non-active partition while running off the active partition.
The first really public version has been committed and many suggestions and offers of patches have started pouring in.
The sources were imported from OpenBSD 3.4R and patched with diffs obtained from the port. Since March the 8th it is linked to the build and install. There is some more work to be done in order make pf a home inside the tree, but the biggest hunk of work was lifted during the past two month.
OpenBSD 3.5 is scheduled for early May, so we might see an update before 5.3R. Work towards integration of the - often requested - ALTQ framework is in progress also, though it is not yet clear how well it goes along with the ongoing work towards a giant free net stack.
Development goes reasonably fast, right now it boots single user. It is still very simics-centric, and it deserves a huge cleanup and a few bug fixes, but there's already a decent amount of code to work with, mostly taken from NetBSD. I now plan to work on real hardware support (as soon as I can get some), to get the missing userland bits (mainly rtld and the pthread libs) so that I can build a full world.
Not much has changed since last report was submitted. The read-only access XFS volumes is quite stable now. The work is underway to rewrite xfs_buf layer to minimize local changes intrusiveness. Initial attempt to make XFS code to compile and run on amd64 is in progress too.
We really need a care-taker for our userland tools.
If nothing bad happened, the icc patches got committed around the date of the deadline for submissions of this report. Please search the archives of -current and/or cvs-all for more information.
The next steps in this project are to
Not much to report. Bluetooth Service Discovery Procotol daemon sdpd was integrated with existing Bluetooth utilities. From now on users should not use GNU sdpd (Linux BlueZ port).
Bluetooth HID profile implementation is almost complete. Thanks to Matt Peterson < matt at peterson dot org > for giving me Bluetooth keyboard and mouse for development.
It has been a year since our last status report, but we haven't slowed down. Since the last report, Alexander Nedotsukov (bland) and Pav Lucistnik (pav) have joined the FreeBSD GNOME team. GNOME 2.4 was released back in September 2003, followed by 2.4.1 and 2.4.2. We are actively working on getting GNOME 2.6.0 out the door at the end of March. GNOME 2.6 Beta releases can be obtained via the project URL above.
To help make GNOME 2.6.0 our best release to date, we have created a script to automate the upgrade from GNOME 2.4. We also have a new GNOME package build server that builds and serves i386 packages for all supported FreeBSD releases. We plan on having the GNOME 2.6.0 packages available the moment 2.6.0 hits the ports tree.
Included in the release of GNOME 2.6 is GTK+ 2.4, the next installment in the GTK+ 2 series. Because GTK+ 2 has become very stable over the past few years, the FreeBSD GNOME Team is pushing for GTK+ 2 support to be included by default in all applications that support it. This has already been done with Mozilla, Firefox, and Thunderbird. A complete GNOME Desktop and application environment can already be built using only GTK+ 2. The ultimate goal is to phase GTK+ 1 out of the ports tree.
This project is aimed at converting the FreeBSD network stack from running under the single Giant kernel lock to permitting it to run in a fully parallel manner on multiple CPUs (i.e., a fully threaded network stack). This will improve performance/latency through reentrancy and preemption on single-processor machines, and also on multi-processor machines by permitting real parallelism in the processing of network traffic. As of FreeBSD 5.2, it was possible to run low level network functions, as well as the IP filtering and forwarding plane, without the Giant lock, as well as "process to completion" in the interrupt handler.
Work continues to improve the maturity and completeness of the locking (and performance) of the network stack for 5.3. The network stack locking development branch has been updated to the latest CVS HEAD, tracking a variety of FreeBSD changes, including tracking and driving changes in the interface and device cloning APIs, push-down and fixes to locking in the Berkeley Packet Filter, consistency improvements in allocation flags for network objects, diagnosis of excessive acquisition of Giant in various system callouts and timeouts, removal of Giant from several system callouts, "const"-ification of a number of global variables in the network stack (IPv4, IPv6, elsewhere) as part of ananalysis of locking requirements, fine-grain locking of a number of pseudo-interfaces (disc, loopback, faith, stf, gif, tap, tun), IP encapsulation and tunneling, initial review and locking of parts of PPP and SLIP, experimentation with PCB assertions on IPv6, additional socket locking assertions, graphing of the FreeBSD sockets layer to support locking analysis, merging of theMT_TAG to m_tag conversion to improve the ability to queue packets, moving of the debug.mpsafenet tunable to controlling Giant over the forwarding plane to Giant over the entire stack("dual-mode" to support non-MPSAFE protocols), adaption of existing network lock assertions to also assert Giant when running non-MPSAFE, analysis of high cost of select() locking, improved locking and synchronization annotations, TCP callouts run MPSAFE, logtimeout() runs MPSAFE, uma_timeout() runs MPSAFE, callout sampling instrumentation, loadav() runs MPSAFE, AppleTalk locking begun: AARP locked down and DDP analysis, rawcb list locked, locking analysis of mrouter and IP ID code, IGMP locked, IPv6 analysis begun, IPX/SPX analysis begun, PPP timeouts converted to callouts, Netgraph analysis begun. Many of these changes have not yet been merged to the main FreeBSDtree, but this is a work in progress.
In related work on Pipe IPC (not quite network stack locking), substantial time was invested in diagnosing an increase in the cost of pipe allocation since FreeBSD 4.x, as well as coalescing the several allocations needed to create a pipe, as well as moving to slab allocation so as to amortize the cost of pipe initialization. Future work here will include caching the VM structures supporting pipe buffers.
Recent contributors include Robert Watson, Sam Leffler, MaxLaier, Maurycy Pawlowski-Wieronski, Brooks Davis, and many others who are omitted here only by accident.
2004 continues on with wonderful progress. Work continues on locking
down the network stack, ACPI made more great strides, an ARM port
appeared in the tree, and the FreeBSD 4.10 release cycle wrapped up.
Once 4.10 is released, the next big focus will be FreeBSD 5.3. We
expect this is be the start of the 5-STABLE branch, meaning that not
only will it be stable for production use, it will also be largely
feature complete and stable from an internal API standpoint. We expect
to release 5.3 in mid-summer, and we encourage everyone to download the
latest snapshots from
Thanks,
Scott Long
After almost three years efforts for porting OpenOffice.org 1.0.x and 1.1.0 for FreeBSD by Martin Blapp (mbr@FreeBSD.org) and other contributors, There are four version of OpenOffice.org (OOo) in ports tree. 1.1.1: stable version, 1.1.2: next stable, 2.0: developer and 1.0.3: legacy.
Stable version 1.1.1 in /usr/ports/editors/openoffice-1.1/ builds/installs/works fine for 5.2.1-RELEASE. Packages for 5.2.1-RELEASE, 26 localized versions and 4.10-PRELEASE only English version, are available at http://oootranslation.services.openoffice.org/pub/OpenOffice.org/ooomisc/ (note: source of OOo 1.1.1.RC3 is identical OOo 1.1.1)
Patches needed to build are currently 18 for 1.1.1, and 161 for 1.0.3 the number of patches are greatly reduced.
OOo 1.1.2, the next stable version in /usr/ports/editors/openoffice-1.1-devel is also builds/installs/works fine for 5.2.1-RELEASE. We are planning to upgrade this port as soon as 1.1.2 will be released.
Next major release, 2.0 (planned to be released at January 2005 according to http://development.openoffice.org/releases/OpenOffice_org_trunk.html), /usr/ports/editors/openoffice-2.0-devel, now compiles for 5.2.1-RELEASE but have big problem that prohibits to remove BROKEN.
Legacy version, OOo 1.0.3: /usr/ports/editors/openoffice-1.0/ I'm not interested in this port. We hope someone else will maintain this.
For builds, my main environment is 5.2.1-RELEASE, and I have no access to 4-series, so several build problems had been reported for 5-current and 4-stable, however, they now seems to be fixed. Please make sure your Java and/or kernel are up-to-date.
For version 1.1.1, yet we have serious reproducible core dumps, this means OOo cannot pass the Quality Assurance protocol of OpenOffice.org (http://qa.openoffice.org), so we cannot release OOo as quality assured package. It seems to be FreeBSD's userland bug, since some reports show that there are no problem for 4-stable but we still searchingthe reason.
Note that developers should sign JCA (Joint Copyright Assignment) before submitting patches via PR or e-mail, otherwise patches won't be integrated to OOo's source tree. We seriously need more developers, testers and builders.
An enhanced network interface cloning API has been created. It allows interfaces to support more complex names than the current name# style. This functionality has been used to enable interesting cloners like auto-configuring vlan interfaces. Other features include locking of cloner structures and the ability of drivers to reject destroy requests. A patch has been posted to the freebsd-net mailing list for review and will be committed in early May. This work is taking place in the perforce repository under: //depot/user/brooks/xname/...
The FreeBSD Dutch Documentation Project is a ongoing project in translating the handbook and other documentation to the Dutch language. Currently we have a small team of individuals who translate, check other's work, and publish them on the internet. You can view the current status on the webpage (listed above). Still we can use more people helping out, since we have a long way to go. Every hand that wants to help, contact me, and i will provide you details on how we work etc. Currently the project has translated the handbook pages of: The X Windows System, and Configuration and Tuning, they only need to be checked before publishing.
Much of the ACPI project is waiting for architectural changes to be completed. For instance, the cpufreq driver requires newbus attachments for CPUs. Support code for this should be committed at the time of publication. Other architectural changes needed include rman support for memory/port resources and a generic hotkey and extras driver. Important work in other areas of the kernel including PCI powerstate support and APIC support have been invaluable in improving ACPI on modern platforms. Thanks go to Warner Losh and John Baldwin for this work.
Code which is mostly completed and will go in once the groundwork is finished includes the cpufreq framework, an ACPI floppy controller driver, and full support for dynamic Cx states.
ACPI-CA was updated to 20040402 in early April. This has some GPE issues that persist in 20040427 that will hopefully be resolved by the date of publication.
I'd like to welcome Mark Santcroos (marks@) to the FreeBSD team. He has helped in the past with debugging ACPI issues. If any developers are interested in assisting with ACPI, please see the ACPI TODO and send us an email.
The verify source reachability option for ipfw2 has been committed on 23. April 2004 to FreeBSD-CURRENT. For more information see the links above.
ipfw2 is built directly into ip_input() and ip_output() and it makes these functions more complicated. For some time now we have the generic packet filter mechanism PFIL_HOOKS which are used by IPFILTER and the new OpenBSD PF firewall packages to hook themselves into the IP input and output path.
This patch makes ipfw2 fully self contained and callable through the PFIL_HOOKS. This is still work in progress and DUMMYNET and IPDIVERT plus Layer2 firewall are not yet fully functional again but normal firewalling with it works just fine.
The patch contains some more cleanups of ip_input() and ip_output() that is work in progress too.
The ARP IP address to MAC address mapping does not belong into the routing table (FIB) as it is currently done. This will move it to its own hash based structure which will be instantiated per each 802.1 broadcast domain. With this change it is possible to have more than one interface in the same IP subnet and layer 2 broadcast domain. The ARP handling and the routing table will be quite a bit simplified afterwards. As an additional benefit full MAC address based accounting will be provided.
Luigi has become the driver of this project and posted a first implementation for comments on 25. April 2004 (see link).
The current TCP send and receive buffers are static and set to a conservative value to preserve kernel memory. This is sub-optimal for connections with a high bandwidth*delay product because the size of the TCP send buffer determines how big the send window can get. For high bandwidth trans-continental links this seriously limits the maximum transfer speed per TCP connection. A moredetailed description from the last status report can be found with the link above.
Work on this project has been stalled due to some other network stack projects with higher precedence (ipfw2 to pfil_hooks and ip_input/ip_output cleanups).
Both bsdtar and libarchive are now part of -CURRENT. A few minor problems have been reported and addressed, including performance issues with many hard-links, and options required by certain packages. For now, the "tar" command is still an alias for "gtar." Those who would like to use bsdtar as the default system tar can define WITH_BSDTAR to make "tar" be an alias for "bsdtar."
My current plan is to make bsdtar be the default in -CURRENT in about another month, probably after the 5-STABLE split, and remove gtar from -CURRENT sometime later. It's still open if and when this switch will occur in 5-STABLE. On the one hand, I see potential problems if 5-STABLE and 6-CURRENT have different tar commands; on the other hand, switching could be disruptive for some users.
GEOM Gate class is now committed as well as ggatec(8), ggated(8) and ggatel(8) utilities. It makes distribution of disk devices through the network possible, but on the disk level (don't confuse it with NFS, which provides exporting data on the file system level).
New locales: Unicode UTF-8 locales have been added to the base system. All of the locales previously supported by FreeBSD now have a corresponding UTF-8 version, along with one or two new ones -- 53 in all.
Library changes: The restartable conversion functions (mbrtowc(), wcrtomb(), etc.) in the C library have been updated to handle partial characters in the way prescribed by the C99 standard. The <wctype.h> functions have been optimized for handling large, fragmented character sets like Unicode and GB18030. Documentation has been improved.
Utilities: The ls utility has been modified to work with wide characters internally when determining whether a character in a filename is printable, and how many column positions it takes on the screen. Character handling in the wc utility has been made more robust. Other text-processing utilities (expand, fold, unexpand, uniq) have been modified, but these changes have not been committed until the performance impact can be evaluated. Work on a POSIX-style localedef utility has started, with the aim to have it replace the current mklocale and colldef utilities in FreeBSD 6. (It is currently on the back-burner awaiting a response to a POSIX defect report.)
Future directions: wide character handling functions need to be optimized so that they are more competitive with the single-byte functions when dealing with 8-bit character sets. Utilities need to be modified to handle multibyte characters, but with a careful eye on performance. Localedef needs to be finished.
formats (Intel, AMI) is also in the works.
The two months after the import was done were actually rather quiet. We imported a couple of minor fixes from the OpenBSD stable branch. The import of tcpdump 3.8.3 and libpcap 0.8.3 done by Bruce M.Simpson in late March finally put us into the position to build a working pflogd(8) and provide rc.d linkage for it. Tcpdump now understandsthe pflog(4) pseudo-NIC packet format and can be used to read the log-files.
There has also been work behind the scenes to prepare an import of the OpenBSD 3.5 sources. The patches are quite stable already andwill be posted shortly. Altq is in the making as well and going alongquite well based on the great work from rofug.ro, but as it needs modifications to every network driver which have to be tested thoroughly it needs more time.
We have finished about 75% of the Handbook translation work. In the last two months we primarily worked on bringing the handbook chapters more up to date. To make the translation more high quality we are also doing some revision on it.
We are still looking for manpower on SGML'ifying the FAQ translation which has been done last year by several volunteers.
ctau(4) driver for Cronyx Tau-ISA was added. Cronyx Tau-ISA is family of synchronous WAN adapters with various set of interfaces such as V.35, RS-232, RS-530(449), E1 (both framed and unframed). This is a second family of Cronyx adapters that is supported by FreeBSD now. The first one was Cronyx Sigma-ISA, cx(4).
Cronyx Tau-PCI family will become a third one. The peculiarity of this driver that it contains private code. This code is distributed as obfuscated source code with usual open source license agreement.Since code is protected by obfuscation it is satisfy needs of commerce. On the other hand it still stays a source code and thus it becomes closer to open source projects. I hope this form of private code distribution will become a real alternative to object form.
As part of my work on synchronous protocol stack a ng_sppp driver was added to the system. This driver allows to use sppp as a Netgraph node. Now I plan to update sppp driver as much as possible to make it in sync with Cronyxs one (PPP part). Also I work on FRF.12 support in FreeBSD (now I have FRF.12 support for Netgraph and SPPP (and for Cronyx linux fr driver) but only End-to-End). I plan to test it by my self within a week and after that I plan to make full support of FRF.12.
If you want to get current version and test it, please feel free to contact me.
Threading developers have been active behind the scenes though not much has been visible. Real Life(TM) has been hard on us as a group however.
Marcel and Davidxu have both (individually) been looking at the support for debugging threaded programs. David has a set of patches that allow gdb to correctly handle KSE programs and patches are being considered for libthr based processes. Marcel added a Thread ID to allow debugging code to unambiguously specify a thread to debug. He has also been looking at corefile support. Both sets of patches are preliminary.
Dan Eischen continues to support people migrating to libpthreads and it seems to be going well.
Doug Rabson has done his usual miracle work and produced a set of preliminary patches to implement TLS (Thread Local Storage) for the i386 platform.
Julian Elischer is investigating some refactoring of the kernel support code.
Platforms:
i386, amd64, ia64 libpthread works.
alpha, sparc64 not implemented.
Having recently passed its first birthday, FreeBSD Update is now being used on about 170 machines every day; on a typical day, around 60 machines will download updates (the others being already up to date). To date, over 157000 files have been updated on over 4200 machines.
Lazy allocation of pci resources has been merged into the main tree. These changes allow FreeBSD to run on computers where PnP OS is set to true. In addition, the saving and restoring of the resources across suspend/resume has helped some devices come back from suspend.
Future work will focus on bus numbering.
The new Book "The Design and Implementation of the FreeBSD Operating System" is the successor of the legendary "The Design and Implementation of 4.4BSD" book which has become the de-facto standard for teaching of Operating System internals in universities world-wide.
This new and completely reworked edition is based on FreeBSD 5.2 and the upcoming FreeBSD 5.3 releases and contains in-details looks into all areas (from virtual memory management to interprocess communication and network stack) of the operating system on 700 pages.
It is now in final production by Addison-Wesley and will be available in early August 2004. The ISBN is 0-201-70245-2.
This patch if for if_wi current. It enables you to disable the ssid broadcasting and it also allows you to disable clients connecting with a blank ssid.
Several folks continue to work on the locking the network stack as noted elsewhere in this report. Outside of the network stack, the following items were worked on during the March and April time frame. Giant was pushed down in the fork, exit, and wait system calls as far as possible. Alan Cox (alc@) continues to lock the VM subsystem and push down Giant where appropriate. A few system calls and callouts were marked MP safe as well.
A few changes were made to the interrupt thread infrastructure. Interrupt thread preemption was finally enabled on the Alpha architecture with the help of the recently added support to the scheduler for pinning threads to a specific CPU. An optimization to reduce context switches during heavy interrupt load was added as well as rudimentary interrupt storm protection.
FreeBSD/arm is now in the FreeBSD CVS tree. Dynamic libraries now work, and NO_CXX=true NO_RESCUE=true buildworld works too (with patches for toolchain that will live outside the tree for now). Now the focus should be on xscale support.
Work has begun on locking down the CAM subsystem. The project is divided into several steps:
While the immediate goal of this work is to lock CAM, it also points us in the direction of separating out the SCSI-specific knowledgefrom the core. This will allow other transports to be written, such as SAS, iSCSI, and ATA.
Progress is being tracked in the FreeBSD Perforce server in the camlock branch. I will make public patches available once it has progressed far enough for reasonable testing. So far, the first two items are being worked on.
This project is aimed at converting the FreeBSD network stack from running under the single Giant kernel lock to permitting it to run in a fully parallel manner on multiple CPUs (i.e., a fully threaded network stack). This will improve performance/latency through reentrancy and preemption on single-processor machines, and also on multi-processor machines by permitting real parallelism in the processing of network traffic. As of FreeBSD 5.2, it was possible to run low level network functions, as well as the IP filtering and forwarding plane, without the Giant lock, as well as "process to completion" in the interrupt handler.
Work continues to improve the maturity and completeness of the locking (and performance) of the network stack for 5.3. The network stack development branch has been updated to the latest CVS HEAD, as well as the following and more:
Netperf patches are proving to be quite stable in a broad variety of environment, as long as non-MPSAFE chunks are avoided. Kqueue, IPv6, and ifnet locking remain the most critical areas where additional functionality is required. Focus is shifting from new development to in depth testing, performance measurement, and interactions with other subsystems.
This work would not be possible without contributions from the following people (and no doubt many others): John Baldwin, Bob Bishop, Brooks Davis, Pawel Jakub Dawidek, Matthew Dodd, Julian Elischer, Ruslan Ermilov, John-Mark Gurney, Jeffrey Hsu, Kris Kennaway, Roman Kurakin, Max Laier, Sam Leffler, Scott Long, Rick Maklem, Bosko Milekic, George Neville-Neil, Andre Oppermann, Luigi Rizzo, Jeff Roberson, Tim Robbins, Mike Silberback, Bruce Simpson, Seigo Tanimura, Hajimu UMEMOTO, Jennifer Yang, Peter Wemm. We hope to present these patches on arch@ within a few days, although some elements required continued refinement (especially socket locking).
The TrustedBSD Mandatory Access Control (MAC) Framework permits the FreeBSD kernel and userspace access control policies to be adapted at compile-time, boot-time, or run-time. The MAC Framework provides common infrastructure components, such as policy-agnostic labeling, making it possible to easily development and distribute new access control policy modules. Sample modules include Biba, MLS, and Type Enforcement, as well as a variety of system hardening policies.
The TrustedBSD MAC development branch in Perforce was integrated to the most recent 5-CURRENT.
mdmfs(8) -l to create multi-label mdmfs file systems (merged).
Diskless boot updated to support MAC.
Re-arrangement of MAC Framework code to break out mac_net.c into mac_net.c, mac_inet.c, mac_socket.c (merged).
libugidfw(3) grows bsde_add_rule(3) to automatically allocate rule numbers (merged). ugidfw(8) grows 'add' to use this (merged).
pseudofs(4) no longer requires MAC localizations.
BPF fine-grained locking now used to protect BPD descriptor labels instead of Giant (merged).
Prefer inpcb's as the source of labels over sockets when creating new mbufs throughout the network stack, reducing socket locking issues for labels.
TrustedBSD "Security-Enhanced BSD" (SEBSD) is a port of NSA's SELinux FLASK security architecture, Type Enforcement (TE) policy engine and language, and sample policy to FreeBSD using the TrustedBSD MAC Framework. SEBSD is available as a loadable policy module for the MAC Framework, along with a set of userspace extensions support security-extended labeling calls. In most cases, existing MAC Framework functions provide the necessary abstractions for SEBSD to plug in without SEBSD-specific changes, but some extensions to the MAC Framework have been required; these changes are developed in the SEBSD development branch, then merged to the MAC branch as they mature, and then to the FreeBSD development tree.
Unlike other MAC Framework policy modules, the SEBSD module falls under the GPL, as it is derived from NSA's implementation. However, the eventual goal is to support plugging SEBSD into a base FreeBSD install without any modifications to FreeBSD itself.
Integrated to latest FreeBSD CVS and MAC branch.
New FreeBSD code drop updated for capabilities in preference to superuser checks.
Installation instructions now available!
The TrustedBSD Project is producing an implementation of CAPP compliant Audit support for use with FreeBSD based on the Apple Darwin implementation.
Experimentally integrated the XNU audit implementation from Apple's Darwin 7.2 into Perforce.
Adapted audit framework to compile into FreeBSD -- required modifying memory allocation and synchronization to use FreeBSD SMPng primitives instead of Mach primitives. Pushed down the Giant lock out of most of the audit code, various other FreeBSD adaptations such as suser() API changes, using BSD threads, td->td_ucred, etc.
Adapted per-thread audit data to map to FreeBSD threads
Cleaned up userspace/kernel API interactions, including udev_t/ dev_t inconsistencies between Darwin and FreeBSD.
Use vn_fullpath() instead of vn_getpath(), which is a less complete solution we'll need to address in the future.
Basic kernel framework now operates on FreeBSD; praudit tool written that can parse FreeBSD BSM and Solaris BSM.
This installment of the Bi-Monthly Status Report is a few days late, but I'm pleased to say that it is chocked full of over 30 articles. May and June were yet again busy months; the Netperf project passed major milestones and can now be run with the debug.mpsafenet tunable turned on from sources in CVS. The ARM, MIPS, and PPC ports saw quite a bit of progress, as did several other SMPng and Netgraph projects. FreeBSD 5.3 is just around the corner, so don't hesitate to grab a snapshot and test the progress!
On a more serious note, it's very important to remember that code freeze for FreeBSD 5.3 will happen on August 15, 2004. This is only a few weeks away and there is still a lot to do. The TODO list for the release can be found at http://www.freebsd.org/releases/5.3R/todo.html. If you are looking for a way to contribute to the release, this TODO list has several items that are in urgent and in need of attention. Testing is also very important. The tree has had some stability stability problems in the past few weeks, but there are work-arounds that should allow everyone to continue testing and using FreeBSD. We absolutely must have FreeBSD 5.3 be a rock-solid release, so every little bit of contributed effort helps!
Thanks,
Scott Long
This project is aimed at converting the FreeBSD network stack from running under the single Giant kernel lock to permitting it to run in a fully parallel manner on multiple CPUs (i.e., a fully threaded network stack). This will improve performance/latency through reentrancy and preemption on single-processor machines, and also on multi-processor machines by permitting real parallelism in the processing of network traffic. As of FreeBSD 5.2, it was possible to run low level network functions, as well as the IP filtering and forwarding plane, without the Giant lock, as well as "process to completion" in the interrupt handler. This permitted both inbound and outbound traffic to run in parallel across multiple interfaces and CPUs.
Work continues to improve the maturity and completeness of the locking (and performance) of the network stack for 5.3. The network stack development branch has been updated to the latest CVS HEAD, as well as the following and more. Many but not all of these changes have been merged to the FreeBSD CVS tree as of the writing of this report. Complete details and more minor changes are documented in the README file on the netperf web page.
In the past two months, opportunities to perform a good chunk of work on FreeBSD/MIPS have arisen and significant issues with context switching, clocks, interrupts, and kernel virtual memory have been resolved. A number of issues with caches were fixed, however those are far from complete and at last check, there were issues when running cached which would prevent booting sometimes. Due to toolchain issues in progress, current kernels are no longer bootable on real hardware.
A 64-bit MIPS emulator has arisen giving the ability to test and debug in an emulator, and much testing has taken place in it. It has been added to the FreeBSD ports tree, and the port will be actively tracking the main codebase as possible. In general, FreeBSD/MIPS kernels should run fine in it.
Before toolchain and cache issues, the first kernel threads would run, busses and some devices would attach, and the system would boot to a mountroot prompt.
The port has been moving along steadily. There have been reports of buildworld running natively. Works is almost complete on make release so there will be bootable CD images in the near future.
IPFilter has been upgraded in both FreeBSD-current and 4-STABLE (post 4.10) from version 3.4.31 to 3.4.35.
The current design attempts to support both per-process and system-wide statistical profiling and per-process "virtual" performance counters. The userland API libpmc(3) is somewhat stable now, but the kernel module's design is being redone to handle MP better. Initial development is targeting the AMD Athlon CPUs, but the intent is to support all the CPUs that FreeBSD runs on.
An early prototype is available under Perforce [under //depot/user/jkoshy/projects/pmc/].
FreeBSD profile.sh is an enhancement to the FreeBSD 5 rcng boot system, targeted at laptops. One can configure multiple network environments (eg, home, work, university). After this initial configuration, the laptop detects automatically in what environment it is started and configures itself accordingly. Not only network settings, but almost everything from under /etc can be configured per environment. It is also possible to suspend the machine in one environment and wake it up in a different one, and reconfiguration will happen automatically.
Currently I work on two directions: if_spppfr.c and sppp locking (on behalf of netperf). At the moment of writing this sppp locking is not ready yet. But it would be ready in couple of days. Also you may find as a part of this work some user space fixes for rwatson netperf code (Only that I was able to catch while world compilation. If you know some others let me know and I'll try to fix them too).
Since sppp code is quite big and state machine is very complicated, it would be difficult to test all code paths. I will glad to get any help in testing all this stuff. More tester more probability to test all possible cases.
Work on FRF.12 (ng_frf12) is frozen since of low interest and lack of time. Current state of stable code: support of FRF.12 End-to-End fragmentation. Support of FRF.12 Interface (UNI and NNI) fragmentation is not tested.
cp(4) driver for Cronyx Tau-PCI was added. Cronyx Tau-PCI is family of synchronous WAN adapters with various set of interfaces such as V.35, RS-232, RS-530(449), X.21, E1, E3, T3, STS-1. This is a third family of Cronyx adapters that is supported by FreeBSD now. Now all three drivers cx(4), ctau(4) and cp(4) are on both major branches (HEAD and RELENG_4).
Busdma conversion was recently finished. Current work is concentrated on locking both for adapters drivers and for sppp (see my other report for additional information).
An enhanced network interface cloning API has been committed. It
allows interfaces to support more complex names then the current
name# style. This functionality has been used to
enable interesting cloners like auto-configuring vlan interfaces.
Other features include locking of cloner structures and the ability
of drivers to reject destroy requests.
Work on userland support for this functionality is ongoing.
Not a lot happened on the SMPng front outside of the work on locking the network stack (which is a large amount of work). The priorities of the various software interrupt threads were corrected and locking for taskqueues was improved. The return value of the sema_timedwait() function was adjusted to be more consistent with cv_timedwait(). A small fix was made to the sleepqueue code to shorten the amount of time that a sleepqueue chain lock is held when waking up threads. Some simple debug code for profiling the hash tables used in the sleep queue and turnstile code was added. This will allow developers to measure the impact of any tweaks to the hash table sizes or the hash algorithm.
Support for programming the polarity and trigger mode of interrupt sources at runtime was added. This includes a mini-driver for the ELCR register used to control the configuration for ISA and EISA interrupts. The atpic driver reprograms the ELCR as necessary, while the apic driver reprograms the interrupt pin associated with an interrupt source as necessary. The information about which configuration to use mostly comes from ACPI. However, non-ACPI systems also force any ISA interrupts used to route PCI interrupts to use active-low polarity and level trigger.
Support for suspend and resume on i386 was also slightly improved. Suspend and resume support was added to the ELCR, $PIR, and apic drivers.
The ACPI PCI-PCI bridge driver was fixed to fall back to the PCI-PCI bridge swizzle method for routing interrupts when a routing table was not provided by the BIOS.
Mixed mode can now be disabled or enabled at boot time via a loader tunable.
The work on converting the build switches/OPTIONS currently present in the ports of the main KDE modules into separate ports in order to make packages available for the software/features they provide is progressing. Porting of KOffice 1.3.2 are nearly completed. The Swedish FreeBSD snapshot server http://snapshots.se.freebsd.org, operated and maintained by members of the KDE/FreeBSD team, is back up and running at full steam. Additional amd64 hardware has been added and amd64 snapshots will be available soon.
I'm working on various GEOM classes. Some of them are already committed and ready for use (GATE, CONCAT, STRIPE, LABEL, NOP). The MIRROR class is finished in 90% and will be committed in very near future. Next I want to work on RAID3 and RAID5 implementations. Userland utility to control GEOM classes (geom(8)) is already in the tree.
The Third Edition of the FreeBSD Handbook has been split into two volumes. The first volume, the User Guide, has been published. Work is progressing on the second volume. The following chapters are included in the second volume : advanced-networking, network-servers, config, boot, cutting-edge, disks, l10n, mac, mail, ppp-and-slip, security, serialcomms, users, vinum, eresources, bibliography, mirrors. Please see the Task List for information about what work remains to be done. In addition to technical and grammatical review, a number of HTML output assumptions in the document need to be corrected.
The portaudit utility is currently an add-on to FreeBSD designed to give administrators and users a heads up with regards to security vulnerabilities in third party software. The VuXML database keeps a record of these security vulnerabilities along with internal security holes. When installed, the portaudit utility periodically downloads a database with known issues and checks all installed ports or packages against it; should it find vulnerable software installed the administrator or user is notified during the daily run output of the periodic scripts.
These utilities are considered to be of production quality and discussion is taking place over whether or not they should be included as part of the base system. All ports committers are urged to add entries when when a vulnerability is discovered; any questions may be sent to eik@ or myself.
Bluetooth code was marked as non-i386 specific. It is now possible to build it on all supported platforms. Please help with testing. Other then this there was not much progress during last few months. I've been very busy with Real Life.
The FreeBSD Dutch Documentation project is a ongoing project translating the FreeBSD handbook {and others} to the dutch language. We are still on the look for translators and people that are willing to check the current html documentation. If you are interested, contact me at the email address shown above. We currently are reading for some checkups and then insert the first documents into the documentation tree.
The FreeBSD Brazilian Documentation Project is an effort of the Brazilian FreeBSD Users Group (FUG-BR) to translate the available documentation to pt_BR. We are proud to announce that we've finished the Handbook and FDP Primer translation and they are being revised. Both should be integrated to the FreeBSD CVS repository shortly.
There are many other articles being translated and their status can be checked at our website. If you want to help please create an account at BerliOS, since our CVS repository is being hosted there, and contact us through our mailing list. Any help is welcome!
We imported pf as of OpenBSD 3.5 stable on June, 17th which will be the base for 5-STABLE pf (according to the current schedule). The most important improvement in this release is the new interface handling which makes it possible to write pf rule sets for hot-pluggable devices and pseudo cloning devices, before they exist. The import of the ALTQ framework enabled us to finally provide the related pf functions as well.
Before 5-STABLE we will import some bug fixes from OpenBSD-current, which have not been merged to their stable branch, as well as some FreeBSD specific features. The planned ALTQ API make-over will also affect pf.
We are (desperately) looking for non-manpage documentation for FreeBSD pf and somebody to write it. Few things have changed so a port of the excellent "PF FAQ" on the OpenBSD homepage should be fitting. There are, however, a couple of points that need conversion. A simple tutorial how to setup a NAT gateway with pf would also help. The in-kernel NAT engine is very easy to use, we should tell people about this alternative. This is even more true since the pf module now plugs into GENERIC without modifications.
The ALTQ framework is part of KAME for more than 4 years and has been adopted by Net- and OpenBSD since more than 3 years. It provides means of managing outgoing packets to do QoS and bandwidth limitations. OpenBSD developed a different way to interact with ALTQ using pf, which was adopted by KAME as the "default for everyday use".
The Romanian FreeBSD Users Group has had a project to work towards integration of ALTQ into FreeBSD, which provided a very good starting point for the final import. The import only provides the "pf mode" configuration and classification API as the older ALTQ3 API does not suit to our SMP approach.
A reworked configuration API (decoupled from pf) is in the making as are additional driver modifications. Both should be done before 5-STABLE is branched, although additional drivers can be imported during the lifetime of 5-STABLE as well.
HP Network Scanjet 5 can unobtrusively run FreeBSD inside the scanner. Those who miss their Unix at work can have a FreeBSD box, un-noticed & un-challenged by blinkered managers who block any non Microsoft PC in the building. http://berklix.com/scanjet/
Registration for EuroBSDCon 2004 taking place in Karlsruhe, Germany, from Oct. 29th to 31st has just opened. An early bird discount will be offered to all registering until Aug. 15th. Please see the conference website for details.
The buf-junta project is underway, I am trying to bisect the code such that we get a struct bufobj which is the handle and method carrier for a buffer-cache object. All vnodes contain a bufobj, but as filesystems get migrated to GEOM backing, bufobj's will exist which do not have an associated vnode. The work is ongoing.
An effort to get the tty subsystem out from under Giant has morphed into an more general effort to eliminate a lot of code which have been improperly copy & pasted into device drivers. In an ideal world, tty drivers would never get near a cdevsw, but since some drivers are more than just tty drivers (for instance sync) a more sensible compromise must be reached. The work is ongoing.
KGI is going slowly but surely. The port of the KGI/Linux accel to FreeBSD is in progress. It's no more than a double buffering API for graphic command passing to the HW engine.
Most of the work in the past months was about console management and more especially dual head console. Otherwise a new driver building tree is now ready to compile Linux and FreeBSD drivers in the same tree.
Documentation about KGI design is in progress.
The system continues to function well. The accuracy of the automatic classification algorithm has been improved by assigning a higher priority to port names found in pieces of Makefiles.
Several bugs had to be fixed due to the transition from bento to pointyhat. For about two weeks the URLs to the build errors were wrong. This has now been corrected (but note that some of the pointyhat summary pages themselves still show the broken links.)
A report was added to show only PRs in the 'feedback' state, so that committers can focus on maintainer and/or responsible timeouts. (As a reminder, the policy is 2 weeks). Another report on 'ports that are in ports/MOVED, but still exist' has also been added to the Anomalies page. Sometimes these are actual errors but not always.
Here are my latest observations about the trends in ports PRs:
Many more text-processing utilities in the FreeBSD base system have been updated to work with multibyte characters, including comm, cut, expand, fold, join, paste, unexpand, and uniq. New versions of GNU grep and GNU sort (from coreutils) have been imported, together with multibyte support patches from developers at IBM and Red Hat.
Future work will focus on modifying the regular expression functions to work with multibyte characters, improving performance of the C library routines, and updating the remaining utilities (sed and tr are two important ones still remaining).
Evaluation board.
Not much coding has taken place on this lately, with the recent focus being on refining the design. We are currently investigating per-CPU completion queues and threads in order to reduce locks and increase concurrency. Also reviewing the BSD/OS CAM lockdown to see what ideas can be shared. Work should hopefully puck back up in late July. Development is taking place in the FreeBSD Perforce repository under the //depot/projects/scottl-camlock/... branch for now.
Project Mini-Evil is an attempt to extend Bill Paul's 'Project Evil' Windows NDIS wrapper layer to the SCSI MiniPort and StorePort layers. While drivers exist for most storage controllers that are on the market today, many companies are integrating software RAID into their products but not providing any source code or design specs. Instead of constantly reverse-engineering these raid layers and attempting to shoehorn them into the ata-raid driver, Project Mini-Evil will run the Windows drivers directly. It will hopefully also run most any SCSI/ATA/RAID drivers that conform to the SCSI Miniport or Storeport specification.
Work on this project is split between making the NDIS wrapper code more general and implementing the new APIs. Development is taking place in the FreeBSD Perforce repository under the //depot/projects/sonofevil/... branch.
With the release of FreeBSD 5.5 and FreeBSD 6.1, the second quarter of 2006 has been productive. Google is sponsoring 14 students to work on FreeBSD as part of their Summer of Code Program (most of which already submitted a report for elaboration on their projects).
Sun's open-source software is starting to make its way into FreeBSD as a port of DTrace is nearing completion and a port to the UltraSparc T1 processor (which gives a great push to the ongoing SMP efforts). Having a powerful debugging tool combined with a CPU that can run up to 32 concurrent threads helps to identify scalability issues.
BSDCan 2006 was yet again a smashing success and much was covered in the 2-day developer summit. As a product of the conference, a new focus on FreeBSD for the embedded sector has started. Various ARM boards are targeted, a MIPS32 port is gearing up and people are looking for other interesting platforms to port FreeBSD to. Preparation for the EuroBSDCon (in Milan, Italy) on November has already issued a call for papers.
In addition, a lot of spring cleaning is taking place in the network stack. After conclusion of the KAME project, IPv6 code integration has been refocused and a fully locked port of SCTP is in the final stage of integration. Of course, all this goes without noting all the progress made with the other network projects.
Please read below for more detailed news on the projects that happened in FreeBSD during the last three months. If you are interested in helping, consider the "Open Tasks lists" provided with some reports. In addition we would like to point you at the list of projects and ideas for volunteers and hope to receive a status report from you next time.
Thanks to all reporters for your excellent work and timing! Enjoy reading.
BSDCan 2006 continues to impress. Again this year, we had a good collection of talks from a wide range of speakers. In all, we had over 200 people from 14 different countries.
Our sponsorship pool continues to grow. This year we had sponsorship from:
The 2007 planning has already begun and we look forward to another popular and successful event.
My thanks to the 2006 program committee, the speakers, the volunteers, the sponsors, and, of course, the attendees.
See you at BSDCan 2007.
The release engineering (RE) team announced the availability of FreeBSD 5.5 and 6.1, both in May 2006. FreeBSD 5.5 is the last planned release from the RELENG_5 branch in CVS. For the most part, its main features consist of bugfixes, security patches, and minor updates. We encourage users to move towards the 6.x series of releases whenever practical. FreeBSD 6.1 is the latest of the releases to come from the RELENG_6 branch in CVS. It includes (among many other things) improved support for WiFi devices, additional network and disk controller drivers, and a number of fixes for filesystem stability. The next release to be issued from this branch will be FreeBSD 6.2, which is currently scheduled for September 2006.
The RE team is currently in a ``between releases'' mode. Current activities include working with security-team@ on some errata fixes for the RELENG_6_1 branch and producing snapshots of HEAD and RELENG_6 at the start of each month.
Several personnel changes have taken place recently. Scott Long has stepped down from his position on the RE team; we thank him for his considerable efforts over the past four years. In his place, Ken Smith has taken over the role of lead release engineer. Bruce A. Mah has rejoined the RE team after a two-year sabbatical.
For some time now I have been working on converting the existing USB device drivers to my new and mutex enabled USB API. I have converted "ulpt", "ums", "uhid", "ukbd", "ugen", "uaudio", and a few others. Around 10 USB device drivers are left to convert. Most of these are network device drivers.
At the present moment I am working on getting scatter and gathering support working for all USB host controllers. Scatter and gathering means that one allocates PAGE_SIZE bytes of memory at a time, and then fills these memory blocks up as much as possible with USB host controller structures and buffers. This should solve problems allocating DMA-able memory when the system memory becomes fragmented.
Some FPU system and kernel memcpy/copyin/copyout changes have been performed. In particular, a per-CPU save area has been introduced (protected with an interlock) in order to assure a stable saving mechanism. copyout/copyin have changed in order to use vectorised version of memcpy and an xmm version of memcpy has been provided.
As part of my SoC 2006 project I am working on implementing a BRIDGE monitoring module for FreeBSD's BSNMP daemon. Initial prototyping is done and some kernel changes are coming to be able to access all needed data. In addition to IETF RFC 4188, which was designed for monitoring a single bridge, this snmp module will support monitoring of multiple bridge devices as supported by FreeBSD.
Anonymous enablings now work. There is a new option in the boot loader menu to load the DTrace modules and trace the kernel boot process.
Sun Microsystems has been very supportive of the FreeBSD port and has generously provided a Sun Fire T2000 server to allow Kip Macy's sun4v port to be merged into the DTrace project tree.
The DTrace project tree sources are now exported to cvsup10.freebsd.org
Refer to the project page for more details.
There are several projects moving forward in the embedded area. For now the main location for new information is www.embeddedfreebsd.org. We have also created a new mailing list, freebsd-embedded@freebsd.org , which is meant to eventually replace the freebsd-small. A call was put out on small for people to move to embedded.
This year's EuroBSDCon will be held in Milan, Italy, on November 10th through 12th.
Hosted in the foggy northern Italy, the fifth EuroBSDCon aims at being a new successful chapter in the itinerant series of European BSD conferences.
EuroBSDCon represents the biggest gathering for BSD developers from the old continent, as well as users and passionates from around the World. It is also a chance to share experiences, know-how, and cultures.
For the first time, parallel to the main event, an event for wives/girlfriends/friends will be organised. It will consist of guided tours of the city of Milan, a probable trip to Como and visits to various museums. We're also working towards offering a show at the Teatro alla Scala.
The FreeBSD developer summit will be also held on November 10th.
Continuing to add IPv6 support to FAST_IPSEC. Test environment is now stable. Can build and run kernels with FAST_IPSEC and INET6 enabled but IPSec in IPv6 is now broken and being worked on.
FreshPorts has seen several new features recently:
A 2U server was recently donated to the FreshPorts / FreshSource / FreeBSD Diary / BSDCan group. We have also received a RAID card. Now we're looking for some hard drives.
Over the past few weeks, work has concentrated on benchmarking the new server and getting it ready for production. Eventually it will need a new home as I don't really want it running in my basement all the time (it's really loud!).
Thanks to iXsystems and 3Ware for their contributions to this project.
GJournal is a GEOM class which provides journaling for GEOM providers. It can also be used to journal various file systems with just a minimal filesystem-specific portion of code. Currently only UFS journaling is implemented on top of gjournal. Being filesystem-independent and operating below the file system level, gjournal has no way to distinguish data from metadata, thus it journals both. One of the nice things about gjournal is that it works reliable even on disks with enabled write cache, which is often not the case for journalled file systems. And remember... fsck no more.
The purpose of gvirstor module is to provide the ability to create a virtual storage device of arbitrarily large size (typically several terabytes) which consists of an arbitrary number of physical storage devices (actually any lower-level GEOM providers, including RAID devices) of arbitrary size (typically 50 GB - 400 GB hard drives). Storage space from these components is carved into small chunks (for example 4 MB) and allocated (committed) to the virtual device on as-needed basis.
Development has started and is progressing as planned (though a little bit slow). Metadata format and virtual storage allocation formats have been defined and more serious coding is in progress.
The FreeBSD list of projects and ideas for volunteers is doing well. Several items were picked up by volunteers and have found their way into the tree. Others are under review or in progress. We are looking forward to hear about new ideas, people willing to act as technical contacts for generic topics such as USB or specific entries (already existing or newly created) and suggestions for existing entries or completion reports for (parts of) an entry.
Initial changes include:
Clement has been working both with libnet and gnn's Python based packet library (PCS) to produce code to test for vulnerabilities in IPv6. To Clement has found some issues, all of which have been reported to his mentor and to Security Officer at FreeBSD.org Vulnerabilities will not be reported here.
Project is in development with initial working software expected mid-July 2006. CPU limits will be implemented with a hierarchical scheduler: (initially) using a round-robin scheduler to select which jail to run a task in and then delegating which task in the jail to be run to a per-jail scheduler.
A simple lexer and parser have almost been completed. Also significant planing for future additions to K have been thought up.
FreeBSD linux emulation layer (linuxolator) currently implements most of the functionality necessary to emulate 2.4.2 linux kernel, but linux world has moved forward and current linux world requires 2.6.x features. The aim of this SoC task is to make Fedora Core 4 linux-base to be able to run with 2.6.x kernel. Currently this means extending clone() syscall and implement pthread related things. This involves TLS implementation (sys_set_thread_area syscall) and possibly tid manipulation (used for pthread_join etc.) and finally futexes (linux fast user-space mutexes implementation). This should enable pthread-linked programs to work. After this is done there may be other things necessary to implement however, only time will tell. I am funded by google.com in their SoC to do this work and I'll continue to work on this after the summer hopefully as a part of my MSc. thesis.
The improved support for the i386 binaries are ready for -exp run. It only allows installing such ports on amd64 and ia64 when there's a compatibility layer compiled into the kernel and the 32-bit libraries are installed under /usr/lib32.
The DESTDIR support are in progress. It works for the simplest ports without USE_* that don't have a [pre|do|post]-install target. There are more complicated issues with e.g. conflict checking in DESTDIR, deinstalling from DESTDIR, those have to be fixed as well.
The translated webpage is almost ready now. This Hungarian translation is a "lite" version of the original English webpages, since there are parts that are irrelevant for the Hungarian community, or has pieces of data that change quickly, so it's no use to translate these pages now, maybe later, if we have more Hungarian contributors, but this webpage would be a good starting point in translating the documentations, and we need a good place to put translated documentations anyway.
I'm going to be very busy with SoC this summer, but I'll try to find people that can help me out in this project. Any help appreciated.
As an intermediate step until FreeBSD will have full network stack virtualisation this work shall provide support for multi-IP IPv4/v6 jails.
These changes are based on Pawel Jakub Dawidek's work for multi-IPv4 jails and some initial work from Olivier Houchard for single-IPv6 jails.
The changes need some more testing but basically things work.
This is not considered to be the right thing todo so do
not
ask for official support or if this will be committed to the
FreeBSD source repository.
After some more cleanup of non-jail related IPv6 changes I will
publish a patch for HEAD and perhaps RELENG_6 for everyone who
wants to give it a try anyway.
Mohan Srinivasan committed his changes to make the NFSv2/3 client MP safe to HEAD this quarter. Changes may be back-ported to 6.x soon.
Robert Watson and Chuck Lever held a discussion about the future of the in-kernel NFSv4 client during BSDCan 2006. The current NFSv4 client is unmaintained. Chuck also pointed out the long series of unfixed PRs against the legacy client (NFSv2/3). These are at the top of his priority list. Robert is also interested in making NFSv4-style ACLs the lingua franca for FreeBSD file systems. There was some discussion about integrating Rick MacKlem's NFSv4 server into 7.x.
Chuck Lever became a full source committer during this quarter.
The basic goals of this SoC 2006 project are moving nsswitch-modules out of the libc, extending the caching daemon and importing nss_ldap into the base source tree. 2 milestones of the project are currently completed.
1. Nss-modules were successfully moved out of the libc into the separate dynamic libraries. In order for static binaries to work properly (they can't use dynamic nss-modules), nss-modules are linked statically into the libc.a. As the side-effect of nss-modules separation, getipnodeby***() functions were rewritten to use gethostby***() functions and not the nsdispatch(3) call. Caching daemon's "perform-actual-lookups" option was extended to support all implemented nsswitch databases.
2. A set of regressions tests was made to test nsswitch-related functions. These tests are also capable of testing the stability of these functions' behaviour after the system upgrade.
pfSense is rapidly approaching release. We are down to a handfull of bugs that should be fixed in the coming weeks. We should have a release around the time of our 2nd annual hackathon which is taking place on July 21st - July 28th. Many exciting sub-projects are taking place within pfSense and the project is gaining new developers monthly.
As an intermediate step towards implementing support for callgraphs and cross-architecture performance measurements, I am creating a BSD-licensed library for ELF parsing & manipulation. This library will implement the SysV/SVR4 (g)ELF[3] API.
Current status: Implementation of the library is in progress. A TET-based test suite for the API and manual pages documenting the library's interfaces are being concurrently created.
Work is being done in FreeBSD's Perforce repository. I hope to be ready for general review by the end of July '06.
During this time, a huge number of ports PRs were committed, bringing us back down below 800 for the first time since the 5.5/6.1 release cycle. This is due to a great deal of work, especially from some of our newest committers.
This is all the more notable given the fact that we have been adding new ports at a rapidly accelerating rate. We have now exceeded the 15,000 port mark!
Three sets of changes have been added to the infrastructure, including updates of default versions of MySQL, PHP, LDAP, and linux_base, and numerous bugfixes and improvements. About 2 dozen portmgr PRs were closed due to this.
In addition, a large-impact commit was made that attempts to move us to a single libtool that is as unmodified from 'stock' libtool as we can. Plans are also in place to do this for the autotools.
Several people are at work on implementing the modularised xorg ports. Most of the work is done but several key pieces remain. Once this is finished, an -exp regression test will be needed (most likely, more than one :-) ) It is possible that before this we will need to do a regression test that moves X11BASE back into LOCALBASE. This is still under study.
Gabor Kovesdan started a Google Summer of Code project on some highly needed improvements on the ports infrastructure (see elsewhere in this report). As this is a long term project, gtetlow kindly imported the most important ports infrastructure files into perforce to ease development. Other developers are encouraged to use perforce for ports development, especially as it can help keeping patches up-to-date while going stale in GNATS. Even though linimon has been pushing hard on running experimental builds on the test cluster, it will take some time to work through the backlog.
erwin added a ports section to the list of projects and ideas for volunteers at the FreeBSD website. Have a look if you want to work on the ports system. Don't hesitate to send additional ideas, and committers are encouraged to add themselves as technical contacts.
sem adopted portupgrade after it had been neglected for some time and has been very active on upgrades and bugfixing.
dougb has continued to enhance his portmaster script and people are finding success with it; although not designed to be as full-featured as portupgrade, it does seem to be easier to understand and use.
shaun has contributed portscout, a scanner for updated distfiles, to the ports collection.
marcus upgraded GNOME to 2.14.1.
As well, there have been new releases of the ports tinderbox code.
edwin has been hard at work on a PR-autoassigner for ports PRs, which has saved a lot of time and been well-received. It has now been installed on a freebsd.org machine (hub).
linimon has been more active in pursuing maintainer-timeouts, and has reset a number of inactive maintainers, with more in the pipeline. The intent is to try to reduce the number of PRs that sit around unanswered for two weeks. In almost all cases the resets are due to no response at all; maintainers who are merely "busy" are not the source of most of these problems, and deserve the benefit of the doubt. Some of the maintainers that have been reset haven't contributed in months or even years.
We have added 10 (!) new committers since the last report.
Since the last status report ports have been created for all parts of the BSDInstaller except the backend.
A snapshot of the BSDInstaller was released during this quarter. This has shown a number of bugs with the installation process. Most have now been fixed.
The patches to allow UFS operate with quotas in Giant-less mode are brewed for long now. Since recent huge pile of fixes into snapshots code, I think the problems you could encounter are caused solely by the patch.
Aside performance benefits, patch has another one, much more valuable. It makes UFS operating in one locking regime whatever options are compiled into kernel. I think, in long term, that would lead to better stability of the system.
We updated the default linux base port to Fedora Core 4 and the default linux X11 libs port to the X.org RPM in FC4.
An update to FC5 or FC6 has to wait until the kernel got support for syscalls of a newer linux kernel. See the corresponding SoC project report for more.
Since the last status report we fixed some more bugs, added basic support for envy24 chips and cleaned up the source for the emu10kx driver in the ports to make it ready for import into the base system.
We also got some patches with a little bit of infrastructure for Intel HDA support. It's not finished and also not usable by end users yet.
The XFS for FreeBSD project is an effort to port the publically available GPL'd sources to SGI's XFS filesystem to FreeBSD.
In December, we imported a version of XFS into FreeBSD-CURRENT which allows FreeBSD to mount an XFS filesystem as read-only.
As a side effort, we have been continuing on the work that PHK started to clean up the mount code in FreeBSD. We can use the existing FreeBSD mount(8) utility to mount an XFS partition, without introducing a new mount_xfs utility.
For the last several months Randall Stewart has been working in HEAD and STABLE to get us ready to integrate the SCTP protocol (Stream Transmission Control Protocol) into FreeBSD. He is currently working on a patch to share with a wider audience but needs to do some integration work first. Randall has a provisional commit bit and will be working with gnn on getting code committed to the HEAD of the tree.
In the time since the last status report, four security advisories have been issued concerning problems in the base system of FreeBSD; of these, one problem was "contributed" code, while three were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 71 new entries have been added, bringing the total up to 757.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD 6.0, and FreeBSD 6.1. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 5.3 and FreeBSD 5.4 will cease to be supported at the end of October 2006, while FreeBSD 6.0 will cease to be supported at the end of November 2006.
I have been working on porting missing features in gvinum from vinum, as well as adding new features.
So far the resetconfig, detach, dumpconfig, setstate (on plexes and volumes) and stop commands have been implemented, as well as some other minor fixes. The attach command is currently being implemented, and started on disk-grouping. Currently most of this is in p4, but patches will be submitted as soon as possible.
The wireless support has been stable for a while so most work has focused on bug fixing and improving legacy drivers.
Max Laier and I worked on improving support for Intel wireless cards. The results of this work included significant improvements to the iwi(4) driver (for 2195/2200 parts) and the firmware(9) facility for managing loadable device firmware. There is also an updated ipw(4) that has improvements similar to those done for iwi that is in early test. Support for the latest Intel devices, the 3945 pci-express cards, is planned for later this summer.
Atheros support was updated with a new hal that fixes a few minor issues and provides known working builds for SPARC, PPC, and ARM platforms. There is also working MIPS support that will be used when the MIPS port is ready to test. Otherwise one useful bug was fixed that affected AP operation with associated stations operating in power save mode.
wpa_supplicant and hostapd were updated to the latest stable build releases from Jouni Malinen.
Experimental changes to support injection of raw 802.11 frames using bpf were posted for comment. This work was done in collaboration with Andrea Bittau.
With the help of Jim Thompson of Netgate ( http://www.netgate.com/ ) the FreeBSD Foundation arranged a purchase of xscale-based boards for folks interested in ARM support. Developers were able to purchase boards at a reduced cost. The goals were to accelerate and/or improve support for the ARM platform and to set forth at least one board as a reference platform for the ARM support. Netgate will be stocking lower-cost models of the board later in the year (a special order was made for boards with only 2 mini-pci slots).
With the introduction of fine grained locking in the SMPng project, the FreeBSD kernel went under a major redesign, and many subsystem changed significantly with it. In particular, device driver's interrupt context ("the bottom half") had the necessity to synchronise with process context ("the top half") and share data in a consistent manner without using spl*(). To overcome this problem, a new interrupt model based around interrupt threads was employed, together with a fast interrupt model dedicated to particular driver handlers that don't block on locks (i.e. serial port, clock, etcetc). Unfortunately, even if the interrupt thread model proved to be a reliable solution, its performance was not on par with the pre SMPng era (4.x), and thus others solutions were investigated, with interrupt filtering being one of that.
As part of my Summer of Code 2006 work, I'm implementing interrupt filtering for FreeBSD, and when the framework will be in place I'll compare the performance of filters, against all the previous models: pre-SMPng(4.x), ithread and polling.
The most important modifications to the src tree so far were:
The project is slowly starting to ramp up after a long move-induced hiatus.
Alan Cox has almost completed making the pmap module Giant-free.
TrustedBSD Audit provides fine-grained security event auditing in FreeBSD 7.x, with a planned merge to 6.x for FreeBSD 6.2. Work performed in the last three months:
This report covers FreeBSD related projects between June and October 2006. This includes the conclusion of this year's Google Summer of Code with 13 successful students. Some of last year's and the current SoC participants have meanwhile joined the committer ranks, kept working on their projects, and improving FreeBSD in general.
This year's EuroBSDCon in Milan, Italy has meanwhile published an exciting program. Many developers will be there to discuss these current and future projects at the Developer Summit prior the conference. Next year's conference calendar has a new entry - in addition to the now well established BSDCan in Ottawa - AsiaBSDCon will take place in Tokyo at the begining of March.
As we are closing in on FreeBSD 6.2 release many bugs are being fixed and new features have been MFCed. On the other hand a lot of the projects below already are focusing on FreeBSD 7.0 and promise a lot of exciting news and features to come.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
Most dhclient changes in HEAD have been merged to 6-STABLE for 6.2-RELEASE. The highlight of these changes is a fix for runaway dhclient processes when packets are not 4 byte aligned. Further changes including always sending client identifiers are scheduled for merge before the release. Work is ongoing to improve dhclient's interaction with alternate methods of setting interface addresses.
The FreeBSD/arm port has grown support for the Atmel AT91RM9200. Boards based on this machine are booting to multiuser off either NFS or an SD card. The onboard serial ports, PIO, ethernet and SD/MMC card controllers are well supported. Support for the SSC, IIC and SPI flash parts in the kernel will be forthcoming shortly.
In addition to normal kernel support, the port includes a boot loader that can initialize memory and boot off IIC eeprom, SPI DataFlash, BOOTP/TFTP and SD memory cards.
The port will be included in forthcoming commercial products.
We had another successful summer taking part in the Google Summer of Code. By all accounts, the FreeBSD participation in this program was an unqualified success. We received over 150 applications for student projects, amongst which 13 were selected for funding. All successful students received the full $4,500.
These student projects included security research, improved installation tools, new utilities, and more. Many of the students have continued working on their FreeBSD projects even after the official close of the program. At least 2 of our FreeBSD mentors will be meeting with Google organizers in Mountain View this month to discuss the program at the Mentor Summit.
The FreeBSD Release Engineering team is currently working on FreeBSD 6.2-RELEASE, which is scheduled for release in early November 2006. Some notable features of this release include the debut of security event auditing as an experimental feature, Xbox support, the FreeBSD Update binary updating utility, and of course many fixes and updates for existing programs. Pre-release images for all Tier-1 architectures are available for testing now; feedback on these builds is greatly appreciated. More information about release engineering activities can be found at the links above.
The focus of this project was to review past vulnerabilities, create vulnerability testing tools and to discover new vulnerabilities in the FreeBSD IPv6 stack which is derived from the KAME project code. During the summer Clement took two libraries, the popular libnet, and his mentor's Packet Construction Set (PCS) and created tools to find security problems in the IPv6 code. Several issues were found, bugs filed, and patches created. At the moment Clement and George are editing a 50 page paper that describes the project which will be submitted for conference publication.
All of the code from the project, including the tools, is online and is described in the paper.
By all measures, this was a successful project. Both student and mentor gained valuable insight into a previously externally maintained set of code. In addition to the new tools development in this effort, the FreeBSD Project has gained a new developer to help work on the code.
This project consisted in the improvement of the Interrupt Handling System in FreeBSD: while retaining backward compatibility with the previous models (FAST and ITHREAD), a new method called 'Interrupt filtering' was added. With interrupt filtering, the interrupt handler is divided into 2 parts: the filter (that checks if the actual interrupt belong to this device) and the ithread (that is scheduled in case some blocking work has to be done). The main benefits of interrupt filtering are:
Moreover, during the development of interrupt filtering, some MD dependent code was converted into MI code, PPC was fixed to support multiple FAST handlers per line and an interrupt stray storm detection logic was added. While the framework is done, there are still machine dependent bits to be written (the support for ppc, sparc64, arm and itanium has to be written/reviewed) and a serious analysis of the performance of this model against the previous one is a work-in-progress
We now have support for limiting CPU and memory use in jails. This allows fairer sharing of a systems' resources between divergent uses by preventing one jail from monopolizing the available memory and CPU time, if other users and jails have processes to run.
The code is currently available as patches against RELENG_6, and Chris is in the process of applying it to -CURRENT. More details can be found at JailResourceLimits on the wiki.
For me, the Google Summer of Code was a new and very exciting experience. I got actively involved in doing Open Source Software and giving something back to the community. Facing some challenges within the project forced me to look behind the scenery of FreeBSD. The result was a better understanding of the overall project. Working with a lot of developers directly also gave a very special spirit to the Google Summer of Code.
I really enjoyed the time and will continue to work on the project after the deadline. For me, it was a great chance to get involved in active development and not just some scripts and hacks at home. Getting paid for the work was just a small part of the overall feeling.
Thanks to the people at the FreeBSD Project and Google for the really, really great time!
The Project consisted of five parts:
Though none of the code was committed yet into the official FreeBSD tree, my experience from the previous year makes me think that this situation is normal. I hope, that the code will be reviewed and committed in the coming months.
Dongmei Liu spent the summer working on the basic footwork required to port the SEREF policy to SEBSD. This work has been submitted and can be viewed in the soc2006/dongmei_sebsd Perforce branch. This work was originated from the SEBSD branch: //depot/projects/trustedbsd/sebsd. Additionally setools-2.3 was ported from Linux and can be found in contrib/sebsd/setools directory. It is hoped that this work will be merged into the main SEBSD development branch.
There are currently patches available for testing. A planned integration to HEAD is set to happen in October.
Moved the HTML pages into the project CVS tree.
First working version of code. Does not pass all TAHI tests, but does pass packets correctly and does not panic.
During the last three months I have finished reworking nearly all USB device drivers found in FreeBSD-7-CURRENT. Only two USB drivers are left and that is ubser(4) and slhci. Some still use Giant, but most have been brought out of Giant. At the moment I am looking for testers that can test the various USB device drivers. Some have already been tested, and confirmed to work, while others have problems which need to be fixed. If you want to test, checkout the USB perforce tree or download the SVN version of the USB driver that is available on my homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at: freebsd-usb@freebsd.org.
This iSCSI initiator kernel module and its companion control program are still under development, but the main parts are working.
GJournal seems to be finished. I fixed the last serious bug and it is now stable and reliable in our tests. I'm planning to commit it really soon now.
The work was sponsored by home.pl
Web site is up and we're soliciting papers and presentations. Some tutorials are already scheduled. Email secretary@asiabsdcon.org if you have questions or submissions.
In the previous quarter we primarily focused on overall quality of the translation rather than just increasing the number of translations, and we have strived to make sure that these translated stuff are up-to-date with their English revisions. Also, we have merged the translated website into the central repository.
In the next quarter we will focus on developing documentation that will help to attract more developers.
EuroBSDCon 2006 is taking place in Milan (Italy), from the 10th to the 12th of November.
EuroBSDCon represents the biggest gathering for BSD developers from the old continent, as well as users and passionates from around the World. It is also a chance to share experiences, know-how, and cultures.
The program is rich in talks about FreeBSD, with topics ranging from "How the FreeBSD ports collection works" to "Interrupt Filtering in FreeBSD". This means that both the novice and the hacker can enjoy the conference.
Registration is open. The EuroBSDCon Organizing Committee hopes to see you in Milan.
Since the last status report, there has been a lot of progress. I investigated a lot of charset issues and found out that HTML tidy breaks some entities when using iso-8859-2, so HTML tidy had to be disabled for Hungarian pages.
In the time since the last status report, six security advisories have been issued concerning problems in the base system of FreeBSD; of these, five problems were in "contributed" code, while one was in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 57 new entries have been added, bringing the total up to 814.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD 6.0, and FreeBSD 6.1. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 5.3 and FreeBSD 5.4 will cease to be supported at the end of October 2006, while FreeBSD 6.0 will cease to be supported at the end of November 2006 (or possibly a short time thereafter in order to allow time for upgrades to the upcoming FreeBSD 6.2).
I spent the months of May through August working on improving Portsnap, FreeBSD Update, and devoting more time to my (continuing) role as Security Officer. FreeBSD Update is now part of the FreeBSD base system and is fully supported by the FreeBSD Security Team; updates are currently only being built for the i386 architecture, but AMD64 updates will become available soon.
In an attempt to reduce the number of people running out of date (and unsupported) FreeBSD releases, I wrote an automatic binary upgrade script for upgrading systems from FreeBSD 6.0 to FreeBSD 6.1; I will be releasing a new script for upgrading to FreeBSD 6.2-(RC*|RELEASE) soon (possibly before this status report is published).
Further improvements to Portsnap are still ongoing.
My work is moving slowly forward. ZVOL is, I believe, fully functional (I recently fixed snapshots and clones on zvols), which means you can put UFS on top of RAID-Z volume, take a snapshot of the volume, clone it if needed, etc. Very cool. The hardest part is the ZPL layer, I'm still working on it. Most file system methods work, but probably need detailed review and many fixes. Most of the time these days I'm spending on implementing mmap(2) correctly. It works more or less in simple tests but fails under fsx program. On the other hand, 'fsx -RW' works very stable and reliable. Other test programs (those that don't use mmap(2)) also work quite well. There is still a lot of work to do, mostly in ZPL area, many clean-ups, etc. Some functionality (like ACLs) I haven't even tried to touch yet.
TSO - TCP Segmentation Offload support has been committed to the network stack of FreeBSD-current in September 2006. With TSO, TCP can send data in the send socket buffer in bulk down to the network card which then does the splitting into MTU sized packets. On bulk high speed sending the performance is increased by 25% (normal writes) to 108% (sendfile). Jack Vogel and Prafulla Deuskar of Intel committed the driver changes for TSO hardware support of em(4) based network cards.
These changes are scheduled to be backported to FreeBSD 6-STABLE shortly after FreeBSD 6.2-RELEASE is published to appear in upcoming FreeBSD 6.3 early next year.
This work was sponsored by the TCP/IP Optimization Fundraiser 2005.
The addition of TSO (TCP Segmentation Offload) has highlighted some shortcomings in the sendfile(2) and sosend_*() kernel implementations.
The current sendfile(2) code simply loops over the file, turns each 4K page into an mbuf and sends it off. This has the effect that TSO can only generate 2 packets per send instead of up to 44 at its maximum of 64K. kern_sendfile() has been rewritten to work in two loops, the inner which turns as many pages into mbufs as it can -- up to the free send socket buffer space. The outer loop then drops the whole mbuf chain into the send socket buffer, calls tcp_output() on it and then waits until 50% of the socket buffer are free again to repeat the cycle. This way tcp_output() gets the full amount of data to work with and can issue up to 64K sends for TSO to chop up in the network adapter without using any CPU cycles. Thus it gets very efficient especially with the readahead the VM and I/O system do.
Looking at the benchmarks we see some very nice improvements: 181% faster with new sendfile vs. old sendfile (non-TSO), 570% faster with new sendfile vs. old sendfile (TSO).
The current sosend_*() code uses a sosend_copyin() function that loops over the supplied struct uio and does interleaved mbuf allocations and uiomove() calls. m_getm() has been rewritten to be simpler and to allocate PAGE_SIZE sized jumbo mbuf clusters (4k on most architectures). m_uiotombuf() has been rewritten to use the new m_getm() to obtain all mbuf space in one go. It then loops over it and copies the data into the mbufs by using uiomove(). sosend_dgram() and sosend_generic() have been changed to use m_uiotombuf() instead of sosend_copyin().
Looking at the benchmarks we see some very nice improvements: 290% faster with new sosend vs. old sosend (non-TSO), 280% faster with new sosend vs. old sosend (TSO).
Newly written is a specific soreceive_stream() function for stream protocols (primarily TCP) that does only one socket buffer lock per socket read instead of one per data mbuf copied to userland. When doing netperf tests with WITNESS (full lock tracking and validation enabled) the receive performance increases from ~360Mbit/s to ~520Mbit/s. Without WITNESS I could not measure any statistically significant improvement on a otherwise unloaded machine. The reason is two-fold: 1) per packet we do a wakeup and readv() is pretty much as many times as packets come it, thus the general overhead dominates; 2) the packet input path has a pretty high overhead too. On heavily loaded machines which do a lot of high speed receives a performance increase should be measureable.
The patches are scheduled to be committed to FreeBSD-current at end of October or early November 2006.
This work was sponsored by the TCP/IP Optimization Fundraiser 2005.
As a participant of Google's Summer of Code 2006, I am focusing on porting Xen to FreeBSD these months. The result of this summer's work include a domU kernel that could be used for installation, a guide for getting started with FreeBSD on Xen, and some other trivial improvements. But there are still a lot of work needing to be done in this area, e.g, the long-expeted dom0 support. So I will continue my work here and try to keep up with the update of Xen itself.
Gvirstor is a GEOM class providing virtual ("overcommit") storage devices larger than physical available storage, with possibility to add physical storage on-line when the need arises. Current status is that it's done and waiting commit to HEAD, scheduled for some time after 6.2 is released.
The ports PRs surged (especially due to a large number of new port submissions), but with some hard work we have been able to get back down to around 900. We are rapidly approaching 16,000 ports.
Due to this acceleration in adding new ports, portmgr is now very concerned that we are outstripping the capacity of both the build infrastructure and our volunteers to keep up with build errors and port updates. Accordingly, we've added a guideline (not a rule) that ports should be of more than just theoretical use to be added to the Ports Collection (e.g. we can't support all of CPAN + all of Sourceforge + everything else). Basically, use common sense as a guideline; certainly no one wants to see any kind of "gateway" procedure to get incoming ports approved.
Seven sets of changes have been added to the infrastructure, mostly refactoring and bugfixing.
As part of a Summer of Code project, we have also incorporated some of gabor@'s changes to incorporate better DESTDIR support. However, due to some unanticipated side-effects, more work is going to be needed in this area. gabor@ is continuing to work on the changes.
netchild@ and bsam@ have been doing a great deal of work to bring the linux emulator ports closer to sanity, including bringing up a regression-test suite.
The long-anticipated import of X.Org 7 has stalled due to developer time, mostly to deal with documentation and upgrade instructions. Hopefully this can get done in the early 6.3 development cycle. See the wiki for more information.
As a part of that work, the decision has been made to move away from using X11BASE and just put everything into LOCALBASE; /usr/X11R6 is simply an artifact at this point. A plan for a transition process is underway; a great deal of testing will need to be done, but in the end the ports tree will be much cleaner. The GNOME team has already done the work to move all of their ports over, and it will be incorporated after the 6.2 release is shipped.
tmclaugh@ is looking for someone to take over the C# ports. He has maintained them for over a year and wants more time to be able to work on other projects.
Some work has been done to get rid of FreeBSD 2.X cruft in ports. Further work is needed to get the 3.X cruft removed.
linimon@ did another pass through resetting inactive maintainers. Another list is waiting in the wings.
linimon@ is also working on adding the ability for portsmon to analyze successful packages (not just failed ones), so that queries such as "show me packages that build on i386 but not amd64" and "show me why dependent package foo was not built on bar". This is currently in alpha testing.
We have added 4 new committers since the last report.
CScout is a refactoring editor and source code browser for collections of C code. The aim of the project is to make it easy for FreeBSD developers to use CScout and to improve the FreeBSD source code quality through CScout-based queries and refactorings.
CScout was first applied to the FreeBSD kernel in 2003. Its application at that point involved substantial tinkering with the build system. The version released in October 2006 makes the running of CScout on the three Tier-1 architectures a fairly straightforward procedure. The current version can also draw a number of call graphs; this might help developers better understand foreign code.
Libelf is a BSD-licensed library for ELF parsing & manipulation implementing the SysV/SVR4 (g)ELF[3] API.
Current status: Implementation of the library is nearly complete. A TET-based test suite for the API is being worked on.
Progress this month has been limited due to my sea-change, moving house to the country.
Sun's OpenSolaris developers have followed through and released the DTrace test suite as part of the OpenSolaris distribution.
jkoshy@'s work on libbsdelf is nearing feature completion for DTrace and will make life easier in FreeBSD for DTrace, given that we have more architectures to support than Sun has.
The FreeBSD project has made available a dual processor AMD64 machine for DTrace porting.
I am currently working through the diffs between the DTrace project in P4 and -current, committing files to -current if they are ready.
The TrustedBSD audit implementation provides fine-grained security event logging throughout the FreeBSD operating system. The big news for the last quarter is that the TrustedBSD audit implementation has been merged into RELENG_6 branch, and appeared in 6.2-BETA2. Over the past few months, work has also occurred in the following areas:
Lots of testing as part of the 6.2-BETA cycle would be much appreciated. Audit support will be considered an experimental feature in FreeBSD 6.2-RELEASE, but we hope that it will be a production feature in 6.3-RELEASE.
The MMC/SD stack got a significant boost this quarter. Warner Losh and Bernd Walter have written a generic MMC/SD flash card stack for FreeBSD, and have implemented a host controller for the AT91RM9200 embedded ARM controller they are each using in separate projects.
The stack is presently experimental in quality. It is being used as the root file system for these embedded projects. There's been no work done to support hot insertion and removal of cards (neither board wires up the pins necessary, and besides, / disappearing is very bad). There are still many rough edges.
This is a freshly written stack. It has been written using the SD 1.0 (and recently 2.0) simplified specification, with the SanDisk MMC application notes supplementing. The Linux stack looks good, although not entirely standards conforming (there's work in progress that I've not seen that is supposed to fix this) and it is contaminated with the GPL. The OpenBSD stack also looks interesting, but Warner's experience porting NEWCARD over from NetBSD suggested that a fresh rewrite may be faster, at least for the bus and driver level. Since MMC is fairly simple, a port of the sdhci driver might be possible.
Please see the open tasks list.
Support for the UltraSparc T1 (Niagara) continues to improve. The code has recently been checked into public CVS under sys/sun4v.
It isn't clear whether or not I will have time to implement full logical domaining support before the APIs become publicly available. Testing indicates that substantial work will be needed before FreeBSD can take full advantage of all 32 threads.
Work on Xen support has slowly been continuing in perforce. The SOC student fixed several bugs and is continuing to work on it. Someone is needed who has the time to complete dom0 support and shepherd it production level stability.
Sufficient interest has been expressed in it that it probably makes sense to check it in to public CVS so that more people can try it out. Time permitting, I will bring it up to date and check it in the next month.
FreeSBIE is a FreeBSD based LiveCD.
On August 19th, Matteo Riondato, a member of the FreeSBIE staff, released an unofficial ISO, codename FreeSBIE GMV, based on FreeBSD -CURRENT (read the Announcement to download it). This is supposed to be the first in a series of four ISOs that will end up with the release of FreeSBIE 2.0. Matteo is now working on another ISO, codename FreeSBIE LVC, which is scheduled to be released October 12th.
FreeSBIE 2.0 will be based on FreeBSD 6.2-RELEASE and will hopefully be released at EuroBSDCon 2006 in Milan. It will be available for the i386 and AMD64 platforms.
Roman Divacky participated in the Google Summer of Code 2006 and implemented a major part of the syscall compatibility to the 2.6.16 Linux kernel. The work has been committed to -CURRENT (the default compatibility still being a 2.4.2 Linux kernel) and we are working on fixing the remaining bugs as time permits.
"Intron" submitted an implementation for the linux aio syscalls. His work has been committed to the Perforce repository.
We also started to consolidate a list of known bugs, open issues and helpful stuff (e.g. regression tests and their status) in -CURRENT on a page in the FreeBSD wiki (see the links-section). It also contains a link to a more or less up-to-date patch with stuff we have in the Perforce repository so that interested people can help with testing. Thanks to the help of Marcin Cieslak we already fixed some bugs (some of the fixes are already MFCed to -STABLE).
Thanks to the nice regression tests of the Linux Test Project (LTP) we have a list of small (and not so small) things which need to be looked at. This list makes up for a quick start into kernel hacking. So if you have a little bit of knowledge about C programming, and if you want to help us a little bit in improving FreeBSD, feel free to have a look at the list and to try to fix a problem or two. Sometimes it is as easy as "if (error condition) return Esomething;" (but you should coordinate with the emulation mailinglist, so that nobody does some work someone else just did too). Even if you do not know how to program, you can help. Have a look at the wiki page and tell us about things which should get mentioned there too. Or download the patch and test it.
Since the last status report we added basic support for envy24ht chips, imported the emu10kx driver into the base system and added support for High Definition Audio (HDA) compatible chips.
Additionally the work of Ryan Beasley as part of his Google Summer of Code 2006 participation is committed. It adds compatibility to the Open Sound System (OSS) v4 API as far as this was possible. This allows for more sophisticated programs to be written. For example it is now possible to synchronize the start of multiple sound channels. It is also possible for a driver to support more than the AC97 mixer devices, but so far no driver has been extended to support this yet. More about it can be found in the wiki and in the official OSS documentation.
The wiki page about the sound system was started to describe the current status of the sound system and to provide some information about where we are heading. But more work needs to be done to reach this goal. So far we collected some information about the status of the most recent work in the soundsystem. So if you have a look at it and you think that something important is missing, just tell us about it. While fully prepared content is very welcome, we are even happy about some ideas what we should list on the wiki page.
Work is almost finished to implement the Rapid Spanning Tree Protocol (RSTP) which supersedes Spanning Tree Protocol (STP). RSTP has a much faster link failover time of around one second compared to 30-60 seconds for STP, this is very important on modern networks. The code will be posted shortly for testing and feedback.
There were a number of OCaml ports in our tree, and each of them was doing the same work by maintaining OCaml ld.conf in the correct state, installing/removing their files/entries etc. To simplify the task of OCaml-language ports creation, the special framework (bsd.ocamk.mk) was developed and most of the ports were converted to use this framework. This allowed a lot of duplicate code to be removed. This new framework handles all the things required to install an OCaml-language library and properly register it. bsd.ocaml.mk also contains knobs to deal with findlib-powered libraries, modify ld.conf in the proper way, etc. Also, a lot of new Ocaml-related ports were added.
Integration of the new innovative e17 window manager into the ports tree is almost completed. A lot of new e17-related applications was ported, all old ports were updated to the latest stable cvs snapshot. The special framework (bsd.efl.mk) was created to support the whole thing and simplify the creation of dependent ports. I'll commit the changes in the days before the ports freeze.
Thanks to Sergey Matveychuk (sem@) for providing a machine to place CVS snapshots on. Without his help it will be impossible.
Last month I was working on a driver/module to update the microcode of Intel or AMD CPUs that support having their microcode updated. As you might know these processors are microcode-driven and this firmware can be updated. Intel(R) often releases microcode updates, and AMD(R) updates can be found in BIOS programs. The work is almost finished now, I just need to find a bit of time to test it on AMD64 systems and perform some code cleanup. The driver also provide a way for userland programs to access the Machine Specific Registers (MSR) and CPUID info for a certain cpu. This will allow some programs like x86info to provide more accurate information about cpus in SMP systems and make assumptions based on the contents of the MSR.
Thanks to John Baldwin, Kostik Belousov, John-Mark Gurney and Divacky Roman for helping during development.
During the Google Summer of Code 2006, Gábor worked on several ideas to improve the ports infrastructure:
The first three items have been completed and the next two items are being worked on. The DESTDIR support was more complicated than presumed and took more time than expected to complete. Gábor will continue working to finish these tasks and other ports related tasks. FreeBSD is happy to have interested him to keep working on ports and ports infrastructure.
I thought that since I sent a status report the last time, I might as well send one now.
Since the last status report I have done work on several of the remaining commands as attach, detach, and finally the concat command to be able to create concatenated volumes with one easy command. The mirror and stripe commands are the next step after this.
The most important thing I've been working on is maybe the implementation of drivegroups. I have posted a bit information on this mailinglists, but basically, it's a way to group drives with the same configuration. This way, you can make many commands operate on groups instead of drives, and the group-abstraction will handle how the underlying subdisks are created on the drives. In the future one will be able to move groups to different machines, etc.
I've created a patch of all my work that is not in HEAD yet here (this is a snapshot of my developement branch, so how thing's are done might be changed quite fast): http://folk.ntnu.no/lulf/patches/freebsd/gvinum/gvinum_all_current.diff
Be aware that a there will probably be bugs in the code, so don't use it in production yet!
Thanks to Greg Lehey for offering to help me on getting this into CVS, and all feedback on this has been good.
I have setup the FreeBSD Multimedia Resources List, a one-stop-shop for FreeBSD related podcasts, vodcasts and audio/video resources. Hopefully this list will make it easier for people to find and keep up to date with these recordings. The overview is available as a normal HTML page and as an XML/RSS feed.
The ultimate goal is to have this list to reside under the www.FreeBSD.org umbrella.
A BRIDGE monitoring module for FreeBSD's BSNMP daemon has been implemented. In addition to RFC 4188 single bridge support and extending the kernel to get access to all the information, a private MIB was designed in order to be able to monitor multiple bridges supported by FreeBSD. The kernel part has already been committed to -CURRENT (thanks to thompsa@), for -STABLE a patch is available (see the wiki), code has already been reviewed.
SoC 2005 work on SNMP client tools is now available too via port (net-mgmt/bsnmptools), thanks to Andrew Pantyukhin for the port.
The dates for BSDCan 2007 have been set: 11-12 May 2007. As is usual, BSDCan will be held at University of Ottawa, with two days of tutorials prior to the conference starting.
The call for papers will go out in mid December. Start thinking about your submissions now!
The new 2U server mentioned in the last report now has a collection of Raptor drives in a RAID-10 configuration. Thanks to very generous donations from the community, I purchased eight of these drives at very good prices. The server will be deployed in the next few weeks.
There has been quite a bit of work since the last report in June. Some highlights include:
For more detail, please review the FreshPorts Blog .
The FreeBSD Foundation continued to support the FreeBSD project and community through various activities. These activities include creating strategies for fund development and actively seeking funding for the FreeBSD community, coordinating a new IBM Bladeserver project, and protecting the image and integrity of FreeBSD by governing the use of the trademarks. We are pleased to be a sponsor of EuroBSDCon and will be sponsoring a few developers to attend the conference through our travel grant program. And finally, we have secured funds for a major project that will be announced later this month.
Happy New Year. This Report covers the last quarter of a exciting year 2006 for FreeBSD development. FreeBSD 6.2 is finally out of the door and work towards FreeBSD 7.0 is gearing up. Some of the projects in this report will be part of that effort, others are already in the tree. Many projects need your help with testing and otherwise. Please see the "Open tasks" sections for more information.
The BSD crowd will meet at AsiaBSDCon March 8-10th in Tokyo and a two day FreeBSD developer summit will be held at BSDCan May 16-19th in Ottawa. Finally, EuroBSDCon September 14-15th in Copenhagen is already looking for papers.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
A toy implementation of GEOM based active/passive multipath is now done and in a perforce repository. Seems to work.
There have been a number of improvements to FreshPorts over the last quarter of 2006. The following are just a few of them. The links take you to the relevant article within the FreshPorts News website .
My thanks to the many people who have contributed suggestions, ideas, and code over the years. Most of you are documented at the above URLs.
Folks!
It is that time of year. You may have missed the
call for papers
, but please put in your proposal right away. This is often a busy
time of year, but please take the time to consider presenting at
BSDCan.
Please read the submission instructions and send in your proposal today!
You may be interested in our sister conference: PGCon. If you have an interest in PostgreSQL , a leading relational database, which just happens to be open source, then we have the conference for you! PGCon 2007 will be held immediately after BSDCan 2007, at the same venue, and will follow a similar format.
FreeSBIE is approaching the 2.0-RELEASE. The first release candidate proved to be good enough but a second one will probably be released. An external developer is working on integrating BSDInstaller in FreeSBIE 2.0 and this may cause a little delay of the release date. Release Notes were written and need to be updated with the current list of packages. A script which allows to switch Tor+Privoxy on and off was added and its usage was documented. The 2.0-RELEASE is near, hopefully near the end of January but this will also depend on when FreeBSD 6.2-RELEASE will be released.
The 'mpt' project is support for the MPT LSI-Logic Host Adapters (SCSI, Fibre Channel, SAS).
The last quarter saw a lot of change supported by Yahoo! and LSI-Logic and many others as things settled out for better support for U320. Some initial Big Endian support was offered by John Birrel and Scott Long.
This project is for support for QLogic SCSI and Fibre Channel host adapters.
The last quarter saw the addition of 4Gb Fibre Channel support and a complete rewrite of fabric management (which is still settling out).
Basic audio capture is working. All of the parameters are set by userland, while the RISC program generation is by kernel. No real audio has been captured as there are no drivers for the NTSC tuner yet. Someone with a real Bt878 NTSC card that is supported by bktr(4) could use this to capture audio without using the sound card.
Due to lack of documentation from DViCO and LG, I have copied magic values from the Linux driver and managed to get ATSC capturing working. There was a bug in the capture driver that was releasing buffers to userland early causing what appeared to be reception issues. Now that we use the RISC status bits as buffer completion bits, capture works cleanly. This does mean that even if you provide more than 4 buffers to the driver, the buffers will be divided into four segments, and returned in segments.
A Python module is available, along with a sample capture application using it. The module is now known to work well with threads so that tuning (expensive due to i2c ioctls) can happen in another thread without causing program slow down. The module is working well with a custom PVR backend.
Additional ioctls have been added to get sibling devices. This allows one to open a bktrau device, and get the correct bktr(4) device that is in the same slot. This is necessary so that when adjusting GPIO pins or sending i2c commands, they are to the correct device.
Following the example of our NetBSD friends, we organized a couple of Bugathons to help decreasing the open PR count. At first, it was decided to make it a monthly event focused on both src, ports and doc. Audience decreased with each Bugathon organized and less non-ports committers attended the events. So from now on, we will focus on ports (making it a Portathon) and organize a new event after the end of each ports freeze (that should be twice a year, at most).
X.org 7.2 release has been delayed more than a month, which gave us more time to fix build failures, to work on a few runtime issues and to determine the easiest way to upgrade from 6.9 to 7.2 (mostly with the help of people on the freebsd-x11@ mailing list ). Everything is in a rather good shape but there's still a little amount of work to do. The merge of new ports is most likely to happen before the end of January.
During the last three months there has not been so much activity in the USB project. Some regression issues have been reported and fixed. Bernd Walter reports that he has got the new USB stack working on ARM processors with some minor tweaks. Markus Brueffer reports that he is working on the USB HID parser and support. A current issue with the new USB stack is that the EHCI driver does not work on the Sparc64 architecture. If someone has got a Sparc64 with FreeBSD 7-CURRENT on and can lend the USB project the root password, a serial console and a USB test device, for example a USB memory stick, that would be much appreciated. Another unresolved issue is that the ural(4) USB device driver does not always work. This is currently being worked on.
If you want to test the new USB stack, check out the USB perforce tree or download the SVN version of the USB driver from my USB homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@FreeBSD.org .
MPD is moving to the next major release - mpd4_0. At the end of October one more beta version (4_0b5) was released and first RC is planned soon.
Since 3_18 and 4_0b4 numerous bugs and cases of incorrect internal handling have been fixed. Performance has been increased and system requirements reduced.
Many new features have been implemented:
Some historically broken features have been reimplemented:
To support compression, two new Netgraph nodes ng_deflate and ng_pred1 have been created and the ng_ppp node has been modified.
Since the last status report we made good progress in improving the compatibility environment. We fixed more than 30 testcases on i386 (130 testcases = 16% still failing) and more than 60 testcases on amd64 (140 testcases = 17% still failing) in the Linux 2.4 compatibility. These numbers compare FreeBSD 6.2 with -CURRENT. Some of those fixes are edge cases in the error handling, and some of them fix real issues -- e.g. hangs -- and improve the stability and correctness of the emulation.
Regarding the Linux 2.6 compatibility there are 140 testcases (17%) on i386 and 150 testcases (18%) on amd64 still failing in -CURRENT. After fixing some showstopper problems with real applications, we should be able to give the 2.6 emulation a more widespread exposure "soon" to find more bugs and to determine the importance of those Linux syscalls which we did not implement yet.
The severity of the broken testcases varies, and some of them will never be fixed, e.g., we will never be able to load Linux kernel modules into a FreeBSD kernel, being able to add swap with a Linux command has very low priority, and fixing stuff which is used by applications like IPC type 17 has high priority.
Some differences in the 2.6 compatibility are because not all i386 changes are merged into the amd64 code, and some testcases are already fixed in our perforce repository but need more review before they can be committed to -CURRENT.
We need some more testers and bug reporters. So if you have a little bit of time and a favorite Linux application, please play around with it on -CURRENT. If there is a problem, have a look at the wiki if we already know about it and report on emulation@ . We are especially interested in reports about the 2.6 compatibility (sysctl compat.linux.osversion=2.6.16), but only with the most recent -CURRENT and maybe with some patches we have in the perforce repository (mandatory on amd64).
We thank all people who tested the changes / submitted patches and thus helped improving the Linux compatibility environment.
Since the last status report there were improvements to the emu10kx driver for High Definition Audio (HDA) compatible chips. Some more chips are supported now and already supported chips should provide a better zero-configuration experience.
The generic sound code got some very nice low latency changes, and fixes which make it multichannel/endian/format safe. We do not support multichannel operation yet, but this work is a prerequisite to work on implementing multichannel operation. This work also fixed some bugs which people may experience as clicks, hickups, truncation or similar behavior in the sound-output.
So far there is no merge to 5.x or 6.x planned for this code, especially because there are API/ABI changes, e.g., several sysctls changed. People who do not care about this can download binary sound modules from Ariff's download page for 6.x and 5.x.
We thank all people who tested the changes / submitted patches and thus helped improving the sound system.
Gábor Kövesdán (gabor@) has submitted the Hungarian translation of the webpages and Giorgos Keramidas (keramida@) has reviewed and committed the pages. The initial rendering issues have also been fixed and the webpage is in a pretty good shape now.
As usual, this translation does not contain every part of the English version, but the most important and useful parts are there. Gábor will maintain this translation and regularly sync the content with the English version and add new translations if such become available.
An initial port of the NetBSD wpi driver has been done and development is happening fast to get this driver ready for the tree. At present basic functionality works. The driver can associate with a non encrypted peer and pass data in 11b and 11g modes. There is still lots to do and testing is welcome.
Many thanks have to go to Sam, Max and Kip for helping the driver reach this point.
Though it is still a work in progress, it now supports more targets, has login CHAP authentication and header/data digest. It will also recover from a lost connection - most of the time.
Platform summary:
Currently the machine is booting FreeBSD 6.1-RELEASE-p10 and operating both single- and multi-user modes; below are highlights of available functionality:
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This will allow for complete networking independence between jails on a system, including giving each jail its own firewall, virtual network interfaces, rate limiting, routing tables, and IPSEC configuration.
The prototype currently virtualizes the basic INET and INET6 kernel structures and subsystems, including the TCP machinery and the IPFW firewall. The focus is currently being kept on resolving bugs and sporadic lockups, and defining the internal and management APIs. It is expected that within the next month the code will become sufficiently complete and stable for testing by early adopters.
The BSNMP bridge module for FreeBSD's BSNMP daemon, which was implemented during SoC 2006, was committed to HEAD. In addition to RFC 4188 single bridge support it also supports monitoring multiple bridges via a private MIB. Since SoC 2006 Rapid Spanning Tree (RSTP) support (RSTP-MIB defined in RFC4318 and additions to the private MIB) was added to the module as well.
A patch for RELENG_6 is available and will be merged to STABLE the next weeks.
During SoC 2005 BSNMP client tools (bsnmptools) were implemented and have since then been available via Shteryana's P4 tree or port net-mgmt/bsnmptools.
In order to finally get the code committed some cleanup was needed which ended in a partly rewrite to minimize duplicate code and to reduce the size of the binaries. This ongoing work is available via Bjoern's P4 tree and will be merged back to upstream trees before it will be committed to HEAD.
In addition to other more detailed reports this is intended to give a summary about other ongoing or upcoming BSNMP related work. To collect some ideas from users and coordinate work a BSNMP TODO Wiki page was created. Feel free to add your ideas or let us know about them.
The recent activities of the Release Engineering team have centered around FreeBSD 6.2-RELEASE, which is now available for downloading. This is the latest release from the RELENG_6 branch, and includes many new performance and stability improvements, bug fixes, and new features. The release notes and errata notes for FreeBSD 6.2 contain more specific information about what's new in this version. We thank the FreeBSD developer and user community for their efforts towards making this release possible.
The Release Engineering Team also produced snapshots of FreeBSD CURRENT in November 2006 and January 2007. These snapshots have not received extensive testing, and should not be used in production environments. However, they can be used for testing or experimentation, and show the kinds of functionality that can be expected in future FreeBSD releases.
Libelf is a BSD-licensed library for ELF parsing & manipulation implementing the SysV/SVR4 (g)ELF[3] API.
Current status: The library is now in -CURRENT. Work continues on its test suite and tutorial, and on deploying it in PmcTools.
The FreeBSD Dutch Documentation Project is an ongoing project to translate the FreeBSD Handbook to the Dutch Language.
Currently we almost translated the entire handbook, and we translated parts of the website, sadly the project went into a slush lately, so we seek out for fresh and new translators that are willing to join the team to continue the effort.
Where have we been?! Not doing status reports, that's for sure. But the FreeBSD GNOME project has been very busy with regular GNOME releases, and other side projects. We are currently shipping GNOME 2.16.2 in the ports tree, and we are testing GNOME 2.17.5 in the MarcusCom tree.
Most recently, work has completed on a cleanup of the FreeBSD backend to libgtop. This module has needed a lot of work, and should now be reporting correct system statistics. The cleaned up version is currently being tested in the MarcusCom tree, and will make it into the FreeBSD ports tree along with GNOME 2.18.
The GStreamer framework has been taken out of direct gnome@ maintainership, and put under a new multimedia@ umbrella. This will give multimedia-savvy developers a chance to collaborate on this important piece of the GNOME Desktop along with other important audio and video components.
The biggest accomplishment of 2006 for the FreeBSD GNOME team had to have been the port of HAL . This effort was started to give FreeBSD users a richer desktop experience. Since the initial FreeBSD release of HAL with GNOME 2.16, it has been incorporated into the FreeBSD release of KDE 3.5.5 as well as PC-BSD 1.3. The FreeBSD backend has also made it upstream into the HAL git repository so future releases of HAL will have FreeBSD support out-of-the-box.
Finally, it is with sadness that we say good-bye to one of our team members. Adam Weinberger stepped down from the FreeBSD GNOME team to save lives instead (priorities, man!). His splash screens and grammar nit-picking will be missed.
Support for in-kernel NAT, redirect and LSNAT for ipfw was committed to HEAD, and i encourage people to test it so we can quickly discover/fix bugs.
To add these features to ipfw, compile a new kernel adding "options IPFIREWALL_NAT" to your kernel config or, in case you use modules, add "CFLAGS += -DIPFIREWALL_NAT" to your make.conf.
Interrupt filtering is a new method to handle interrupts in FreeBSD that retains backward compatibility with the previous models (FAST and ITHREAD), while improving over them in some aspects. With interrupt filtering, the interrupt handler is divided into 2 parts: the filter (that checks if the actual interrupt belongs to a device) and a private per-handler ithread (that is scheduled in case some blocking work has to be done). The main benefits of this work are:
During the last quarter many improvements were made up to the point where 3 archs (i386, amd64 and arm) are reported to work, and the project can be considered feature complete.
I definitely want to make it part of the 7.0 release.
The FreeBSD Bugbusting team is a team of volunteers keeping track of various PR tickets in the GNATS application. Currently the Bugbusting team is investigating old PR tickets, checking whether they are still accurate, checking what needs to be done to fix the issues reported and make sure that the developers team can focus on the latest releases.
The team is always in need of volunteers willing to give a hand to resolve the old tickets and get the best feedback that is needed for the open tickets.
Please contact FreeBSD-bugbusters@FreeBSD.org if you want more information about the things that need to be done.
The FreeBSD Foundation ended 2006 raising over $100,000. We received commitments for another $55,000 in donations for the Fall Fundraiser. We fell short of our goal of raising $200,000. But, we are working hard to fill this gap, early in 2007, so we can continue with the same level of support for the project and community. Please go to http://www.freebsdfoundation.org/donate/ to find out how to make a donation to the foundation.
We added a donors page to our website to acknowledge our generous donors. We negotiated and are now actively managing a joint technology project with NLNet and the University of Zagreb to develop virtualized network stack support for FreeBSD. We sponsored AsiaBSDCon and are now accepting travel grant applications for this conference.
We are working to upgrade the project's network testbed with 10Gigabit interconnects. Cisco has generously donated a 10Gigabit switch and we have received network adapters from Myricom, Neterion, Intel, and Chelsio. Adapters from other vendors are being solicited so that we can do interoperability testing.
For more information on what we've been up to, check out our end-of-year newsletter at http://www.freebsdfoundation.org/press/2006Dec-newsletter.shtml .
The ports count has jumped to 16347. The PR count, despite a jump, has gone back down to around 700.
Not much work has been committed on the ports infrastructure due to the long 6.2 release cycle. However, many test runs have been done for several upcoming features, such as making sure that ports will work with the new release of gcc (4.1), and do not have /usr/X11R6 hard-coded into them. The intention of the latter is to move all ports to $LOCALBASE, which can then be selected by the user. This should help consistency going forwards, albeit at the cost of a one-time conversion.
GNOME was updated to 2.16 during the release cycle.
In addition, we are in the process of moving the FORTRAN default from f77 to gfortran. See the ports mailing list for details.
The new xorg ports are still being worked on as well; they are intended to all live in $LOCALBASE. Hopefully this can get done in the early 6.3 development cycle. See the wiki for more information.
A new version of the ports Tinderbox code is available, which is mostly a bugfix release.
We have also added Pav Lucistnik as a new portmgr member, who we hope will help us work on the portmgr PR backlog. Welcome!
We have also added 8 new committers since the last report.
linimon continues to work on resetting committers who are no longer interested in their ports; as well, several ports commit bits have been stored for safekeeping. This is part of an attempt to keep the best match between volunteers and work to be done.
In the time since the last status report, four security advisories have been issued concerning problems in the base system of FreeBSD (three in 2006 and one in 2007); of these, one problem was in "contributed" code, while the remaining three were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 55 new entries have been added, bringing the total up to 869.
In order to streamline security team operations and ensure that incoming emails are promptly acknowledged, Remko Lodder has been appointed the security team secretary.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.5, FreeBSD 6.0, FreeBSD 6.1, and FreeBSD 6.2. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 4.11 and FreeBSD 6.0 will cease to be supported at the end of January 2007.
Michael Richardson has been spearheading work to improve the crypto subsystem used by various parts of the kernel including Fast IPSec and geli. This work is sponsored by Hifn and has been happening outside the CVS repository. A main focus of this work is to add support for higher-level hardware operations that can significantly improve the performance of IPSec and SSL protocols.
Results of this work are now being readied for CVS. These redesign the core/driver APIs to use the kobj facilities and recast software crypto drivers as pseudo devices. The changes greatly improve the system and permit new functionality such as specifying which crypto device to use when multiple are available. The redesign will also enable load balancing of crypto work across multiple devices and the addition of virtual crypto sessions by which small operations can be done in software when the overhead to set up a hardware device is too costly.
In addition to the changes to the core crypto system several crypto drivers have been updated to improve their operation. Top of this list is the hifn(4) driver where many longstanding bugs have been fixed for 7955/756 parts.
FreeBSD is running multi-user on a variety of Gateworks Avila boards with most of the on-board devices supported. These include the compact flash/IDE slot, wired network interfaces, realtime clock, and environmental sensors. Several different minipci cards have been tested including those supported by the ath(4) and hifn(4) drivers. Remaining devices that need support are the onboard flash, optional 4-port network switch, and optional USB interface. Crypto acceleration for IXP425 parts is planned but will likely be done at a later time.
The Network Processor Engine (NPE) support is done with an entirely new replacement for the Intel Access Layer (IAL). The most important hardware facilities are supported (e.g. the hardware Q manager) and the wired NIC driver was also done from scratch. The resulting code is approximately 1/10th the number of lines of the equivalent IAL code.
The ZFS file system works quite well on FreeBSD now. The first patchset has already been published on the freebsd-fs@FreeBSD.org mailing list .
All file system methods are already implemented (except ACL-related). Basically all stress tests I tried work, even under very high load. There is still a problem with memory allocation, which can get out of control, but from what I know the SUN guys also work on this.
Recently I have been working on a file system regression test suite. From what I found, there are no such test suites for free. I've already more than 3000 tests and I'm testing correctness of most file system related syscalls (chflags, chmod, chown, link, mkdir, mkfifo, open, rename, rmdir, symlink, truncate, unlink). I'm also working to make it usable on other operating systems (like Solaris, where it already works and Linux).
Few days ago I also (almost) finished NFS support. You can't use the 'zfs share' command yet, but you can export file systems via /etc/exports and you can also access snapshots. It was quite hard, because snapshots are separate file systems and after exporting the main file system, we need to also serve data from snapshots under it.
The one big thing which is missing is ACL support. This is not an easy task, because we first have to make some decisions. Currently we use POSIX ACLs in our UFS, but the market is moving slowly to NTFS/NFSv4-type ACLs. In Solaris they use POSIX ACLs for UFS and NFSv4-type ACLs for ZFS and we probably also want to use NFSv4-type ACLs in our ZFS, which requires some work outside ZFS.
TrustedBSD priv(9) replaces suser(9) as an in-kernel interface for checking privilege in FreeBSD 7.x. Each privilege check now takes a specific named privilege. This allows both centralization of jail logic relating to privilege, which is currently distributed around the kernel at the point of each call to suser(9), and allows instrumentation of the privilege logic by the MAC Framework. Two new MAC Framework entry points, one to grant and the other to limit privilege, are now available, providing fine-grained control of kernel privilege by policy modules. This lays the kernel infrastructure groundwork for further refinement and extension of the kernel privilege model. The priv(9) implementation has been committed to FreeBSD 7-CURRENT.
This software was developed by Robert N. M. Watson for the TrustedBSD Project under contract to nCircle Network Security, Inc.
Most work on the MAC Framework during this period, other than as relates to the priv(9) project described in a separate status report, has been in refinement of the structure of the framework.
FreeBSD 6.2-RELEASE, the first release of FreeBSD with experimental audit support is now available. The plan is to make audit a full production feature as of FreeBSD 6.3-RELEASE, with "options AUDIT" compiled in by default. A TODO list has been posted to trustedbsd-audit.
OpenBSM 1.0 alpha 13, which includes support for XML record printing, additional 64-bit token types, additional audit events, and more cross-platform build support, has been released. OpenBSM 1.0 alpha 14, which adds support for warnings clean building with gcc 4.1, will be released shortly. The new OpenBSM release will be merged to FreeBSD CVS in late January or early February.
Just this week I got routing working for the FAST_IPSEC and IPv6 code. Now there are memory smash problems, and then we need to remove the old GIANT lock. I hope to produce another patch with the routing code working in the next week.
Normally the socket buffers are static (either derived from global defaults or set with setsockopt) and do not adapt to real network conditions. Two things happen: a) your socket buffers are too small and you can't reach the full potential of the network between both hosts; b) your socket buffers are too big and you waste a lot of kernel memory for data just sitting around.
With automatic TCP send and receive socket buffers we can start with a small buffer and quickly grow it in parallel with the TCP congestion window to match real network conditions.
FreeBSD has a default 32K send socket buffer. This supports a maximal transfer rate of only slightly more than 2Mbit/s on a 100ms RTT trans-continental link. Or at 200ms just above 1Mbit/s. With TCP send buffer auto scaling and the default values below it supports 20Mbit/s at 100ms and 10Mbit/s at 200ms. That's an improvement of factor 10, or 1000%. For the receive side it looks slightly better with a default of 64K buffer size.
The automatic send buffer sizing patch is currently running on one half of the FTP.FreeBSD.ORG cluster w/o any problems so far. Against this machine with the automatic receive buffer sizing patch I can download at 5.7 MBytes per second. Without patch it maxed out at 1.6 MBytes per second as the delay bandwidth product became equal to the static socket buffer size without hitting the limits of the physical link between the machines. My test machine is about 35ms from that FTP.FreeBSD.ORG and connected through a moderately loaded 100Mbit Internet link.
New sysctls are:
Work on wireless support has continued to evolve in the public CVS tree while other work has been going on behind the scenes in the developer's perforce repository.
Support was recently added to HEAD for half- and quarter-rate channels as found in the 4.9 GHz FCC Public Safety Band. This work was a prerequisite to adding similar support in the 900 MHz band as found in Ubiquiti's SR9 cards. Adding this functionality was straightforward due to the design of the net80211 layer, requiring only some additions to handle the unusual mapping between frequencies and IEEE channel numbers. The ath(4) driver currently supports hardware capable of operating on half- and quarter-rate channels.
Kip Macy recently made significant advances preparing legacy drivers for the re-architected net80211 layer that has been languishing in perforce. With his efforts this code is nearly ready for public testing after which it can be merged into CVS. Our goal is to complete this merge in time for the 7.x branch (otherwise it will be forced to wait for 8.0 before it appears in a public release). This revised net80211 layer includes advanced station mode facilities such as background scanning and roaming and support for Atheros' SuperG extensions. Getting the revised scanning work into CVS will greatly simplify public distribution of the Virtual AP (VAP) code as a patch as well as enable addition of 802.11n support.
Benjamin Close is working on support for the Intel 3945 parts commonly found in laptops. The work is going on in the perforce repository with public code drops for testing.
Atheros PCI/Cardbus support was updated with a new HAL that fixes a few minor issues and corrects a problem that kept AR2424 parts from working. The new HAL also enables more efficient use of the hardware keycache for TKIP keys; on newer hardware you can now support up to 57 stations without faulting keys into the cache. Support for the latest 802.11n parts found in the new Lenovo and Apple laptops (among others) is in development; initial release will support only legacy operation.
Support for Atheros USB devices is coming. Atheros has agreed to license their firmware with the same license applied to the HAL which means it can be committed to the tree and distributed as part of releases. The driver is still in development.
wpa_supplicant and hostapd were updated to the latest stable build releases from Jouni Malinen. Shortly the in-tree code base will switch to the 0.5.x tree which will bring in much new functionality including dynamic VLAN tagging that will be especially useful once the multi-bss support is available.
The support for injection of raw 802.11 frames was committed to HEAD. This work was done in collaboration with Andrea Bittau. At this point there are no plans to commit this to the STABLE branch as it requires API changes.
The sixth EuroBSDCon will take place in Copenhagen, Denmark on Friday the 14th and Saturday 15th of September 2007 . The conference will be held at Symbion Science Park . Sunday the 16th there will be an optional tour to LEGOland.
The call for papers was sent out right after EuroBSDCon 2006 in Milan in November and abstracts are due February 1st! So hurry up and send in all your fantastic and amazing papers to papers at eurobsdcon dot dk.
This report covers FreeBSD related projects between January and March 2007. This quarter ended with a big bang as a port of Sun's critically acclaimed ZFS was added to the tree and thus will be available in the upcoming FreeBSD 7.0 release. Earlier this year exciting benchmark results showed the fruits of our SMP work. Read more on the details in the "SMP Scalability" report.
During the summer, FreeBSD will once again take part in Google's Summer of Code initiative. Student selection is underway and we are looking forward to a couple of exciting projects to come.
BSDCan is approaching rapidly, and will be held May 16-19th in Ottawa.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
The Schedule and the Tutorials have been released. Once again, we have a very strong collection of Speakers .
BSDCan: Low Cost. High Value. Something for Everyone.
Everyone is going to be there. Make your plans now.
We have added Remko Lodder to the bugmeister team. Remko has been doing a great deal of work to go through antique PRs, especially in the i386 category, and it was time to recognize that hard work. As a result of his work the i386 count is at a multi-year low.
Remko has also been instrumental in working with some new volunteers who are interested in finding out how they can contribute. Our current plans are to ask them to look through the PR backlog and, firstly, ask for feedback from the submitters, and secondly, identify PRs that need action by committers. We also have some committers who have volunteered to review those PRs. If you are interested in helping, please subscribe to bugbusters@FreeBSD.org. Our thanks to our current helpers, including Harrison Grundy.
The overall PR count has dropped to around 5100, a significant reduction.
The sixth EuroBSDCon will take place at Symbion in Copenhagen, Denmark on Friday the 14th and Saturday 15th of September 2007.
The estimated price for the two day conference is 200EUR, excluding Legoland trip and social event. The whole-day trip to Legoland is expected to cost around 130EUR including transportation, some food on the way, and entry fee. Arrangements have been made with a newly renovated Hostel which offers beds for 23EUR per night and 10EUR breakfast. A lounge with sponsored Internet connection will be available at the Hostel. Staying at the hostel is of course entirely optional and several Hotels exists in the area. Reservation for the conference and exact prices are expected to be ready no later than 1st of May.
As of this writing 10 presentations have been accepted and more are in the process of being evaluated.
For FreeBSD Developers, a by invitation Developers summit will be held in connection with the conference. Exactly when this will take place has not yet been decided.
We are still looking for more sponsors.
A public IRC channel #eurobsdcon on EFnet has been created for discussion and questions about the conference.
More details will follow on the EuroBSDCon 2007 web site as they become available.
There are currently two p4 branches being used for this work: gnn_fast_ipsec: a dual stack branch which contains both Kame and FAST_IPSEC with v6 enabled. gnn_radical_ipsec: a single stack branch, still in progress, where Kame IPsec has been removed and only FAST remains.
The FreeBSD Foundation ended Q1 raising over $65,000. We're a quarter of the way to our goal of raising $250,000 this year. We continued our mission of supporting developer communication by helping FreeBSD developers attend AsiaBSDCon. We are a sponsor of BSDCan and are currently accepting travel grant applications for this conference.
The foundation provided support that helped the ZFS file system development. We continued working to upgrade the project's network testbed with 10Gigabit interconnects. We attended SCALE where we received an offer from No Starch Press to include a foundation ad in their BSD books. Our first ad will appear in the book "Designing BSD Rootkits."
For more information on what we've been up to, check out our website at http://www.freebsdfoundation.org .
A version of GCC 4.1 is being prepared for inclusion into FreeBSD 7.0-CURRENT. Work was started late in 2006 but progress on certain technical points (e.g. correctly integrating and bootstrapping a shared libgcc_s into the build) was slow due to lack of developer time. The remaining outstanding issue is that compiling with -O2 is shown to lead to runtime failures of certain binaries (e.g. some port builds); it is not currently known whether these are due to application errors or GCC miscompilations. It is believed that the current snapshot is otherwise ready for inclusion, and this will likely happen within a week or two.
The above URL documents some work done around January to build an emulation layer for the Linux kernel API that would allow Linux device driver to be built on FreeBSD with as little as possible modifications. Initially the project focused on USB webcams, a category of devices for which there was basically no support so far. The emulation layer, available as a port ( devel/linux-kmod-compat ) simulates enough of the Linux USB stack to let us build, from unmodified Linux sources, two webcam drivers, also available as ports ( multimedia/linux-gspca-kmod and multimedia/linux-ov511-kmod ), with the former supporting over 200 different cameras.
While some of the functions map one-to-one, for others it was necessary to build a full emulation (e.g. collecting input from various function calls, and then mapping sets of Linux data structures into functionally equivalent sets of FreeBSD data structures). But overall, this project shows that the software interfaces are reasonably orthogonal to each other so one does not need to implement the full Linux kernel API to get something working. More work is necessary to cover other aspects of the Linux kernel API, e.g. memory mapping, PCI bus access, and the network stack API, so we can extend support to other families of peripherals.
Since the last status report AMD64 was feature synced with i386. Notably TLS and futexes are now available on AMD64. Many thanks to Jung-Uk Kim for doing the TLS work.
Currently the focus is to implement the *at() family of linux syscalls and to find and fix the remaining futex problems.
We need some more testers and bug reporters. So if you have a little bit of time and a favorite linux application, please play around with it on -CURRENT. If there is a problem, have a look at the Wiki if we already know about it and report on emulation@. We are specially interested in reports about the 2.6 compatibility (sysctl compat.linux.osversion=2.6.16), but only with the most recent -current and maybe with some patches we have in the perforce repository (available from the wiki).
We would like to thank all the people which tested the changes / submitted patches and thus helped improve the linux compatibility environment.
malloc(3) has recently been enhanced to reduce memory overhead, fragmentation, and mapped memory retention. As an added bonus, it tends to be a bit faster. See the above URL for my email to the -current mailing list for a more detailed description of the enhancements.
Stable release 4.1 of mpd4 branch was released in February providing many new features and fixes. Mpd3 branch was declared legacy.
Since the release several new features have been implemented in CVS:
ng_car node has been updated, to support shaping and very fast Cisco-like rate-limiting. ng_ppp node has been completely re-factored to confirm to the protocol stack model.
The ports count is nearing 17,000. The PR count has been stable at around 700. The 'new port' PR backlog is at a multi-year low. We appreciate all the hard work of our ports committers.
Since the long 6.2 release cycle ended, portmgr has once again been able to do experimental ports runs. As a result of six run/commit cycles, the portmgr PR count is now the lowest in quite some time. Please see the CHANGES and UPDATING files for details. Many thanks to Pav among others for keeping the build cluster busy.
We have received new hardware, resulting in a significant speedup of our package building capability: the AMD64 package builds now use 4 8-core machines (and one lonely UP system), which means a full AMD64 build is about 5 times faster than it was. Also, the i386 cluster gained an 8-core and roughly doubled its performance too. Two of the sparc64 build machines have recently brought back online, so package builds there have been restarted there after a long period offline.
linimon continues to work on improvements to portsmon to allow graphing of the dependent ports of ignored/failed ports. This work will be presented at BSDCan. In addition, pages that show the state of port uploads on ftp*.FreeBSD.org have been added, as well as ports that have NO_PACKAGE set. Also, the individual port overview page now shows the latest package that has been uploaded to the ftp servers for each buildenv.
A number of absent maintainers have been replaced by some new volunteers who had been sending PRs to update and/or fix their ports. Welcome! This helps to spread the workload.
Since the last report, support for FreeBSD 4.X has been dropped from the Ports Collection. Anyone still using RELENG_4 should have stayed with the ports infrastructure as of the RELEASE_4_EOL tag, as later commits remove that support. 4.X served us long and well but the burden of trying to support 4 major branches finally became too much to ask of our volunteers. Use of 4.X, even with the RELEASE_4_EOL tag, is no longer recommended; we recommend either 6.2-RELEASE or RELENG_6, depending on your needs.
There have been new releases of the ports tinderbox code, the portmaster update utility, and portupgrade. A new utility, pkgupgrade, has been introduced by Michel Talon, which appears interesting.
KDE was updated to 3.5.6.
GNOME was updated to 2.18.
XFree86 version 3 was removed as being years out of date.
We have added 3 new committers since the last report.
During the past quarter, the Release Engineering team has begun planning and preparing for FreeBSD 7.0, which is scheduled for release later in 2007. The HEAD codeline has been placed in a "slush" mode, meaning that large changes should be coordinated with the Release Engineering team before being committed.
The RE team also produced snapshots of FreeBSD 6.2-STABLE and 7.0-CURRENT for February and March 2007, corresponding roughly to the state of those development branches at the start of the respective months. While they have not had the benefit of extensive testing, and should not be used in production, they can be useful for experimenting with or testing new features.
In the time since the last status report, one security advisory has been issued concerning a problem in the base system of FreeBSD; this problem was in "contributed" code maintained outside of FreeBSD. In addition, several Errata Notices have been issued in collaboration with the release engineering team, including one concerning FreeBSD Update. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 21 new entries have been added, bringing the total up to 890.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 5.5, FreeBSD 6.1, and FreeBSD 6.2. Of particular note, FreeBSD 4.11 and FreeBSD 6.0 are no longer supported. The respective End of Life dates of supported releases are listed on the web site.
Over the past few months there has been a substantially increased focus on improving scalability of FreeBSD on large SMP hardware. This has been driven in part by the new availability of 8-core hardware to the project, which allows easy profiling of scalability bottlenecks and benchmarking of proposed changes. Significant progress has been made on certain application workloads such as MySQL and PostgreSQL, with the result that FreeBSD 7 now has excellent scaling to at least 8-CPU systems with prospects for further improvements. Progress with other application workloads has been limited by the need to set up a suitable test case; please contact me if you are interested in helping. As part of this general effort, work is progressing steadily on removing the last remaining Giant-locked code from the kernel. A complete list of remaining Giant-locked code is found here: http://wiki.freebsd.org/SMPTODO Many of these sub-tasks have owners, but some do not. The major remaining Giant-locked subsystem with no owner is the TTY subsystem. In parallel, profiling of contention and bottlenecks in other subsystems has lead to a number of experimental changes which are being developed. Work is in progress by Jeff Roberson and Attilio Rao to break up the global scheduler spinlock in favor of a set of per-CPU scheduling locks, which is expected to improve performance on systems with many CPUs. Experimental changes by Robert Watson to allow for multiple netisr threads show good promise for improving loopback IP performance on large SMP systems, which can otherwise easily saturate a single netisr thread. A variety of other changes are being profiled and evaluated to improve SMP performance under various workloads. The majority of these changes are collected in the //depot/user/kris/contention/ Perforce branch.
Work has completed to port over trunk(4) from OpenBSD and this also includes merging 802.3ad LACP from agr(4) in NetBSD. This driver allows aggregation of multiple network interfaces as one virtual interface using a number of different protocols/algorithms.
This will be committed shortly, further testing is welcome.
During the last three months not too much has changed. Here is a quick list of changes:
In my last status report I asked for access to Sparc64 boxes with FreeBSD installed. Testing is ongoing and some problems remain with EHCI PCI Cards. I am not exactly sure where the problem is, but it appears that DMA-able memory does not get synced properly.
Markus Brueffer is still working on the USB HID parser and support. Nothing has been committed yet.
Several people have reported success with my new USB stack. Some claim 2x improvements, others have seen more. But don't expect too much.
If you want to test the new USB stack, checkout the USB perforce tree or download the SVN version of the USB driver from my USB homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@freebsd.org .
Work is slowly continuing on this driver, focusing mainly on dealing with the newly released firmware for the card. The old firmware was not redistributable, the new firmware can be redistributed but has a completely different API. With the new firmware changes almost complete, the driver is approaching a state ready for -CURRENT.
X.Org 7.2 is now on final approach for landing into the ports tree. Work had proceeded at a slow pace for the first few months of the year due to reduced availability of flz@, the single developer working on integration. Recently lesi@ was recruited back into the task and readiness of the ports collection was pushed to completion (i.e. there are no major regressions apparent on package builds). The remaining tasks which need to be completed are a review of the diff to make sure no unintentional changes or regressions slip in to the CVS tree in the big merge, and completion of an upgrade script to manage the migration from X.Org 6.9 (X.Org 7.2 is so fundamentally different that it cannot be upgraded "automatically" using the existing tools like portupgrade). We hope to have these finished within a week or two, at which stage the ports collection will be frozen for the integration, and we will likely remain in a ``mini-freeze'' for a week or two in order to focus committer attention on resolving the inevitable undetected problems which will emerge from this major change.
The ZFS file system in now part of the FreeBSD operating system. ZFS was ported from the OpenSolaris operating system and is under CDDL license. As an experimental feature ZFS will be available in FreeBSD 7.0-RELEASE.
