Index: head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml
===================================================================
--- head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml (revision 48046)
+++ head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml (revision 48047)
@@ -1,3290 +1,3340 @@
This is a draft of the October–December 2015
status report. Please check back after it is finalized, and
an announcement email is sent to the &os;-Announce mailing
list.
The fourth quarter of 2015 was another productive quarter for the &os; project and community. [...]
Thanks to all the reporters for the excellent work!
The deadline for submissions covering the period from January to March 2016 is April 7, 2016.
?>LKL ("Linux Kernel as a Library") is a special "architecture" of the full Linux kernel that builds as a userspace library on various platforms, including &os;. One application of such a library is using Linux's filesystem drivers to implement a FUSE backend.
fusefs-lkl's lklfuse binary is such a FUSE filesystem. It can mount ext4/3/2, XFS, and BTRFS read-write, using the native drivers from Linux.
The sysutils/fusefs-lkl port may now be installed from packages or ports, providing access to these filesystems on &os; via FUSE.
Use of bool is now allowed. It was allowed previously, as well, but now it's really allowed. Party like it's 1999!
Specify style(9)'s opinion on iso646.h.
Fix intmax_t to be 128-bit on platforms where __int128_t is used.
This quarter, support was added for fixed-width sysctls (signed and unsigned 8-bit, 16-bit, 32-bit, and 64-bit integers). The new KPIs are documented in the sysctl(9) manual page. The sysctl(8) command line tool supports all of the new types.
sysctl(8) gained the '-t' flag, which prints sysctl type information (the original patch was submitted by Yoshihiro Ota). This support includes the newly added fixed-width types.
I/OAT DMA engines are bulk memory operation offload engines built into some Intel Server/Storage platform CPUs.
This quarter, several enhancements were made to the driver. The driver now avoids memory allocation in locked paths, which should avoid deadlocking in memory pressure scenarios. Support for Broadwell-EP devices has been added. The "blockfill" operation and a non-contiguous 8 KB copy operation have been added to the API. The driver can recover from various programming errors by resetting the hardware.
XOR and other advanced ("RAID") operation support.
ntb_hw(4) is now up-to-date with the Linux NTB driver as of the work-in-progress 4.4 kernel (and actually, contains some fixes that haven't landed in the mainline Linux tree yet but will land in 4.5). Only Back-to-back ("B2B") configurations are supported at this time. Going forward, newer hardware may only support the B2B configuration.
if_ntb(4) is mostly up-to-date with the Linux NTB netdevice driver. Notably absent is support for changing the MTU at runtime.
Improving if_ntb(4) to avoid using the entire Base Address Register (BAR) when very large BAR sizes are configured (e.g., 512 GB).
Improving pmap_mapdev(9) to somehow allocate only superpage mappings for large BARs, on platforms that support superpages. (NTB BARs can be as large as 512 GB.)
This quarter, we made the changes necessary to support the Amlogic Meson Ethernet controller on the Hardkernel ODROID-C1 board which has an Amlogic aml8726-m8b SoC. The main effort needed was to write a glue driver for the Ethernet controller — the Amlogic Meson Ethernet controller is compatible with Synopsys DesignWare 10/100/1000 Ethernet MAC (if_dwc).
The Mellanox &os; team is proud to announce support for the ConnectX-4 series of network cards in &os; 11-current and &os; 10-stable. These devices deliver top performance, with up to 100GBit/s of raw transfer capacity, and support both Ethernet and Infiniband. Currently, the Ethernet driver is ready for use and the Infiniband support for ConnectX-4 is making good progress. We hope that it will be complete before &os; 11.0 is released. For more technical information, refer to the mlx5en(4) manual page in 11-current. The new driver for ConnectX-4 cards is called mlx5 and is put under /sys/dev and not under /sys/ofed as was done for the previous mlx4 driver. The mlx5en(4) kernel module is compiled by default in GENERIC kernels.
FreeBSD Mastery: Specialty Filesystems is now in copyediting. The ebook should be available by the end of January at all major vendors, and the print in February.
The book covers everything from removable media, to FUSE, NFSv4 ACLs, iSCSI, CIFS, and more.
If you act really quickly, you can get the electronic early access version at a 10% discount. You'll get the final ebook when it comes out as well. (This offer evaporates when the final version comes out.)
The KGDB option is now on by default in the devel/gdb port.
The changes to support cross-debugging of crashdumps in libkvm were committed to HEAD in r291406.
A new thread target for &os; that is suitable for merging upstream has been written and lightly tested. However, it is not yet available as an option in the port. This thread target uses ptrace(2) directly rather than libthread_db and as such supports threads on all ABIs (such as &os;/i386 binaries on &os;/amd64 and possibly Linux binaries, though that is not yet tested). It also requires less-invasive changes in the MD targets in GDB compared to the libthread_db-based target.
Add a port option for the new 1:1-only thread target.
Test the new 1:1-only thread target.
Figure out why the powerpc kgdb targets are not able to unwind the stack past the initial frame.
Add support for more platforms (arm, mips, aarch64) to upstream gdb for both userland and kgdb.
Add support for debugging powerpc vector registers.
iMX.6 is a family of SoC used in multiple hobbyist ARM boards such as the Hummingboard, RIoTboard, and Cubox. Most of these products have HDMI output, but until recently, &os; did not benefit from it. As of r292574, there is basic video output support so you can use the console on iMX6-based boards and probably run Xorg (not yet tested).
Due to the lack of some kernel functionality (see open tasks), the only supported mode is 1024x768.
Proper pixel clock initialization (relies on a clock framework).
More flexible video output path (support multiple IPUs and DIs).
There are two working proof-of-concept drivers for the AM335x touchscreen and for the official Raspberry Pi's touchscreen LCD.
Proper touchscreen support would consist of a userland event reading API, a kernel event reporting API, and kernel hardware drivers for specific devices. There is an ongoing effort to port the Linux evdev API to &os; so applications that use libraries like libinput or tslib could be used without any major changes. Since it is not yet complete, I created a naive evdev-like API for both kernel and tslib and was able to run a demo on a Beaglebone Black with 4DCAPE-43T.
Once evdev makes it into the tree, both hardware drivers can be modified to include "report events" portions and committed.
This completed project includes changes to better manage the vnode freelist and to streamline the allocation and freeing of vnodes.
Vnode cache recycling was reworked to meet free and unused vnodes targets. Free vnodes are rarely completely free; rather, they are just ones that are cheap to recycle. Usually they are for files which have been stat'd but not read; these usually have inode and namecache data attached to them. The free vnode target is the preferred minimum size of a sub-cache consisting mostly of such files. The system balances the size of this sub-cache with its complement to try to prevent either from thrashing while the other is relatively inactive. The targets express a preference for the best balance.
"Above" this target there are 2 further targets (watermarks) related to the recyling of free vnodes. In the best-operating case, the cache is exactly full, the free list has size between vlowat and vhiwat above the free target, and recycling from the free list and normal use maintains this state. Sometimes the free list is below vlowat or even empty, but this state is even better for immediate use, provided the cache is not full. Otherwise, vnlru_proc() runs to reclaim enough vnodes (usually non-free ones) to reach one of these states. The watermarks are currently hard-coded as 4% and 9% of the available space. These, and the default of 25% for wantfreevnodes, are too large if the memory size is large. E.g., 9% of 75% of MAXVNODES is more than 566000 vnodes to reclaim whenever vnlru_proc() becomes active.
The vfs.vlru_alloc_cache_src sysctl is removed. New code frees namecache sources as the last chance to satisfy the highest watermark, instead of selecting source vnodes randomly. This provides good enough behaviour to keep vn_fullpath() working in most situations. Filesystem layouts with deep trees, where the removed knob was required, is thus handled automatically.
As the kernel allocates and frees vnodes, it fully initializes them on every allocation and fully releases them on every free. These are not trivial costs: it starts by zeroing a large structure, then initializes a mutex, a lock manager lock, an rw lock, four lists, and six pointers. Looking at vfs.vnodes_created, these operations are being done millions of times an hour on a busy machine.
As a performance optimization, this code update uses the uma_init and uma_fini routines to do these initializations and cleanups only as the vnodes enter and leave the vnode zone. With this change, the initializations are done kern.maxvnodes times at system startup, and then only rarely again. The frees are done only if the vnode zone shrinks, which never happens in practice. For those curious about the avoided work, look at the vnode_init() and vnode_fini() functions in sys/kern/vfs_subr.c to see the code that has been removed from the main vnode allocation/free path.
The QLogic HBA driver isp(4) received a substantial set of changes. Their primary goal was to make Fibre Channel target role work well with CTL, but many other things were also fixed/improved:
The code is committed to &os; head and stable/10 branches.
NVRAM data reading is hackish and requires rework.
FCoE support for 26xx cards was not tested yet.
The Raspberry Pi SoC consists of two parts: ARM and GPU (VideoCore). Many interesting features like OpenGL, video playback, and HDMI controls are implemented on the VideoCore side and can be accessed from the OS through libraries provided by Broadcom (userland repo). These libraries were ported to &os; some time ago, so Mikaël created the port misc/raspberrypi-userland for them. He also created a port for omxplayer (a low-level video player that utilizes VideoCore APIs) and is working on a port for Kodi (ex-XBMC), a more user-firendly media player software with Raspberry Pi support.
LXQt is the Qt port of and the upcoming version of LXDE, the Lightweight Desktop Environment. It is the product of the merge between the LXDE-Qt and the Razor-qt projects.
The porting effort remains heavily a work in progress: it needs some components of Plasma 5 (the new major KDE's workspace).
Currently, only the 0.10 branch is functional. See our wiki page for a complete list of applications.
We also sent updates for some components of LXDE (required for the LXQt desktop):
Binary packages are available (only for test purposes) which are regularly tested with the KDE development repository.
Port libsysstat to BSD systems.
Fix some issues that need to be resolved, especially the shutdown and reboot commands.
Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient — perfect for data-intensive real-time applications that run across distributed devices.
The goal of this project is to make it easy to install the modules available in the npm package registry.
Currently, the repository contains slightly fewer than 300 new ports, in particular:
We have improved the USES framework:
Each port is up-to-date.
Update the pre-draft documentation.
Bring in grunt.js (and modules), the JavaScript task runner.
Xfce is a free software desktop environment for Unix and Unix-like platforms, such as &os;. It aims to be fast and lightweight, while still being visually appealing and easy to use.
During this quarter, the team has kept these applications up-to-date:
We also follow the unstable releases (available in our experimental repository) of:
Propose a patch to upstreamto fix Xfdashboard with our version of OpenGL (it currently coredumps).
I recently became involved with &os; (as in, the last 2-3 weeks), and found myself quickly involved with Ports development. What quickly struck me was the difficulty in providing a Python package that was depended upon by multiple versions of Python. As it turns out, poudriere can currently only generate one package per port, meaning that a Python version-neutral (compatible with 2.x and 3.x) port cannot simultaneously be packaged for each variant at the same time.
I discussed the issue with &a.koobs;, who suggested that I look into implementing a "variants protocol" within the Ports framework and the necessary changes to poudriere in order to allow a port to generate more than one package.
Support for variants is strongly needed in Ports and provides significant benefits.
For a simple example, editors/vim-lite could be folded into the editors/vim port, while still generating a vim and vim-lite package. For Python, VARIANTS can be derived from the already used USES flags and generate compatible packages. py27-foobar and py34-foobar could now be consistently generated by poudriere without issue.
Fortunately, this is not a wishful thinking piece. I dug in my heels and have implemented a proof-of-concept implementation of variants in the Ports framework, including the necessary modifications to poudriere in order to support it. It was mildly upsetting to find that poudriere is mostly written in Bourne shell scripts, but press on I did nonetheless.
I started with the prototype made by &a.bapt; as a base, and built from there. The poudriere PoC aims to limit changes as much as possible to merely adding support for the new variants flags, while also at the request of &a.koobs; making the logging output more package-centric (as opposed to port-centric) as a result of these changes.
This is a work in progress, and I would love to hear your feedback. I've enjoyed my first few weeks working on &os;, and I hope to stay here for quite some time.
Any constructive feedback on the implementation would be very welcome!
Hopefully the code will be of sufficient quality to be considered for formal review in the coming months.
When I starting working on ports for &os; in the last couple of weeks, I found that my workflow was not as efficient as it could be, using just the available tools, so I made a few that could be useful to the development community at large. All of these have been added to the Ports tree, or otherwise will soon be added, so you can play with them today!
pytoport is a command-line application that generates a skeleton port for a given PyPI package name. It attempts to generate the correct dependencies, makes a good attempt at guessing the license using spdx-lookup, and generates a pkg-descr. This made generating the fifteen or so ports I was working on a complete breeze.
While doing this, however, I noticed that some ports were bringing in dependencies that I did not expect, and I needed some way to visualise this. skog builds a dependency tree from the depends lists output by the Ports framework, and displays it on the command line (with extra shiny output if you are using UTF-8). No more pesky example and documentation dependencies being dragged in when you clearly toggled that OPTION as far off as it would go.
While doing all of this, I found it cumbersome to be copying ports back and forth between my small development tree living in git and the larger upstream SVN tree I was using in poudriere. I built a tool called bandar that takes advantage of the FUSE version of unionfs to easily overlay my dev tree on the upstream tree, run linting, poudriere and generate archives with ease.
I'm very impressed with how easy it was to build more tooling for &os;. I hope some of these tools will be of some use to you, and as always, I'd love to hear your feedback!
Improve skog to support searching a tree for a certain port.
Get the bandar port completed.
Continue to improve pytoport, adding trove support and better depedency handling.
Patches welcome for all of the above!
The Out of Memory (OOM) code is intended to handle the situation where the system needs free memory to make progress, while no memory can be reused. Most often, the situation is that to free memory, the system needs more free memory. Consider a case where the system needs to page-out dirty pages, but needs to allocate structures to track the writes. OOM "solves" the problem by killing some selection of user processes. In other words, it trades away system deadlock by suffering a partial loss of user data. The assumption is that it is better to kill a process and recover data in other processes, than lose everything.
Free memory in the &os; Virtual Memory (VM) system appears from two sources. One is the voluntary reclamation of pages used by a process, for example unmapping private anonymous regions, or the last unlink of an otherwise unreferenced file with cached pages. Another source is the pagedaemon, which forcefully frees pages which carry data, of course, after the data is moved to some other storage, like swap or file blocks. OOM is triggered when the pagedaemon definitely cannot free memory to satisfy the requests.
The old criteria to trigger OOM action was a combination of low free swap space and a low count of free pages (the later is expressed precisely with the paging targets constants, but this is not relevant to the discussion). That test is mostly incorrect, e.g., a low free page state might be caused by a greedy consumer allocating all pages freed by the page daemon in the current pass, but this does not preclude the page daemon from producing more pages. Also, since page-outs are asynchronous, the previous page daemon pass might not immmediately produce any free pages, but they would appear some short time later.
More seriously, low swap space does not necessarily indicate that we are in trouble: lots of pages may not require swap allocations to freed, e.g., clean pages or pages backed by files. The last notion is serious, since swap-less systems were considered as having full swap.
Instead of trying to deduce the deadlock from looking at the current VM state, the new OOM handler tracks the history of page daemon passes. Only if several consequtive passes failed to meet the paging target is an OOM kill considered neccessary. The count of consequent failed passes was selected empirically, by testing on small (32M) and large (512G) machines. Auto-tuning of the counter is possible, but requires some more architectural changes to the I/O subsystem.
Another issue was identified with the algorithm which selects a victim process for OOM kill. It compared the counts of pages mapping entries (PTEs) installed into the machine paging structures. For different reasons, machine-dependent VM code (pmap) may remove the pte for a memory-resident page. Under some circumstances, related to other measures to prevent low memory deadlock, very large processes which consume all system memory, could have few or no ptes, and the old OOM selector ignored the process which caused the deadlock, killing unrelated processes.
A new function vm_pageout_oom_pagecount() was written which applies a reasonable heuristic to estimate the number of pages which would be freed by killing the given process. This eliminates the effect of selecting small unrelated processes for OOM kill.
The rewrite was committed to HEAD in r290917 and r290920.
A new driver, cxgbei, that enables hardware accelerated iSCSI with Chelsio's T5- and T4-based offload-capable cards, has been committed to HEAD. Both Initiator and Target are supported. The wire traffic is standard iSCSI (SCSI over TCP as per RFC 3720, etc.) so an Initiator/Target using this driver will interoperate with all other standards-compliant implementations.
Hardware assistance provided by the T5 and T4 ASICs includes:
The cxgbei(4) man page is missing but will be committed shortly.
The driver is in advanced stage QA and will see some bugfixes and performance enhancements in the very near future. MFC is possible as soon as the QA cycle completes.
OpenBSM is a BSD-licensed implementation of Sun's Basic Security Module (BSM) API and file format. It is the user-space side of the CAPP Audit implementations in &os; and Mac OS X. Additionally, the audit trail processing tools are expected to work on Linux.
Progress has been slow but steady this quarter, culminating in OpenBSM 1.2 alpha 4, the first release in three years. It features various bug fixes and documentation improvements; the complete list of changes is documented in the NEWS file on GitHub. The release was imported into &os; HEAD and merged to &os; 10-STABLE. As such it will be part of &os; 10.3-RELEASE.
Test the new release on different versions of &os;, Mac OS X and Linux. In particular, testing on Mac OS X 10.9 (Mavericks) and newer would be greatly appreciated.
Fix problems that have been reported via GitHub and the &os; bug tracker.
Implement features mentioned in the TODO list on GitHub.
&os; has been ported to run on the Marvell Armada38x platform. This SoC family boasts single/dual high-performance ARM Cortex-A9 CPUs.
The multi-user SMP system is fully working and has been tested on Marvell DB-88F6288-GP and SolidRun ClearFog development boards.
The root filesystem can be hosted on a USB 3.0/2.0 drive or via NFS using a PCIe network card. Experimental support is available for on-chip Gigabit Ethernet (NETA).
Additional features:
The port is under community review and will be integrated into HEAD soon.
Optimize performance of NETA and prepare for submission.
GitLab is a web-based Git repository manager with many features that is used by more than 100.000 organizations including NASA and Alibaba. It also is a very long-standing entry on the "Wanted Ports" list of the &os; Wiki.
In the last quarter, there was steady progress in the project itself and the porting. The current release of GitLab 8.3 is now based on Rails 4.2, which obsoletes the need for around 50 new ports. Now there are only 5 dependencies left to be committed!
While the new version of GitLab 8.3 eases the porting, there are big changes between the last working port of GitLab 7.14. Nonetheless, it could be expected to see the next working port in the first quarter of 2016
Update the patches from GitLab 7.14 to 8.3.
Update the documentation.
Provide an updated patch.
There are more and more machines on the internet that only support IPv6. I manage some of them, and was regularly hit by missing IPv6 support when building ports.
I did some research into the impact of missing IPv6 support on the ports tree. The results are that 10.308 of 25.522 ports are not fetchable when using IPv6. This renders — through dependencies — a total of 17.715 ports unbuildable from IPv6-only systems. All you can do than is wait and hope that distcache.FreeBSD.org caches the distfile. But this will take some time, which may not be a luxury available when a piece of software in use is hit by a security issue.
Based on the research, a promotion campaign for IPv6 was started. Some volunteers will contact the relevant system administrators and try to convince them to support IPv6. This will start in January 2016 and will hopefully create some progress soon.
The KDE on &os; team focuses on packaging and making sure that the experience of KDE and Qt on &os; is as good as possible.
The team kept busy during the last quarter of 2015. Quite a few big updates were committed to the ports tree, and a few more are being worked on in our experimental repository.
As in previous quarters, we would like to thank several people who have contributed with machines, patches and general help. Tobias Berner, &a.madpilot; (madpilot@), Adriaan de Groot, Ralf Nolden, &a.swills; (swills@), and &a.jpaetzel; (jpaetzel@) have been essential to our work.
The following big updates were landed in the ports tree this quarter. In many cases, we have also contributed patches to the upstream projects.
Work on updating the Qt5 ports to their latest version, as well as porting KDE Frameworks 5 and Plasma 5 to &os;, is well underway in our experimental area51 repository. At the moment, it contains Qt5 5.5.1, KDE Frameworks 5.17.0, Plasma 5.5.1 and KDE Applications 15.12.0.
Users interested in testing those ports are encouraged to follow the instructions in our website and report their results to our mailing list. Qt5 5.5.1 is in our "qt-5.5" branch, and Plasma 5 and the rest is in the "plasma5" branch (which also contains Qt 5.5.1).
Commit the Qt5 5.5.1 update.
Land the KDE Frameworks 5 and Plasma 5 ports to the tree.
Investigate what needs to be done to make QtWebEngine, the Chromium-based replacement for QtWebKit, work on &os;.
As of the end of Q4 the ports tree holds a bit more than 25,000 ports, and the PR count is around 2,000. The activity on the ports tree remains steady, with about 7,000 commits performed by almost 120 active committers.
On the problem reports front, figures show an encouraging trend, with a significant increase in the number of PRs fixed during Q4. Indeed, almost 1,800 reports were fixed, which makes an increase of about 20% compared to Q3.
In Q4 8 commit bits were taken in for safekeeping, following an inactivity period of more than 18 months (lioux, lippe, simon, jhay, max, sumikawa, alexey, sperber). Three new developers were granted a ports commit bit (Kenji Takefu, Carlos Puga Medina, and Ian Lepore), and one returning committer (miwi) had his commit bit reinstated.
Also related to the management of ports commit bits, nox's grants were revoked, since the &os; developers learnt that Juergen Lock passed away.
On the management side, no changes were made to the portmgr team during Q4.
On QA side 33 exp-runs were performed to validate sensitive updates or cleanups. Amongst those noticeable changes are the update to GCC 4.9, CMake to 3.4.1, PostgreSQL to 9.4, and ruby-gems to 2.5.0. Some infrastructure changes included the usage of a WRKSRC different from WRKDIR when NO_WRKSUBDIR is set, the removal of bsd.cpu.mk from sys.mk, and the move of QT_NONSTANDARD to bsd.qt.mk.
We would like to remind everyone that the ports tree is built and run by volunteers, and any help is greatly appreciated. While Q4 saw a significant increase in the number of problem reports fixed, we encourage all ports committers to have a look at the issues reported by our users and try to fix as many as possible. Many thanks to all who made a contribution during Q4, and keep up the good work in 2016!
This quarter, the bugmeister team has gained a new member, Mahdi Mokhtari (mokhi64@gmail.com). Mahdi has been contributing to the &os; Project for just over one month. After getting started by creating ports for Chef-Server and MySQL 5.7 (With Bernard Spil's help), an introduction to &a.koobs; led to guidance on appropriate projects, such as Bugzilla development, helping Bugmeister, the Bugzilla Triage team, Developers, and the Community by making issue tracking better. This is how things are going so far:
Issue Tracking can be either "Defect Tracking for Systems" or "Bug-Tracking for Systems". System Defect Tracking is to allow individual or groups of developers to keep track of outstanding issues in their product effectively. We use Bugzilla to manage issues for the &os; project.
We are pleased to announce some developments on our issue management systems:
Major improvements to templates for usability and simplicity.
Further improvements to automation (for example, additional processing of commit logs).
One of the long-missing features of &os; was the ability to boot up with a temporary rootfs, configure the kernel to be able to access the real rootfs, and then replace the temporary root with the real one. In Linux, this functionality is known as pivot_root. The reroot projects provides similar functionality in a different, slightly more user-friendly way: rerooting. Simply put, from the user point of view it looks like the system performs a partial shutdown, killing all processes and unmounting the rootfs, and then partial bringup, mounting the new rootfs, running init, and running the startup scripts as usual.
The project is finished. All the relevant code has been committed to &os; 11-CURRENT, and is expected to ship with &os; 11.0.
An important missing piece of the RCTL resource limits mechanism was the ability to limit disk throughput. This project aims to fill that hole by making it possible to add RCTL rules for read bytes per second (BPS), write BPS, read I/O operations per second (IOPS), and write IOPS. It also adds a new throttling mechanism, to delay process execution when a limit gets hit.
The project is at the late implementation stage. The major piece of work left, apart from testing, is to integrate it with ZFS. The project is expected to ship with &os; 11.0.
The &os; Release Engineering Team is responsible for setting and publishing release schedules for official project releases of &os;, announcing code freezes and maintaining the respective branches, among other things.
During the last quarter of 2015, the Release Engineering team added support for three additional &os;/arm systems: BANANAPI, CUBIEBOARD, and CUBIEBOARD2.
In addition to regular development snapshot builds for &os; 11.0-CURRENT and &os; 10.2-STABLE, several changes and enhancements were made to the release build code. Of note, the release build code no longer produces MD5 checksums, in favor of SHA512.
Toward the end of the year, much of the primary focus was centered around the upcoming &os; 10.3 release cycle, which will begin during January 2016.
As always, help testing development snapshot builds is crucial to producing quality releases, and we encourage testing development snapshots whenever possible.
The goal of this project is to reimplement the exisitng MMC/SD stack using the CAM framework. This will permit utilizing the well-tested CAM locking model and debug features. Additionally, it will be possible to process interrupts generated by the inserted card, which is a prerequisite for implementing the SDIO interface.
The first version of the code was uploaded to Phabricator for review. The new stack is able to attach to the SD card and bring it to an operational state, so it is possible to read and write to/from the card.
The only supported SD controller driver is ti_sdhci, which is used on the BeagleBone Black. Modifying other SDHCI-compliant drivers should not be difficult.
Rework bus/target/LUN enumeration and the locking model — I don't really understand the CAM locking and am likely to do it incorrectly.
Modify the SDHCI driver on at least one x86 platform — this will make development and collaboration easier.
Begin implementing SDIO-specific bits.
Several important ports were updated: Mesa to 11.0.8, the X.Org server to 1.17.4, libdrm to 2.4.65, as well as many applications and libraries. The latest release of the X.Org server, 1.18, is being tested in our Ports development tree.
On the kernel side, the i915 update is almost ready to land. There are a couple known regressions for currently supported GPUs that we want to fix before committing.
We started a discussion on the FreeBSD-x11@ mailing list to organize future contributions to the kernel drivers. We have already received some valuable comments. We are confident that future updates will happen at a faster pace, thanks to several motivated people!
FOSDEM is held in Brussels on the 30th and 31st of January. We will attend this conference. It will be a perfect time to see people again from &os; and from the XDC. On Sunday, we will give a talk about how to contribute to the Graphics Stack.
Our blog is currently down because the service was discontinued. We hope to get a dump of our data to put it back online elsewhere. Unfortunately, there is no ETA for this item.
See the "Graphics" wiki page for up-to-date information.
Kernel crash dumps contain information about currently running processes. This can include sensitive data, for example passwords kept in memory by a browser when a kernel panic occurred. An entity that can read data from a dump device or a crash directory can also extract this information from a core dump. In order to prevent this situation, the core dump should be encrypted before it is stored on the dump device.
This project allows a kernel to encrypt a core dump during a panic. A user can configure the kernel for encrypted dumps and save the core dump after reboot using the existing tools, dumpon(8) and savecore(8). A new tool decryptcore(8) was added to decrypt the core files.
A patch has been uploaded to Phabricator for review. The project is currently being updated to address the review comments, and should be committed as soon as it is accepted. For more technical details, please visit the FreeBSD-security mailing list archive or see the Phabricator review.
By the end of the Q4 2015 period, &a.koobs; (koobs@) started an initiative to form an experimental Bugzilla Triage Team. The main goals of the team are to increase community involvement (addition/training of new triagers) and enhance current procedures and tools, among others. This experiment was started with the participation of Vladimir (blackflow on irc/freenode) and Rodrigo (DanDare on irc/freenode), who approached koobs@ with a desire to contribute and get more involved with the &os; Project. This experimental pilot project has the task of setting up procedures for enhanced Issue (Problem Report) management that include better classification/prioritization, eventually leading to faster resolution of the issues.
We are now happy to report on the progress of this experimental team:
Since the Issue Triage Team is very young, we expect more information be available and more actions be reported upon in the next Status report.
Set up the Wiki namespace and organize the brainstorm document into a meaningful set of documents.
Recruit more suitable triagers into the team.
The relaunchd project provides a service management daemon that is similar to the original launchd introduced in Apple OS X.
It is not limited to the original features of launchd, however: interesting work is being done to add support for launching programs in jails, passing socket descriptors from the host to a jail, and launching programs within a preconfigured capsicum(4) sandbox. Additionally, relaunchd uses UCL for its configuration files, so jobs can be defined in JSON or other formats supported by UCL.
While there is still work to be done, most of the important features of the original launchd have been implemented, and relaunchd has been made available in the &os; Ports Collection. It should still be considered experimental and not ready for production use, but everyone is welcome to try it, report issues, and contribute code or ideas for improvement.
Add support for restarting jobs if they crash.
Implement the cron(8) emulation feature.
Add support for monitoring files and directories for changes, and launching jobs when changes are detected.
Finish things that are incomplete, such as support for jails and passing open socket descriptors to child processes.
Improve the documentation and providing more examples of how to use it.
When &os; virtual machines (VMs) run on Hyper-V, in order to get the best network and storage performance and make full use of all the benefits that Hyper-V provides, it is recommended to use Hyper-V synthetic devices. The collection of drivers that are required to run Hyper-V synthetic devices in &os; are known as &os; Integration Services (BIS). Some of the BIS drivers (like network and storage drivers) have existed in &os; 9.x and 10.x for years, but there are still some performance and stability issues and bugs. Additionally, compared with Windows and Linux VMs, the current BIS lacks some important features, such as the virtual Receive Side Scaling (vRSS) support in the Hyper-V network driver and the support for UEFI VM (boot from UEFI), among others.
We are now working more on the issues and performance tuning to make &os; VM run better on Hyper-V and the Hyper-V based cloud platform Azure.
Our work during 2015Q4 is documented below:
Building on the new in-kernel iSCSI initiator stack released in &os; 10.0 and the recently added iSCSI offload interface, Mellanox Technologies has developed iSCSI extensions for RDMA (iSER) initiator support to enable efficient data movement using the hardware offload capabilities of Mellanox's 10, 40, 56, and 100 Gigabit Infiniband (IB)/Ethernet adapters.
Remote Direct Memory Access (RDMA) has been shown to have a great value for storage applications. RDMA infrastructure provides benefits such as zero-copy, CPU offload, reliable transport, fabric consolidation, and many more. The iSER protocol eliminates some of the bottlenecks in the traditional iSCSI/TCP stack, provides low latency and high throughput, and is well suited for latency aware workloads.
This work includes a new ICL module that implements the iSER initiator. The iSCSI stack is slightly modified to support some extra features such as asynchronous IO completions, unmapped data buffers, and data-transfer offloads. The user will be able to choose iSER as the iSCSI transport with iscsictl.
The project is in the process of being merged to &os; 11-CURRENT and is expected to ship with &os; 11.0.
Numerous improvements for the ARMv6/v7 kernel and tools have been developed by the Semihalf team. Those include:
Most of the introduced changes have been committed to HEAD and more are on the way.
Finish upstreaming the hardware watchpoints support.
Two major concerns have occupied much of core's attention during the last quarter: the reorganisation of the Security Team and the question of whether to import GPLv3 licensed code into the source repository.
The Security Team reorganisation, first proposed to Core during a meeting at BSDCan this year by Gleb Smirnoff — core member and newly-appointed deputy Security Officer (SO) — has now been accomplished. In order to improve the project's responsiveness to security alerts, to maintain security on privileged information received in confidence before general publication and, not least, to reduce the work load on the security officer, the role of the SO team has been redefined as the controller of the distribution of security sensitive information within the project; they are responsible for interfacing with external bodies and individuals reporting security problems, and connecting them with appropriate individuals within the project with the technical expertise to address the identified concerns. The SO team was cut down to just the Security Officer and their deputy, assisted by a secretary, and with input and help in drafting security advisories from former and any potential future Security Officers plus liasons with Core, Cluster Administration and Release Engineering.
Core would particularly like to thank the former members of the Security Team group for their past contributions, now that the Security Team role has been merged into the Security Officer's responsibilities.
The other large question concerning Core is how to provide a modern toolchain for all supported achitectures. Tier 1 architectures are required to ship with a toolchain unencumbered by onerous license terms. This is currently provided for i386 and arm64 by the LLVM suite, including the Clang compiler, LLD and LLDB. However LLVM support for other (Tier 2 or below) architectures is not yet of sufficient quality to be viable, and the older but pre-existing GPLv2 toolchain cannot support some of the interesting new architectures such as arm64 and RISC V. Pragmatically, in order for the project to support these, until LLVM support arrives we must turn to the GNU project's GPLv3 licenced toolchain.
The argument here is whether to import GPLv3 licensed code into the &os; src repository with all of the obligations on patent terms and source code redistribution that would entail, not only for the &os; project itself but for numerous downstream consumers of &os; code. Not having a toolchain readily available is a big impediment to working on a new architecture.
One potential solution is to create a range of "GPLv3 toolchain" base-system packages out of a completely separate source code repository, for instance within the &os; area on Github. These would be distributed equivalently to the other base system binary packages when that mechanism is introduced.
Core recognises that this is a decision with wide-ranging consequences and will be producing a position paper for circulation amongst all interested parties in order to judge community opinion on the matter. Core welcomes feedback from all interested parties on the subject.
Beyond these two big questions, Core has handled a number of lesser items:
This quarter also saw a particularly large influx of new commit bit requests, with on occasion, four votes running simultaneously. Please welcome Kurt Lidl, Svatopluk Kraus, Michal Meloun, Jonathan Looney (Juniper), Daisuke Aoyama, Phil Shafer (Juniper), Ravi Pokala (Panasas), Anish Gupta and Mark Bloch (Mellanox) to the ranks of src committers. In addition, core was delighted to restore commit privileges for Eric Melville after a hiatus of many years.
No commit bits were taken in during the quarter. A non-committer account was approved for Kevin Bowling of LimeLight Networks. Kevin will be doing systems administration work with clusteradm with particular interest in the parts of the cluster that are now hosted in LLNW's facilities. Deb Goodkin of the &os; Foundation was added to the developers mailing list: she was one of the few members of the Foundation Board not already on the list, and having awareness of what is going on in the developer community will help her to support the project more effectively.
The projects/routing Subversion branch is a FreeBSD routing system rework aimed at providing performance, scalability and the ability to add advanced features to the routing stack.
Currently, the packet output path suffers from excessive locking, acquiring and releasing 4 distinct contested locks is required in order to convert a packet to a frame suitable to put on the wire. The first project goal is to reduce the number of locks needed to just two rmlock(9)s for the output path, which permits close-to-linear scaling.
Since September, one of the locks (used to protect link-level entries) is completely eliminated from the packet data path. A new routing API was introduced, featuring better scalability and hiding routing internals. Most of the consumers of the old routing API were converted to use the new API.
&a.bdrewery; (bdrewery@) has been working to improve the build framework as well as buildworld build times. The build system has largely been untouched by large-scale changes for many years. Most of the effort has been on improving the recent META_MODE merge that was presented at BSDCan 2014. This is a new build system that is not currently enabled by default but brings many benefits. Beyond that, some highlights of the work changing buildworld are:
See the &os;-arch mail for more information on planned work.
The ELF Tool Chain project provides BSD-licensed implementations of compilation tools and libraries for building and analyzing ELF objects. The project began as part of &os; but later became an independent project in order to encourage wider participation from others in the open-source developer community.
In the last quarter of 2015 the ELF Tool Chain tools were updated to a snapshot of upstream Subversion revision 3272. Improvements include better input file validation, RISC-V support, support for Xen ELF notes, additional MIPS and ARM relocations, better performance, and bug fixes.
The ELF Tool Chain project is planning a new release in the first quarter of 2016, which should facilitate wider testing and use by projects other than &os;.
Add missing functionality (PE/COFF suppor) to elfcopy and migrate the base system build.
Fix issues found by fuzzing inputs to the tools.
Add automatic support for separate debug files.
LLDB is the debugger from the LLVM family of projects. Originally developed for Mac OS X, it now also supports &os;, NetBSD, Linux, Android, and Windows. It builds on existing components in the larger LLVM project, for example using Clang's expression parser and LLVM's disassembler.
LLDB in the &os; base system was upgraded to version 3.7.0 as part of the Clang and LLVM upgrade, and it will similarly be upgraded again, to 3.8.0 for &os; 11.0-RELEASE.
LLDB is now enabled by default on the amd64 and arm64 platforms. It is now a functional basic debugger on arm64, after a number of fixes were made in the last quarter to both LLDB and the &os; kernel.
Rework the LLDB build to use LLVM and Clang shared libraries.
Port a remote debugging stub to &os;.
Add support for local and core file kernel debugging.
Improve support on architectures other than amd64 and arm64.
A number of UEFI bug fixes were committed over the last quarter, further improving compatibility with different UEFI implementations. Specifically, on some implementations, &os; failed to boot with an "ExitBootServices() returned 0x8000000000000002" error. This has been fixed with a retry loop (as required by UEFI), in r292515 and r292338.
UEFI improvements from other developers have recently been commited or are in progress. These include support for environment variables set on the EFI loader command line, improved text console mode setting, support for nvram variables, and root-on-ZFS support.
Test &os;-CURRENT snapshots on a variety of UEFI implementations.
Merge UEFI changes to stable/10, for &os; 10.3-RELEASE.
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the &os; Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage development projects, conferences and developer summits, and provide travel grants to &os; developers. The Foundation purchases hardware to improve and maintain &os; infrastructure and publishes &os; white papers and marketing material to promote, educate, and advocate for the &os; Project. The Foundation also represents the &os; Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.
Here are some highlights of what we did to help &os; last quarter:
On the advocacy front, the Foundation attended and sponsored EuroBSDcon, which took place Oct 1-4 (https://2015.eurobsdcon.org/ )in Stockholm, Sweden. Two days prior, during the developer summit, Deb Goodkin ran a session on Recruiting to &os;. The Foundation was also very active during the event itself; in addition to Deb, we had &a.dru;, &a.mckusick;, &a.erwin;, &a.emaste;, &a.hrs;, and &a.trasz; attend the conference. Deb and Ed gave a presentation on how the Foundation supports a BSD project. Kirk gave a presentation on "a Brief History of the BSD Fast File System," and he taught the two-day tutorial "Introduction to the &os; Open-Source Operating System."
Deb then attended the 2015 Grace Hopper Conference that was held in Houston, TX, October 14-16. The conference is for women in computing and most of the attendees were female computer science majors, female software developers, and college professors. The Foundation was proud to be a Silver Sponsor. The conference was very successful for us; our presence allowed us to to raise awareness of the Project, help recruit more women, and get more professors to include &os; in their curriculum.
&a.gnn; traveled to Bangkok, Thailand to present talks on DTrace, &os;, and teaching with DTrace, at Chulalongkorn University, which is the largest University in Thailand with the largest engineering school. The first talk he gave was his practitioner's introduction to DTrace in which he explains the technology, history and usage, without diving into all the kernel subsystems. The second was the sales pitch for teaching with Dtrace and with &os;. The pitch was well received and there was some very good points made by the audience. The facts that the course materials are both open source and hosted on github were also well received.
&a.mckusick; completed a 10-hour tutorial about &os; for Pearson Education in their "Live Lesson" program. In particular, there is a great free snippet from that course comparing &os; against Linux here: http://youtu.be/dTpqALCwQ1Y?a . Find out more about the whole session at: http://click.linksynergy.com/fs-bin/click?id=NZS3W7D*uS0&subid=&offerid=163217.1&type=10&tmpid=3559&RD_PARM1=http%253A%252F%252Fwww.informit.com%252Fstore%252Fintroduction-to-the-freebsd-open-source-operating-system-9780134305868 .
Anne Dickison resumed the Faces of &os; series with interviews featuring Michael Dexter and Erin Clark. She also continued to produce and distribute &os; materials for conferences, as well as advocating for &os; over our social channels.
&a.gnn; headed up the latest Silicon Valley Vendor and Developer Summit, November 2-3, at the NetApp campus in Sunnyvale, California. Topics of discussion ranged over new developments in persistent memory, the use of &os; by a company that builds rackscale systems, developments in our compiler and tool suite, as well as others. Additional Foundation Board and Staff attending the summit included: Deb Goodkin, &a.gjb;, &a.gibbs;, &a.mckusick;, &a.emaste;, and &a.hrs;. The complete schedule, and some of the slides, are available on the &os; Wiki https://wiki.freebsd.org/201511VendorDevSummit .
Notes from the always lively "Have/Need/Want session" are available at https://wiki.freebsd.org/201511VendorDevSummit/HaveNeedWant .
While in the Bay Area, some Foundation members visited commercial users of &os; to help understand their needs, update them on the work the Foundation is doing, and facilitate collaboration between them and the Project.
We were a sponsor of the 2015 OpenZFS Developer Summit, which took place October 19-20, in San Francisco, CA. &a.gibbs; and &a.mckusick; attended the conference.
&a.gibbs; continued his semester long class teaching Intro to Computer Science using &os; at a middle school.
&a.emaste;, &a.trasz;, and &a.kib; continue to make progress on Foundation funded development projects. More specifically:
&a.emaste; presented a &os;/arm64 talk and a hands-on demo at ARM Techcon, which took place November 10-12, 2015, in Santa Clara, CA.
We continued publishing our monthly newsletters and acquiring new company testimonials about using &os;, including from Verisign and Nginx.
Anne Dickison, &a.dru;, and &a.gjb; represented the Foundation at USENIX LISA '15, which took place November 3-8, in Washington D.C.. The Foundation had a booth in the Expo Hall and participated in a BoF. In addition to connecting with current community members, we spoke with attendees who were interested in getting involved with the Project, and helped set them on the correct path. We also took the opportunity to remind those who had not used &os; in a while what they were missing. Glen also attended the USENIX Release Engineering Summit, which was co-located with LISA '15.
We published the Sept/Oct and Nov/Dec issues of the &os; Journal.
&a.gnn; and &a.rwatson; announced the release of their TeachBSD initiative: http://teachbsd.com/. TeachBSD offers a set of reusable course materials designed to allow others to teach both university students and software practitioners &os; operating system fundamentals. The Foundation is proud to have partly sponsored their efforts to teach the initial graduate level course on operating systems with tracing at the University of Cambridge.
Deb Goodkin invited a representative from the Outreachy program to talk at the Ottawa &os; Developer Summit about the program and how we can get involved.
Deb also started discussions with CS professors from the University of Colorado, Boulder to offer some Intro to &os; workshops.
&a.gjb; continued wearing many hats to support to the Project. For Release Engineering:
&a.emaste; attended the Reproducible Builds World Summit, which took place in Athens, Greece, December 1-3, 2015.
We wrapped up our 2015 fundraising efforts with our End-of-Year fundraising campaign by participating in #GivingTuesday, and continuing with weekly email and social media requests for support of the Foundation. Final fundraising numbers will be available in Q1 2016.
Xen is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. Xen support for &os; on x86 as a guest was introduced in version 8, and ARM support is currently being worked on. Support for running &os; as an amd64 Xen host (Dom0) is available in HEAD.
Regarding x86, the work done during this quarter has been focused in rewriting the PVH implementation inside of Xen, to what is now being called HVMlite (in order to differentiate it with the previous PVH implementation). The Xen side of patches have already been committed to the Xen source tree, and will be available in Xen 4.7 (the next version). Work has also begun on implementing HVMlite Dom0 support, although no patches have yet been published.
HVMlite support for &os; has not yet been committed, although an initial implementation is available in a personal git repository. The plan is to completely replace PVH with HVMlite on &os; as soon as HVMlite supports Dom0 mode.
Apart from this, Wei Liu is working on improving netfront performance on &os;; initial patches have been posted to the &os; review system.
During this time frame, the x86 unmapped bounce buffer code has also been improved, and unmapped IO support has been added to the blkfront driver.
Finish HVMlite Dom0 support inside of Xen.
Deprecate and remove PVH support from Xen.
Remove PVH support from &os; and switch to HVMlite.
Generalize the event channel code so it can be used on ARM.
Improve the performance of the various backends (netback, blkback).
Support was added for kernel modules. This included adding the needed relocation types to the in-kernel relocator, and updating the build logic to build modules for arm64. CTF data is currently not generated for modules due to a linker bug.
Shared page support was added. This allows gettimeofday(2) to be implemented in userland by directly accessing the timer register. This reduces the overhead of these calls as we no longer need to call into the kernel. This also moves the signal trampoline code away from the stack allowing for the stack to become non-executable.
CloudABI support for arm64 was added. This included moving the machine-independent code into a separate file to be shared among all architectures. An issue in the arm64 kernel was found and fixed thanks to the CloudABI test suite.
Self-hosted poudriere package builds have been tested. These complement the previous build strategy of using qemu usermode emulation. With this combination of self-hosted and qemu usermode building, many ports that used to be broken on arm64 have been fixed resulting in over 17000 ports building for the architecture.
The machine-dependent portions of kernel support for single-stepping userland binaries has been started. This will allow debuggers, such as lldb, to step through an application while debugging.
Many small fixes have been made to &os;/arm64. These include fixing stack tracing through exceptions, printing more information about "data abort" kernel panics, cleaning up the atomic functions, supporting multi-pass driver attachment, fixing userland stack alignment, cleaning up early page table creation, fixing asynchronous software trap handling, and enabling interrupts in exception handlers.
The SoftIron Overdrive 3000 is an ARMv8 based server with an + 8-core AMD Opteron A1100 processor. The Overdrive 3000 has + two 10Gbase-T ethernet ports, two PCI Express ports, and eight + SATA ports. &os; has been updated to be able to boot on this + hardware.
+ +Support for the SATA device was added to the + ahci(4) driver. Unlike on x86, this is a Memory Mapped + (mmio) device, and not on the PCI bus. To support this, a new + ahci mmio driver attachment has been added.
+ +The generic PCIe driver has been updated to improve interrupt + handling. This includes supporting the interrupt-map devicetree + property, and supporting MSI and MSI-X interrupts on arm64.
+ +Support for MSI and MSI-X interrupts has been added to the ARM + Generic Interrupt Controller v2 (gicv2) driver. This allows devices + to use these interrupts. This has been tested with a collection + of PCIe NIC hardware.
+ + +Write a driver for the 10Gbase-T NIC.
+