Index: head/en_US.ISO8859-1/htdocs/news/status/report-2015-07-2015-09.xml
===================================================================
--- head/en_US.ISO8859-1/htdocs/news/status/report-2015-07-2015-09.xml (revision 47667)
+++ head/en_US.ISO8859-1/htdocs/news/status/report-2015-07-2015-09.xml (revision 47668)
@@ -1,3685 +1,3685 @@
The third quarter of 2015, from July to September, was
again a period of busy activity for &os;: for the second quarter
in a row we have the largest report yet published. The Foundation continues to play a strong role, bringing
both a developer and evangelist presence to conferences, funding
much of the hardware that the cluster administration team uses to
keep things running, and sponsoring many development projects for
&os;. This quarter we also hear from some of the student projects
funded by Google Summer of Code 2015, ranging a wide gamut from
the bootloader to additional ARM support, but also at a range of
completion status. Some of the GSoC output is in the tree
already, but others could benefit from additional attention to
help out our budding new contributors as their schedules fill with
the return to classes. ZFS and the network stack continue to be strong areas for
&os;, with both receiving active maintenance and feature
improvements during this quarter. Substantial work continues on
arm64, potentially putting it on the path toward a promotion to
Tier-1 status, and a new port to the RISC-V architecture has
made great headway in a short period of time. But it is not just
our strengths and exciting new areas that have seen attention this
cycle; there are also some parts of the system that are frequently
perceived as unchanging infrastructure that have received
attention and improvements, with truss and
(k)gdb receiving significant overhauls, new
implementations for the man page tools being brought in, the
website receiving a new skin, and a brand new system for
translating documentation that greatly lowers the barrier to
entry. Nonetheless, despite its record length, this report does
not and cannot cover all of the work being done on &os; throughout
the reporting period — there are many bug fixes too minor to
mention here, and developers too busy working on the next project
to write up an entry for the previous project. It is not just the
developers committing to Subversion that comprise the ongoing
activities of &os;, but also the users testing unreleased
code or reporting bugs in released code, and participants on the
mailing lists and forums helping each other solve their problems.
Even the chats on IRC that wander far from the stated topic of a
channel contribute to the community around &os;; it is that
community whose effectiveness and helpfulness is a key component
of the effectiveness and usefulness of &os; itself. Not just to
the developers listed in this report, but to everyone in the
community, thank you for making &os; a great operating system. —Ben Kaduk
Please submit status reports for the fourth quarter of 2015 (from October to December) by January 7, 2016.
A new driver, ioat(4), was added to the tree. ioat(4) supports Intel's I/O Acceleration Technology devices which are found on some Intel server systems.
These devices are DMA offload engines, which can accelerate some I/O-heavy applications by offloading memory copies from the main CPU to the I/OAT unit. This acceleration is not transparent; applications must be adapted to take advantage of the hardware.
Some I/OAT models support more advanced copying modes, such as XOR; these modes are not yet supported in the ioat(4) driver.
Further testing, especially on a range of device models other than BDXDE (looking for volunteers here).
Support for the more advanced copy modes.
IPsec is now enabled by default in the GENERIC kernel configuration, and work is proceeding to speed things up in various ways. The latest changes are the addition, by &a.jmg;, &a.eri;, and &a.gnn;, of AES modes both in hardware and in software. Part of this work also includes more benchmarks undertaken using Conductor in the netperf project. Results have been reported at BSDCan and vBSDcon with more to come at EuroBSDcon and BSDCon Brasil.
Performance improvements and other tweaks are ongoing.
With the advent of DTrace we are able to replace many of the internal kernel debugging options, such as TCPDEBUG, with statically defined tracepoints (SDTs). Tracepoints have now been added to the system that replicate the functionality of the TCPDEBUG kernel option. No new kernel options need to be added — they are standard with any kernel that has DTrace, which is included in the default GENERIC kernels in 10.X and HEAD.
The Acer C720 Chromebook is an affordable (under $200) and powerful little laptop that provides a battery life of up to six hours running &os;. It is a great machine for travelling and coding in general. The machine is fully functional, meaning that all essential devices work: keyboard, trackpad, light sensor, backlight control, display in VESA mode (fast), external Display on HDMI (only VESA mirror mode), sound, USB ports, SD card slot, camera, and Atheros wireless.
This quarter, this project extended previous work on the boot process and keyboard driver as well as the smbus(4) driver. It added three new drivers: ig4(4) for the I2C bus, cyapa(4) for the trackpad, and isl(4), for the ambient light sensor.
Much of the development was originally done in late 2014. Since then, the patches have been massively improved and merged into HEAD, so that all relevant devices work without manual patching.
For those who are unable to run HEAD, there is a backported patch to 10.2-RELEASE.
Thanks to everyone who helped in the process. I couldn't have done it without you (you know who you are).
This project aims to add support for the LiquidIO family of high-performance programmable accelerator 10/40-gigabit Ethernet network adapters. The currently developed kernel driver supports CN6640- and CN6880-based PCIe cards, enabling these features:
The project is currently being developed in house and is being prepared for upstream. We plan on making it available in &os; 11.
The &os; Release Engineering Team is responsible for setting and publishing release schedules for official project releases of &os;, announcing code freezes, and maintaining the respective branches, among other things.
In mid-August, the &os; Release Engineering Team released &os; 10.2-RELEASE, two weeks earlier than the original schedule anticipated.
The &os; Release Engineering Team would like to thank all that have tested the BETA and RC builds and reported issues during the release cycle.
The &os; Release Engineering Team, with approval from the &os; Core Team, appointed &a.marius; as the Deputy Lead.
This summer we have started porting bhyve onto ARMv7 platforms. The low-level routines for ARM processors were rewritten while trying to preserve the hypervisor API originally created for the x86 architectures. We managed to bring up a &os; guest up to the point of initializing interrupts. There is still work to be done in order to virtualize the interrupts and the timer. Our short-term plan after finishing the interrupts and the timer is porting to a real hardware platform (Cubie2).
Virtualize interrupts and timer.
Port to a real hardware platform.
Create SMP support for bhyve-on-arm.
Port to ARMv8.
GitLab is a web-based Git repository manager with many features, used by more than 100.000 organizations, including NASA and Alibaba. It also is a very long-standing entry on the "Wanted Ports" list on the &os; Wiki.
In the last month there was steady progress, finally resulting in the PR for adding the new port. In addition to the many dependencies &a.pgollucci; is working on, there was already a large amount of work done. Along with many new or updated rubygems, Rails 4.1 was resurrected. A large group of committers were involved in the process and guided us through the various problems and pitfalls.
Because of the number of dependencies — we nearly hit 100 — making progress takes some time. In the meantime, a new major version of GitLab has already been released, requiring even more dependencies and updates. Work on this version is in progress, but the first goal is to get the latest stable version from the 7.14 branch into the ports tree.
Closing all the PRs of the dependencies
Committing the GitLab port itself
Updating the port to the latest version of the 8.x branch
Xfce is a free software desktop environment for Unix and Unix-like platforms, such as &os;. It aims to be fast and lightweight, while still being visually appealing and easy to use.
During this quarter, the team has kept these applications up-to-date:
We also follow the unstable releases (available in our experimental repository) of:
In the trunk branch, x11-wm/xfce4-panel contains a patch to support sysutils/xfce4-panel-switch (available through the panel preferences).
Test the new stable release of GLib 2.46.x with the kqueue/kevent backend enabled (it was disabled with revision - r393663. + r393663). Currently several features are broken, especially in Thunar, xfce4-panel, and Xfdashboard.
Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.
The goal of this project is to make it easy to install the modules available in the npm package registry.
Currently, the repository contains more than 100 new ports, in particular:
We have also written several helpers for the porting, available in our experimental repository.
Bring in grunt.js (and modules), the JavaScript task runner.
Put more effort into support of node-gyp in the +
Put more effort into support for node-gyp in the USES framework
A feature long missing from &os; was the ability to boot up with a temporary rootfs, configure the kernel to be able to access the real rootfs, and then replace the temporary root with the real one. In Linux, this functionality is known as pivot_root. The reroot project aims to provide similar functionality in a different, slightly more user-friendly, way. Simply put, from the user's point of view it is as simple as running reboot -r. The system performs a partial shutdown, killing all processes and unmounting the rootfs, and then partial bringup, mounting the new rootfs, running init, and running the startup scripts as usual.
The kernel part of the project has been committed to 11-CURRENT. The userland part is at the "finishing touches" stage, and is expected to be committed soon. A merge to stable/10 is planned and reroot support is planned be included in &os; 10.3.
We have updated clang, llvm, lldb, compiler-rt, and libc++ in base to the 3.7.0 release. These all contain numerous improvements. Please see the linked release notes for more detailed information. This brings us completely up-to-date with the latest upstream versions of these projects. Meanwhile, &a.emaste; is working on importing the llvm.org version of libunwind.
Like the 3.5.x and 3.6.x releases, these components require C++11 support to build. At this point, &os; 10.0 and later provide that support, at least on x86. Currently, there are no solid plans to MFC these versions to any stable branches, due to the difficulties this would introduce for the usual upgrade scenarios.
Thanks to &a.emaste; and &a.andrew; for their help with this import, and thanks to &a.antoine; for several ports exp-runs.
During the first ports exp-run, some major problems were found, one introduced by a clang bug which caused pow() to generate floating point exceptions in some cases. This in turn caused libpng to fail to build, and one bug in libjpeg-turbo, which was caused by undefined behavior. These two problems took some time to fix, after which another exp-run was done, and this resulted in about a dozen newly failed ports. For almost all of these new failures, fixes were submitted and linked to the original PR 201377 for the exp-run.
A number of UEFI bug fixes were committed over the last quarter, improving compatibility with different UEFI implementations. This includes improvements to EFI's vt(4) framebuffer driver, efifb, to handle systems with high resolution displays and unusual framebuffer stride values. In particular, this improves compatibility with a large number of recent Apple MacBook Pros and other Macs.
The Mesa ports were updated to 10.6.8. At the same time, the ports received a major overhaul to make sure all ports are correctly configured. Dual version support was removed. There is only one Mesa version for all supported &os; versions. The libosmesa port, which provided the off-screen version of Mesa, was merged into the Mesa framework.
Another big item that was included in the Mesa port is OpenCL. There are two GPU-based OpenCL implementations: lang/clover for supported Radeon cards, and lang/beignet for supported Intel cards (currently only Ivybridge). Thanks go to Johannes Dieterich, O. Hartmann, and Koop Mast for making this happen.
Now that Mesa is up-to-date, we can apply the same update procedure to the X.Org server. It is currently at 1.14, and an update to 1.17 is ready. It will be committed shortly.
On the kernel side, progress has been made with the i915 update. The driver is able to attach. There are some reports that the X.Org server starts but Mesa is unhappy, so acceleration does not work yet. If you want to test, instructions will be posted on the wiki in the i915 update article (see links). At this stage, we can only accept patches, though — we will not be able to provide support.
We attended two conferences: XDC 2015 in Toronto and EuroBSDcon 2015 in Stockholm. Reports will be posted on the blog.
See the Graphics wiki page for up-to-date information.
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the &os; Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage development projects, conferences and developer summits, and provide travel grants to &os; developers. The Foundation purchases hardware to improve and maintain &os; infrastructure and publishes &os; white papers and marketing material to promote, educate, and advocate for the &os; Project. The Foundation also represents the &os; Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.
Here are some highlights of what we did to help &os; last quarter:
Anne Dickison and Deb Goodkin attended OSCON to promote &os;.
&a.rwatson; organized and ran the Cambridge &os; Developer Summit 2015 ("BSDCam"). We provided travel grants to two &os; developers to attend the summit. Three Foundation board/staff members attended too.
&a.gnn; attended the ARM Partner Meeting where he met with 15 silicon and systems vendors to present the unique traits and qualities of &os; and work on setting up partnerships with the companies building and deploying ARM hardware.
George and &a.rwatson; collaborated in Cambridge on developing further &os;-based teaching material at undergraduate and masters levels. Part of this project was funded by the Foundation.
George planned and ran the DevSummit at vBSDCon 2015.
We were proud to be a sponsor of
Cheryl Blain and &a.jhb; promoted the Foundation and &os; at the SNIA 2015 Storage Developer Conference, in Santa Clara, California, Sept 21-24. The Foundation was also a sponsor.
We sponsored Andy Turner to attend Linaro Connect in San Francisco, Sept 21-25.
&a.emaste;, our project development director, attended the X.Org Developer's Conference (XDC) in Toronto, Ontario.
We sponsored the 2015 nginx Conference and sent &os; community member &a.jhb;.
George Neville-Neil continued planning the
&a.bcr; and &a.erwin; helped plan and organize the EuroBSDCon &os; Developer Summit. This included setting up the working groups, securing the venue, and getting the T-shirts made.
Benedict helped organize, and he and &a.dru; participated
in the
&a.dru; taught a &os; class in Berlin, Germany July 29-31.
We were a sponsor of
We continued to publish our monthly newsletters, keeping the community informed on what we are doing, including event recaps, testimonials, project updates, and upcoming events. We received testimonials from Microsoft, NYCBus, and ScaleEngine. We also continued to approach companies to provide us with testimonials to help promote their use of &os;.
Anne Dickison rebooted the Faces of &os; series and is working with &os; contributors on writing their stories. She continued to produce more &os; Swag and literature to promote &os;, as well as advocating for &os; over our social channels and with new partnerships.
We reached our 2015 goal of 10,000 &os; Journal subscribers, and we published a new Open Journal article on our website, to help promote the Journal. We also started offering a new subscription bundle, where you can buy all the 2014 issues. The July/August issue was published.
&a.gibbs; began teaching a semester-long &os; class at a middle school in Boulder, Colorado. We are using the BeagleBone Black (BBB) to run &os; connected to Macs and PCs. We have received a lot of support, both internally, and from the Project, to get the &os; images to work on the BBB with the Macs and PCs. It has been a great collaborative effort with community members, and this will help future classes in being able to support inexpensive platforms for teaching &os;.
Work continued on creating a &os; curriculum for a half day workshop. Hopefully this will be available in late Spring.
We provided legal support for the Project including granting trademark permission for some users and companies who requested permission to put the &os; logo on their websites and marketing literature.
We met with commercial users to get their input on what they would like to see supported in &os;. We also do this to help connect &os; developers with commercial users to help facilitate collaboration.
&os; Foundation employee and Release Engineer &a.gjb; was extremely busy during this quarter, working on a number of exciting areas of the &os; Project. Some of the highlights include:
The aim of this project is to design and implement infrastructure to validate that a number of the network stack's multiqueue behaviours are functioning as expected.
At present, most of this project has been implemented. It mainly consists of two parts:
However, it still needs further refinements to make it suitable for committing to &os; head.
The &os; GNOME Team maintains the GNOME, MATE, and CINNAMON desktop environments and graphical user interfaces for &os;. GNOME 3 is part of the GNU Project. MATE is a fork of the GNOME 2 desktop. CINNAMON is a desktop environment using GNOME 3 technologies but with a GNOME 2 look and feel.
This quarter, GNOME 3.16 and MATE 1.10 were committed to the ports tree, followed up by some incremental improvements. A chapter covering the use of USE_GNOME within individual ports' Makefiles was written and committed to the Porter's Handbook.
GNOME 3.18 has been ported. There are, however, some issues that need to be resolved before it can be committed to the ports tree.
The &os; GNOME website is stale. Work is under way to improve it.
Please give feedback on and suggest improvements to the chapter in the Porter's Handbook on the USE_GNOME functionality.
Continue working on investigating the issues blocking GNOME 3.18.
Atomic operations serve two fundamental purposes. First, they are the building blocks for expressing synchronization algorithms in a single, machine-independent way using high-level languages. In essense, atomics abstract the different building blocks supported by the various architectures on which &os; runs, making it easier to develop and reason about lock-less code by hiding hardware-level details.
Atomics also provide the barrier operations that allow software to control the effects on memory of out-of-order and speculative execution in modern processors as well as optimizations by compilers. This capability is especially important to multithreaded software, such as the &os; kernel, when running on systems where multiple processors communicate through a shared main memory.
Each machine architecture defines a memory model, which specifies the possible effects on memory of out-of-order and speculative execution. More precisely, it specifies the extent to which the machine may visibly reorder memory accesses to optimize performance. Unfortunately, there are almost as many models as architectures. Some architectures, for example IA32 or Sparcv9 TSO, are relatively strongly ordered. In contrast, others, like PowerPC or ARM, are very relaxed. In effect, atomics define a very relaxed abstract memory model for &os;'s machine-independent code that can be efficiently realized on any of these architectures.
Most &os; development and testing still happens on x86 machines, which, when combined with x86's strongly ordered memory model, leads to errors in the use of atomics, specifically, barriers. In other words, the code is not properly written to &os;'s abstract memory model, but the strong ordering of the x86 architecture hides this fact. The architectures impacted by the code that incorrectly uses atomics are less popular or have limited availability, and the resulting bugs from the misuse of atomics are hard to diagnose.
The goal of this project is to audit and upgrade the usage of lockless facilities, hopefully fixing bugs before they are observed in the wild.
&os; defines its own set of atomics operations, like many other operating systems. But unlike other operating systems, &os; models its atomics and barriers on the release consistency model, which is also known as acquire/release model. This is the same model which is used by the C11 and C++11 language standards as well as the new 64-bit ARM architecture. Despite having syntactical differences, C11 and &os; atomics share essentially the same semantics. Consequently, ample tutorials about the C11 memory model and algorithms expressed with C11 atomics can be trivially reused under &os;.
One facility of C11 that was missing from &os; atomics was fences. Fences are bidirectional barrier operations which could not be expressed by the existing atomic+barrier accesses. They were added in r285283.
Due to the strong memory model implemented by x86 processors, atomic_load_acq() and atomic_store_rel() can be implemented by plain load and store instructions with only a compiler barrier. No additional ordering constraints are required. This simplification of atomic_store_rel() was done some time ago in r236456. The atomic_load_acq() change was done in r285934, after careful review of all its uses in the kernel and user-space to ensure that no hidden dependency on a stronger implementation was left.
The only reordering in memory accesses which is allowed on x86 is that loads may be reordered with older stores to different locations. This results from the use of store buffers at the micro-architecural level. So, to ensure sequentially consistent behavior on x86, a store/load barrier needs to be issued, which can be done with an MFENCE instruction or by any locked read-modify-write operation. The latter approach is recommended by the optimization guides from Intel and AMD. It was noted that careful selection of the scratch - memory location, which is modified by the locked RWM - operation, can reduce the cost of barrier by avoiding false + memory location, which is modified by the locked RMW + operation, can reduce the cost of the barrier by avoiding false data dependencies. The corresponding optimization was committed in r284901.
The atomic(9) man page was often a cause of confusion due to both erroneous and ambiguous statements. The most significant of these issues were addressed in changes r286513 and r286784.
Some examples of our preemptive fixes to the misuse of atomics that would only become evident on weakly ordered machines are:
RISC-V is an open source Instruction Set Architecture (ISA) designed at UC Berkeley. It is freely available for all uses without requiring fees or license agreements. The RISC-V team intends to provide freely available BSD licensed CPU designs.
&a.br; (University of Cambridge) now has &os; booting to a single user shell on a RISC-V simulator. The porting effort started only two months ago and is very much a work in progress, requiring significant refactoring and clean up before it reaches a committable state. Nonetheless, this is exceptional progress in a short time. The porting effort also identified a number of proposed ISA improvements.
The port currently uses the GNU tool chain (GCC and binutils), and runs on the Spike simulator. Improved RISC-V support in Clang/LLVM and related tools is highly desired.
The Ports Collection typically receives less attention on Tier-2 architectures than on Tier-1 architectures, although several build-runs were performed at various points in the past, and broken ports were marked as such at those times.
Some of the Tier-2 platforms, such as PowerPC and ARM, have improved considerably recently, both on &os;'s and the compilers' sides, but as the tree is not rebuilt on the cluster very often, it was suspected that many ports are marked BROKEN while they in fact now build and run correctly.
Over the past several weeks, 26 ports that were indeed broken on at least PowerPC had been fixed, 58 ports that were incorrectly marked as broken (leftovers from the old times) were marked as working, and fewer than 40 ports still have issues requiring further work.
The Ports Collection could benefit a lot from more frequent sweeps targeting Tier-2 systems.
Recent work on QEMU-backed emulators and the much-anticipated cross-building of ports are essential pieces to bring &os; packages on par with the base system's support, architecture-wise.
The biggest task handled by the Core Team during this quarter was developing and publishing the new Code of Conduct. The Code of Conduct describes how people are expected to behave on all &os; official communication channels, as well as how developers and other people involved with the project are to behave when representing the project in public.
The Code of Conduct was generally well received and elicited numerous comments and suggestions for improvements from the community, many of which have been integrated.
The next task handled by Core was the restoration of Babak Farrokhi's ports commit bit. Babak resides in Iran. A few years ago, legal advice suggested that allowing contributions from Iranian residents might violate US trade sanctions. After several years, Core was asked to revisit the issue. On the advice of counsel, Core decided that it could restore commit privileges to commmitters residing in Iran.
The CTM service came under security review. Given that the lack of use of routine authenticity checking made the injection of trivial exploit code relatively easy, the service was held to be too risky to continue as an official part of the &os; base system. CTM has very few remaining users but they should be able to install CTM from the Ports Collection in order to continue doing so.
Core learned that ISC was ceasing its hosting service, which has entailed a rapid rework of plans on the movement of significant portions of the &os; cluster to that data center. Cluster administration has taken ownership of the situation and is making progress.
Core fielded an enquiry about NextBSD and whether this should be the future direction for the whole &os; project. Core's position is that NextBSD is an interesting project, and we regard it, like the other BSD projects, as a potential source of good ideas. However, we currently have no plans to adopt NextBSD as the official &os; distribution.
Beyond these issues, Core also spent time in various routine activities. During this quarter we issued three new src commit bits, and took none in for safekeeping. Welcome to &a.allanjude;, &a.araujo;, and &a.avos;.
As of the end of Q3 the ports tree holds just over 25,000 ports, and the PR count is above 2,000. The summer period saw less activity on the ports tree than during the previous quarter, with fewer than 7,000 commits performed by 120 active committers. Unfortunately, the number of problem reports closed also decreased significantly, with fewer than 1,500 problem reports fixed during Q3.
In Q3 several commit bits were taken in for safekeeping, following an inactivity period of more than 18 months (fluffy), or on committer's request (xmj, stefan, brix). One new developer was granted a ports commit bit (&a.junovitch.email;), and one returning committer (&a.farrokhi;) had his commit bit reinstated.
On the management side, no changes were made to the portmgr team during Q3.
On the QA side, 25 exp-runs were performed to validate sensitive updates or cleanups. Amongst those, the noticeable changes are the update to pkg 1.6, the automake14 removal, and several important port updates such as doxygen to 1.8.10, gnome3 to 3.16, cmake to 3.3.1, and the Qt4 ports to 4.8.7. The default jdk was also set to openjdk8. Some infrastructure changes included the addition of new options helpers: opt_VARS, opt_VARS_OFF, opt_IMPLIES, and opt_PREVENTS. Some macros were also removed, such as UNIQUENAME and LATEST_LINK.
We would like to remind everyone that the ports tree is built and run by volunteers, and any help is greatly appreciated. This is more important than ever, since the number of problem reports cannot seem to stop increasing. So if you use ports or packages, please consider jumping in and helping! This is also true for existing porters: it would be great if you would consider the next step, which is to share your knowledge and mentor someone more junior with the ports tree internals. And if you already do these tasks, many thanks to you!
The &os; Cluster Administration Team consists of the people responsible for administering the machines that the project relies on for its distributed work and communications to be synchronised.
Our primary cluster has been hosted as a guest in California for many years. Our ongoing project is relocating the core functionality to a location in New Jersey with a formal hosting arrangement. This is an equipment refresh, consolidation for better use of resources, and for better continuity of service.
There is a significant amount of behind-the-scenes work to make this happen. The original cluster was implemented with a common, shared, assumed-to-be secure network with ubiquitous NFS everywhere. This structure does not lend itself well to being distributed across geographically diverse locations, particularly when Internet transit is required. The bulk of the work is rebuilding services to be portable, stand-alone components that do not depend on shared-network access and are safe enough to use across the insecure Internet.
Highlights this quarter:
Ongoing:
Overall, we have updated the following ports this quarter:
In our development repository, we have done the following work:
Work on getting the stuff from plasma5 branch into ports. (This is a major update to nearly all KDE applications, so testers are very welcome.)
Finalize the work on PyQt5.
Port qt5-webengine. Qt-5.5 will probably be the last release shipping a www/webkit-qt5 port.
Numerous cleanups and fixes have been applied to the arm64 kernel. This includes fixes to exception handling, asynchronous signals, ddb, and pmap. ddb has been updated to better handle accessing memory that may be unmapped. The pmap code was made more complete by implementing more functions as needed.
Further work on SMP means that &os; now boots on all 48 cores on the Cavium ThunderX platform. This includes adding support for the ARM GICv3 interrupt controllers and fixing the memory mapping to be shareable between CPUs.
The test suite has been run on both qemu and hardware. Most of the test cases are passing, with around 30 tests either broken or failing. Work on diagnosing the issues with the remaining test cases is ongoing.
Port to more SoCs.
The HiKey is a low-cost ARMv8 development board from the Linaro 96boards initiative. It contains a HiSilicon Kirin 6220 with eight ARMv8 cores and 1GB of ram.
&os; has been ported to run on the HiKey with a minimal set of drivers. As of this report, &os; supports the micro-SD slot and USB host, and will boot off the SD card to multi-user mode using a recent arm64 snapshot.
The kernel is missing a number of device drivers. However, it is at a usable state for people interested in testing &os; on ARMv8 hardware.
A driver for SDIO and the onboard WiFi.
Fix the MMC driver to access the eMMC.
Support the USB in OTG mode.
Support a display via HDMI.
Add thermal management.
The &os; documentation translation process has been in need of modernization for quite some time. The existing process was just too difficult for translators to keep translations up to date.
With help from Benedict Reuschling, Shaun McCance, Ryan Lortie, Hiroki Sato, and many others, the availability of a new PO translation system was announced in August.
PO translations handle most of the overhead of the translation process. Translators do not have to keep track of commits to the upstream English version. The actual work of translating is quicker and easier. PO editors show how much of the document has been translated. If a translation is already available for a given string, it can be easily reused.
Early testing has been very successful. Most issues involve discovering and documenting the new processes rather than fixing bugs. New translations of English documents have already been committed.
There will certainly be additional changes and improvements, but the system works. We will continue to discover how to share work between translation teams and the project as a whole. This work will be much easier now that the initial hurdle of being able to use PO software has been passed.
Continue testing. The system is new to us and there are bound to be bugs and situations with unexpected results.
Improve documentation on using the new PO translations. Much of this involves things that rarely happened with the old system, like adding a completely new language directory.
Add new translations for existing documents. There is much less work to create and update a translated document now. Existing and new translators are working on adding and updating translations of the English documentation.
Figure out how to generate and share translation memory with other members of a language team or translators outside the team.
Test new PO editors like Pootle and Virtaal.
Determine a method to allow translators commit access for translations.
Develop and test code to translate manual pages.
The &os; website has remained essentially unchanged in appearance for many years. Like other legacy systems, it is difficult to change. It is heavily used and therefore subject to non-trivial bikeshedding.
The CSS shrunk the reader's font from the size they had requested. It specified hardcoded font and object sizes in pixels. On wide monitors, only the middle third of the screen was used. Hardware has changed from what existed when this version of the site was created. Screens have become larger and wider, and increased in resolution at the same time.
It was time for a change. Font sizes were set to percentages, with none smaller than 100%. The width of the main box was changed to 90%. Other small adjustments were added. These limited changes produced a rendered site that better respects the reader's settings, is much easier to read, and shows more information.
Although no content changed, the appearance was so different that some viewers thought we had redesigned the site. It is gratifying to know that so many people are using it. We would also like to thank people for the response, which was overwhelmingly positive and hardly bikesheddy at all.
Fix other outdated assumptions in the CSS. Alternately, rework the entire site. However, that is a much more complex and ambitious project than it might seem.
pkg 1.6.0 has been released. Many changes have been made since pkg 1.5:
Add a notion of priority to the list of files to ensure that certain files are the first to be replaced. This was a blocker for packaging base.
Investigate replacing openssl by mbedtls.
Xen is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. Xen support for &os; on x86 as a guest was introduced in version 8 and ARM support is currently being worked on. Support for running &os; as an amd64 Xen host (Dom0) is available in HEAD.
On the x86 front, most of the work during this quarter focused on the implementation of PVH inside Xen. Consequently, most of the activity happened inside of the hypervisor. Patches for a clean PVH implementation have been posted, with the aim of having them merged in the next Xen release (4.7). Once that is done, work will continue adding new features to both &os; and Xen to have feature parity with traditional PV guests/hosts.
Apart from this, work is ongoing to import a new netfront from Linux in order to support new features, like split event channel and multiple queue support.
On the ARM front, this quarter's work focused on getting &os;/arm64 booting as a Xen guest. The current activity is to upstream patches preparing Xen drivers to support arm64. This includes a rework of the console driver.
Generalize the event channel code so it can be used on ARM.
Improve backend (netback, blkback) performance.
Work with upstream Xen to improve PVH and make it stable.
Improve the generic bounce buffer code for unmapped bios in order to support the alignment requirements of the blkfront driver.
bhyve is a hypervisor that runs on the &os;/amd64 platform. At present, it runs &os; (8.x or later), Linux i386/x64, OpenBSD i386/amd64, NetBSD/amd64, Illumos, and Windows Vista/7/8/10/2008r2/2012r2/2016 x64 guests. Current development is focused on enabling additional guest operating systems and implementing features found in other hypervisors.
A combined bhyve and ZFS BoF was held during vBSDCon 2015, hosted by Michael Dexter and Allan Jude. Questions asked about bhyve included live migration and suspend/resume support, and configurations using ZFS.
Three bhyve-related projects were selected for GSoC 2015: NE2000 device emulation, porting bhyve to ARM, and ptnetmap support.
The major enhancement for bhyve this quarter was support for external firmware, along with a port of the Intel edk2 UEFI firmware. This allows bhyve to run Windows in headless mode, and also Illumos.
Improve the documentation.
bhyveucl is a work-in-progress script for starting bhyve instances based on a libUCL config file. More information at https://github.com/allanjude/bhyveucl.
Add support for virtio-scsi.
Flexible networking backends: wanproxy, vhost-net.
Support running bhyve as non-root.
Add filters for popular VM file formats (VMDK, VHD, QCOW2).
Implement an abstraction layer for video (no X11 or SDL in base system).
Suspend/resume support.
Live migration.
Nested VT-x support (bhyve in bhyve).
Support for other architectures (ARM, MIPS, PPC).
CAM Target Layer (CTL), when originally developed by Copan/SGI, had support for High Availability clustering. Unfortunately, significant portions of the HA code were never published with the main body of the source code. Now, the missing part has been reimplemented from scratch, fixed, and improved.
This code supports dual-node HA with Asynchronous LUN Unit Access (ALUA) in four modes:
In the case of lost interlink connectivity to primary node, the secondary node falls into the Transitioning state, which is like Unavailable and cannot answer most requests, but makes the initiator wait for recovery or cluster failover.
CTL also got a large number of other improvements, including support for emulation of CD/DVD drives and removable disks, live LUN reconfiguration, and so on.
The code is committed to &os; head and was recently merged to the stable/10 branch.
The ZFS codebase received a significant batch of merges, and is now in sync with the latest Illumos. Among other things, this update includes:
The code is committed to &os; head and was recently merged to the stable/10 branch.
mandoc is a suite of tools for compiling mdoc, the roff macro language of choice for BSD manual pages.
mandoc is the default renderer for manpages on &os; head. This quarter, the apropos(1) utility was switched to use mandoc's version, which offers a new database format (in SQLite) bringing more powerful, fine-grained ways to search man pages.
While mandoc is very good for man pages, we also provide lots of other documentation in plain roff format. The Heirloom toolchain is being studied to replace groff in base. The Heirloom nroff toolchain has multiple benefits: it has very good unicode support and very good compatibility with groff.
A great deal of work as been done testing the Heirloom nroff toolchain with all the roff documents in the base system (including man pages), and upstream has been very proactive in fixing reported bugs.
The soelim(1) utility has been replaced with a BSD-licensed version which is good enough to work with all available roff toolchains to ease the transition. This version of the soelim(1) utility, originally written solely for &os;, is now part of the mandoc tool suite.
In coordination with Ingo Schwarze from OpenBSD, the col(1) utility has been cleaned up and updated to recognize both SUSv2-style escape-digit and BSD-style escape-control-char sequences in the input stream.
The checknr(1) utility has been cleaned up and extended to support modern roff(7) macros, including synchronizing code from NetBSD and the Heirloom doctools version.
Many roff fixes were made to documentation and man pages, having been discovered while testing the new toolchain.
Support for following children after forks for &os; was implemented and merged upstream to GDB's master branch, and was included in GDB 7.10.
Work has continued on porting kgdb to newer gdb. The amd64, i386, powerpc, powerpc64, and sparc64 backends have all been ported and are now available via a new KGDB option in the devel/gdb port.
The MD backends for libkvm have been rewritten to support cross-debugging crashdumps, and the kgdb targets for amd64 and i386 have been reworked to support cross-debugging. Both i386 and amd64 kgdb binaries have been able to cross-debug the other architecture's vmcores with these changes. This changeset for libkvm is not yet in the tree, but is awaiting more testing.
Test the libkvm changes on platforms other than amd64, i386, and powerpc64.
Figure out why the powerpc kgdb targets are not able to unwind the stack past the initial frame.
Add support for more platforms (arm, mips, aarch64) to upstream gdb for both userland and kgdb.
Write a new 1:1-only thread target for &os; that can be sent upstream.
Add support for debugging powerpc vector registers.
The interface between the ABI-specific backends and the truss core was refactored, reducing duplicated code. This prompted additional follow-on work to add support for more ABIs, including aarch64 and CloudABI.
ptrace(2) was extended to return more information about the currently executing system call. This restored behavior that had been present in a previous version of truss: knowing the correct number of arguments for all system calls.
The fork-following support in truss was reworked to use native fork following in ptrace(2) rather than forking a new truss process for each child of a traced process.
Support for decoding more arguments has been added in the last quarter as well.
Create a new libsysdecode library to hold shared code between truss(1) and kdump(1).
Decode more system call arguments.
Add appropriate system call decoding specifications for freebsd32 system calls.
Implement an ABI for 64-bit Linux binaries under &os;/amd64.
sesutil(8) was originally created as a more universal way to blink the "locate" LEDs on most hot-swappable drive enclosures.
This work is based on the original SES tools created by Matthew Jacob in 2000, which have been available in the share/examples section of the source tree, but were not built by default.
The new utility extends the original code with a number of very useful features:
Test sesutil(8) against more hardware.
Diagnose an issue where the locate command sometimes needs to be sent twice to activate the LED.
Add support for libxo output types.
Cavium’s ThunderX is a high-performance 64-bit ARMv8 CPU, available in configurations with up to 48 cores per package. ThunderX is the initial reference platform for the &os;/arm64 porting effort.
Additional Semihalf-sponsored work on ThunderX support brought brand new features such as:
Moreover, previously introduced functionalities have been improved and committed to HEAD. This includes:
The remaining features are being reviewed and will be integrated into HEAD soon. However, the GENERIC kernel already supports and runs on ThunderX.
Upstream the remaining features: 2-socket support, VNIC driver, and PCIe fixes
Multipath TCP (MPTCP) is an extension to TCP that allows for the use of multiple network interfaces on a standard TCP session. The addition of new addresses and scheduling of data across them occurs transparently from the perspective of the TCP application.
The goal of this project is to deliver an MPTCP kernel patch that interoperates with the reference MPTCP implementation, along with additional enhancements to aid network research.
The v0.5 patch was released, which is the first patch of the re-written implementation. We are in the process of documenting the new design and addressing some feedback as provided from the community.
Work has commenced on improved input handling, as the current method of receiving and reassembling segments has been the cause of some instability and stalls during connection shutdown. This will involve re-using the subflow receive buffers and an upcall to enqueue a MP-layer reassembly task without the need to take a lock on the MP control block. The improvements should also allow bypassing mptcp_usrreq for standard TCP connections.
The MPTCP commit history was synchronized with hg-beta.FreeBSD.org, and we have made the repository available on BitBucket (see links). Future patch releases will be tagged there. The tree is now merged with &os; head weekly. An updated v0.51 patch is ready for release.
Release the v0.51 patch.
Populate documentation and the issue tracker on the BitBucket repository.
Improvements to receive-side code before further testing.
Prepare a technical report detailing the design of the current patch.
CloudABI is a POSIX-like runtime environment that uses Capsicum as its sole access control mechanism. CloudABI allows you to develop software that is better hardened against security vulnerabilities, is easier to test, and is easier to migrate across systems.
As of August, all of the kernel modifications that are needed to run CloudABI programs have been integrated into &os; head. After loading the cloudabi64 kernel module, you can either run CloudABI programs directly from the shell or by using the cloudabi-run tool (sysutils/cloudabi-utils). cloudabi-run allows you to inject sockets, files, and directories into the launched program in a more structured way.
In the meantime, work has started on developing a Ports Collection that contains cross-compiled utilities and libraries for CloudABI. The intent is that this framework generates native packages for a number of operating systems, making it possible to develop CloudABI applications on any operating system, regardless of whether that operating system actually supports CloudABI.
If you are interested in CloudABI, be sure to go to the project page on GitHub, watch recordings of talks at conferences, or wait for the upcoming edition of the FreeBSD Journal, which will feature an article on CloudABI.
CloudABI is currently only available for amd64. It would make sense to port CloudABI to additional architectures like aarch64.
Support for CloudABI has only been integrated into &os;. If we manage to upstream support for CloudABI into other operating systems, it should be possible to run the same binary on multiple operating systems without recompilation.
The CloudABI Ports Collection currently has only 60 packages. Although these packages already the building blocks of some interesting software, we are always interested in expanding.
UEFI-enabled boot1.efi and loader.efi have been modified to support loading and booting from a ZFS filesystem, as described in the previous report. The ZFS-enabled loader.efi can be treated as a chainloader when using ZFS-enabled GRUB.
During this quarter, several successful tests have been reported on simple ZFS setups, using both boot1.efi with loader.efi as well as GRUB and loader.efi.
Successful tests have been reported for UFS with the patched boot1.efi and loader.efi as well.
Test reports are needed for more complex ZFS setups, such as with log/l2arc vdevs, mirroring, striping, and raidz.
Pending successful reports, the patch needs to be reviewed and committed.
The Allwinner A10 and A20 chips are ARM CPUs found in increasingly common development boards and other devices, such as the Cubieboard/Cubieboard 2 and the Banana Pi.
With the end of a GSoC project by Pratik Singhal, our A10 and A20 support has improved. Pratik helped with the implementation and testing of the SD card and SATA support for the Allwinner chips.
&a.loos; added support for the dwc network interface on the A20, which is capable of gigabit speeds.
&a.gjb; kindly added support for official &os; images for Cubieboard 2 and the Banana Pi.
Some drivers are still missing: audio, video/HDMI/framebuffer, IR, I2C, SPI, PWM.
Fix if_dwc for better performance.
The nosh project is a suite of system-level utilities for initializing, running, and shutting down BSD systems, and for managing daemons, terminals, and logging. It supersedes BSD init and the NetBSD rc.d system, drawing inspiration from Solaris SMF for named milestones, daemontools-encore for service control/status mechanisms, UCSPI, and IBM AIX for separated service and system management. It comprises a range of compatibility mechanisms, including shims for familiar commands from other systems, and an automatic import mechanism that takes existing configuration data from /etc/fstab, /etc/rc.conf{,.local}, /etc/ttys, and elsewhere, applying them to its native service definitions and creating additional native services. It is portable (including to Linux) and composable, it provides a migration path from the world of systemd Linux, and it does not require new kernel APIs. It provides clean service environments, orderings and dependencies between services, parallelized startup and shutdown (including fsck), strictly size-capped and autorotated logging, the service manager as a "subreaper", and uses kevent(2) for event-driven parallelism.
The past few months have seen a growth in the import mechanism, with full import of /etc/fstab and /etc/ttys available in version 1.18 in July, and importing PC-BSD Warden and &os; 9 jails, and full import of gbde and geli mount/unmount mechanisms in version 1.21 in October. It has also gained the ability to automatically re-generate host.conf and sysctl.conf whenever their source files change.
Other developments in the past few months include fully independent shutdown support, no longer relying upon an externally provided shutdown command from another toolset, and a full suite of binary packages. As of version 1.20, it became possible to have a fully-nosh-managed system, on both &os; and Linux, using just precompiled binary packages.
The biggest task remaining is one that was set a while ago: the creation of enough native service bundles and ancillary utilities to entirely supplant the rc.d system. A lot of this has been achieved, with the original target list of 157 items now down to just 39 remaining. These are the tricky ones, of course, where help is most needed.
There are still a few rc scripts left that should be easy to convert, such as /etc/rc.d/gptboot and /etc/rc.d/growfs as oneshot services, /etc/rc.d/routing, and /etc/rc.d/kldxref.
&os;'s /etc/rc.d/bluetooth is over 360 lines long. In 2011, Iain Hibbert wrote a "simpler" bluetooth for NetBSD. This can perhaps be used as a simpler basis for a nosh translation.
Add kernel support for passing a -b option to pid 1, and support for a boot_bare variable in the loader, to allow "emergency" (where even no shell dotfiles are loaded) and "rescue" mode bootstraps, akin to Linux. (History: The -b mechanism and idea date back to version 2.57d of Miquel van Smoorenburg's System 5 init clone, dated 1995-12-03, and was already known as "emergency boot" by 1997.)
Add support to &os;'s fsck(8) for outputting machine-readable progress reports to a designated file descriptor, so that nosh can provide progress bars for multiple fscks running in parallel. nosh already provides this functionality on Linux, where fsck(8) does provide machine-readable output.
Identify when the configuration import system needs to be triggered, such as when bsdconfig alters configuration files, and create the necessary hooks to import external configuration changes into nosh.
Investigate how &os;/PC-BSD could be improved by taking advantage of some available nosh package mechanisms.
PCI Express (PCIe) hot-plug is used on both laptops and servers to allow peripheral devices to be added or removed while the system is running. Laptops commonly include hot-pluggable PCIe as either an ExpressCard slot or a Thunderbolt interface. ExpressCard has built-in USB support that is already supported by &os;, but ExpressCard PCIe devices like Gigabit Ethernet adapters and eSATA cards are only supported when they are present at boot, and their removal may cause &os; to crash.
The goal of this project is to allow these devices to be inserted and removed while &os; is running. This work will provide the basic infrastructure to support adding and removing devices, though it is expected that additional work will be needed to update individual drivers to support hot-plug.
Current testing is focused on getting a simple UART device functional. Basic hot swap is currently functional.
A set of the patches for the work done in this project is now available on github.com.
Get suspend/resume functional by saving and restoring the necessary registers. This should be addressed by r281874.
Make sure that upon suspend, devices are removed so that we are not fooled if they are replaced with different devices while the machine is suspended.
Improve how state transitions are handled, possibly by using a proper state machine.
The automtud script will allow a &os; machine to send jumbo frames to machines that support them, while using normal-sized frames for other machines.
There are various advantages to using jumbo frames, such as reduced protocol overhead. It also means that TCP streams will not be segmented as much, although TSO helps mitigate the disadvantages of such segmentation. In cases where LRO does not work well, fewer packets will be received.
The script currently does not restore the system to its original state when it exits. This means that you must manually change the interface MTU and delete host routes after stopping the script.
Fix up various Ethernet drivers to better support jumbo frames. Most Ethernet drivers, though they support scatter/gather, use a physically contiguous zone to do so, which can cause resource shortages.
More testing is needed to ensure that things behave as expected. This means that when running the script, communication to all machines functions normally, without slowdown or connectivity issues. Check vmstat -z | grep mbuf to ensure that such issues are not due to running out of jumbo_9k or jumbo_16k buffers due to Ethernet driver issues.
The &os; bootloader has used Ficl 3 for quite some time. This project was intended to update the version of Ficl in use to Ficl 4. Ficl 4 is not only faster but also has a smaller memory footprint, both being important advantages for a bootloader.
As part of the Google Summer of Code program, I worked on importing the Ficl 4 sources to get a bootloader running Ficl 4. The first half of the summer consisted of setting up my test environment, as well as arranging the sources in the tree properly and modifying the build files to point to the new locations. Once that was complete, the sources had to be modified to build correctly and to add back in some of the &os;-specific parts from Ficl 3. Unfortunately, after all those tasks were completed, a few bugs in the Ficl project were discovered that delayed the bootloader update, so it is not finished. The Illumos project has faced similar issues with Ficl 4 so I received some good tips from them, but since school has started back up I have not been able to put much work into fixing the bugs.
&os; includes several programs that work with file system hierarchy descriptions in the mtree(5) format. These descriptions, also called specifications, have a broad range of uses, from automatically creating directory structures to security auditing.
Each of the programs, namely mtree, bsdtar, install, and makefs, has its own implementation of the mtree format. This not only adds maintenance overhead, but also makes interoperability difficult, as each of the implementations only supports a limited subset of the format.
The goal of this project was to create a new libmtree library, implementing everything the mtree format has to offer, and wrapping it with an expressive API which all the listed programs can use. We also wanted libmtree to be portable, as one of the major users of the mtree format is libarchive, the library implementing most of bsdtar.
Currently, the library is functionally complete, ready to be downloaded and receive everyone's attention. We have also decided to bundle the mtree program along with it. The bundled mtree has also been modified for better portability.
The project included modifying libarchive, install and makefs to use libmtree. These modified versions are also available.
Please see the Wiki page for more information, download locations, and an example of using the libmtree API.
Test and review the library code and API, and the modifications made to the programs.
Fix the known problems that are mentioned on the Wiki page.
ZFSguru started as a front-end to ZFS but has since grown into a multifunctional server appliance with its own unique features. While the project is still in early development, it already offers multiple unique features not found in other projects. Unlike similar projects, nothing is stripped away from the base operating system, meaning ZFSguru behaves as a normal &os; installation and thus is very versatile. The web-interface is designed to unite both novice and advanced users, providing both very easy to use basic functionality as well as features to be appreciated by more experienced users. The modular nature of the project combats the danger of bloat, whilst still allowing extended functionality to be easily deployed.
On the 15th of August, version 0.3 of ZFSguru was released. Some highlights of the new version:
Other progress made in the months July, August, and September:
The new Gnome 3 desktop does not work for everyone and still has issues. Anyone capable of diagnosing these issues can give the Gnome 3 LiveCD a try. Please see the linked forum thread for more information.
Several ports fail to compile with our own build infrastructure, and require bug reports in order to get them fixed upstream.
A 'State of the Project 2015' is due in Q4, providing an overview for future development of the project.