Index: head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml (revision 47981) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml (revision 47982) @@ -1,1026 +1,1104 @@ October-December 2015
Introduction

This is a draft of the October–December 2015 status report. Please check back after it is finalized, and an announcement email is sent to the &os;-Announce mailing list.

This report covers &os;-related projects between October and December 2015. This is the last of four reports planned for 2015.

The fourth quarter of 2015 was another productive quarter for the &os; project and community. [...]

Thanks to all the reporters for the excellent work!

The deadline for submissions covering the period from January to March 2016 is April 7, 2016.

?>
team &os; Team Reports proj Projects kern Kernel arch Architectures bin Userland Programs ports Ports doc Documentation misc Miscellaneous Linux Kernel as a Library Added to the Ports Collection Conrad Meyer cem@FreeBSD.org Upstream LKL Github repository

LKL ("Linux Kernel as a Library") is a special "architecture" of the full Linux kernel that builds as a userspace library on various platforms, including &os;. One application of such a library is using Linux's filesystem drivers to implement a FUSE backend.

fusefs-lkl's lklfuse binary is such a FUSE filesystem. It can mount ext4/3/2, XFS, and BTRFS read-write, using the native drivers from Linux.

The sysutils/fusefs-lkl port may now be installed from packages or ports, providing access to these filesystems on &os; via FUSE.

 Style(9) enhanced to allow C99 'bool' Bruce Evans brde@optusnet.com.au Conrad Meyer cem@FreeBSD.org Bruce's email requesting bool be added to style(9) Differential Revision for the change

Use of bool is now allowed. It was allowed previously, as well, but now it's really allowed. Party like it's 1999!

EMC / Isilon Storage Division

Specify style(9)'s opinion on iso646.h.

Fix intmax_t to be 128-bit on platforms where __int128_t is used.

 Sysctl enhancements Conrad Meyer cem@FreeBSD.org Ravi Pokala rpokala@FreeBSD.org Marcelo Araujo araujo@FreeBSD.org Wikipedia Entry on C99 Fixed-Width Integer Types sysctl(8) -t Submission PR

This quarter, support was added for fixed-width sysctls (signed and unsigned 8-bit, 16-bit, 32-bit, and 64-bit integers). The new KPIs are documented in the sysctl(9) manual page. The sysctl(8) command line tool supports all of the new types.

sysctl(8) gained the '-t' flag, which prints sysctl type information (the original patch was submitted by Yoshihiro Ota). This support includes the newly added fixed-width types.

EMC / Isilon Storage Division ioat(4) driver enhancements Conrad Meyer cem@FreeBSD.org Wikipedia on I/OAT Last quarter's ioat(4) report

I/OAT DMA engines are bulk memory operation offload engines built into some Intel Server/Storage platform CPUs.

This quarter, several enhancements were made to the driver. The driver now avoids memory allocation in locked paths, which should avoid deadlocking in memory pressure scenarios. Support for Broadwell-EP devices has been added. The "blockfill" operation and a non-contiguous 8 KB copy operation have been added to the API. The driver can recover from various programming errors by resetting the hardware.

EMC / Isilon Storage Division

XOR and other advanced ("RAID") operation support.

 ntb_hw(4)/if_ntb(4) driver synced up to Linux Conrad Meyer cem@FreeBSD.org Jon Mason's NTB wiki Intel NTB whitepaper

ntb_hw(4) is now up-to-date with the Linux NTB driver as of the work-in-progress 4.4 kernel (and actually, contains some fixes that haven't landed in the mainline Linux tree yet but will land in 4.5). Only Back-to-back ("B2B") configurations are supported at this time. Going forward, newer hardware may only support the B2B configuration.

if_ntb(4) is mostly up-to-date with the Linux NTB netdevice driver. Notably absent is support for changing the MTU at runtime.

EMC / Isilon Storage Division

Improving if_ntb(4) to avoid using the entire Base Address Register (BAR) when very large BAR sizes are configured (e.g., 512 GB).

Improving pmap_mapdev(9) to somehow allocate only superpage mappings for large BARs, on platforms that support superpages. (NTB BARs can be as large as 512 GB.)

 &os; on newer ARM boards John Wehle john@feith.com Ganbold Tsagaankhuu ganbold@FreeBSD.org &os; on Odroid-C1 Commit adding glue driver

This quarter, we made the changes necessary to support the Amlogic Meson Ethernet controller on the Hardkernel ODROID-C1 board which has an Amlogic aml8726-m8b SoC. The main effort needed was to write a glue driver for the Ethernet controller — the Amlogic Meson Ethernet controller is compatible with Synopsys DesignWare 10/100/1000 Ethernet MAC (if_dwc).

 Mellanox Drivers Hans Petter Selasky hselasky@FreeBSD.org Hardware Information Commit Adding the Driver

The Mellanox &os; team is proud to announce support for the ConnectX-4 series of network cards in &os; 11-current and &os; 10-stable. These devices deliver top performance, with up to 100GBit/s of raw transfer capacity, and support both Ethernet and Infiniband. Currently, the Ethernet driver is ready for use and the Infiniband support for ConnectX-4 is making good progress. We hope that it will be complete before &os; 11.0 is released. For more technical information, refer to the mlx5en(4) manual page in 11-current. The new driver for ConnectX-4 cards is called mlx5 and is put under /sys/dev and not under /sys/ofed as was done for the previous mlx4 driver. The mlx5en(4) kernel module is compiled by default in GENERIC kernels.

Mellanox Technologies "FreeBSD Mastery: Specialty Filesystems" early access version now available Michael Lucas mwlucas@michaelwlucas.com Book site Early access version

FreeBSD Mastery: Specialty Filesystems is now in copyediting. The ebook should be available by the end of January at all major vendors, and the print in February.

The book covers everything from removable media, to FUSE, NFSv4 ACLs, iSCSI, CIFS, and more.

If you act really quickly, you can get the electronic early access version at a 10% discount. You'll get the final ebook when it comes out as well. (This offer evaporates when the final version comes out.)

 Updates to GDB John Baldwin jhb@FreeBSD.org New 1:1-only thread target for &os;

The KGDB option is now on by default in the devel/gdb port.

The changes to support cross-debugging of crashdumps in libkvm were committed to HEAD in r291406.

A new thread target for &os; that is suitable for merging upstream has been written and lightly tested. However, it is not yet available as an option in the port. This thread target uses ptrace(2) directly rather than libthread_db and as such supports threads on all ABIs (such as &os;/i386 binaries on &os;/amd64 and possibly Linux binaries, though that is not yet tested). It also requires less-invasive changes in the MD targets in GDB compared to the libthread_db-based target.

Add a port option for the new 1:1-only thread target.

Test the new 1:1-only thread target.

Figure out why the powerpc kgdb targets are not able to unwind the stack past the initial frame.

Add support for more platforms (arm, mips, aarch64) to upstream gdb for both userland and kgdb.

Add support for debugging powerpc vector registers.

 iMX.6 video output support Oleksandr Tymoshenko gonzo@FreeBSD.org Commit Adding Basic Video Support

iMX.6 is a family of SoC used in multiple hobbyist ARM boards such as the Hummingboard, RIoTboard, and Cubox. Most of these products have HDMI output, but until recently, &os; did not benefit from it. As of r292574, there is basic video output support so you can use the console on iMX6-based boards and probably run Xorg (not yet tested).

Due to the lack of some kernel functionality (see open tasks), the only supported mode is 1024x768.

Proper pixel clock initialization (relies on a clock framework).

More flexible video output path (support multiple IPUs and DIs).

 Touchscreen support for Raspberry Pi and Beaglebone Black Oleksandr Tymoshenko gonzo@FreeBSD.org Beaglebone Black with 4DCAPE-43T Demo Input Stack Plans evdev Port

There are two working proof-of-concept drivers for the AM335x touchscreen and for the official Raspberry Pi's touchscreen LCD.

Proper touchscreen support would consist of a userland event reading API, a kernel event reporting API, and kernel hardware drivers for specific devices. There is an ongoing effort to port the Linux evdev API to &os; so applications that use libraries like libinput or tslib could be used without any major changes. Since it is not yet complete, I created a naive evdev-like API for both kernel and tslib and was able to run a demo on a Beaglebone Black with 4DCAPE-43T.

Once evdev makes it into the tree, both hardware drivers can be modified to include "report events" portions and committed.

 Kernel Vnode Cache Tuning Kirk McKusick mckusick@mckusick.com Bruce Evans bde@FreeBSD.org Konstantin Belousov kib@FreeBSD.org Peter Holm pho@FreeBSD.org Mateusz Guzik mjg@FreeBSD.org MFC to stable/10

This completed project includes changes to better manage the vnode freelist and to streamline the allocation and freeing of vnodes.

Vnode cache recycling was reworked to meet free and unused vnodes targets. Free vnodes are rarely completely free; rather, they are just ones that are cheap to recycle. Usually they are for files which have been stat'd but not read; these usually have inode and namecache data attached to them. The free vnode target is the preferred minimum size of a sub-cache consisting mostly of such files. The system balances the size of this sub-cache with its complement to try to prevent either from thrashing while the other is relatively inactive. The targets express a preference for the best balance.

"Above" this target there are 2 further targets (watermarks) related to the recyling of free vnodes. In the best-operating case, the cache is exactly full, the free list has size between vlowat and vhiwat above the free target, and recycling from the free list and normal use maintains this state. Sometimes the free list is below vlowat or even empty, but this state is even better for immediate use, provided the cache is not full. Otherwise, vnlru_proc() runs to reclaim enough vnodes (usually non-free ones) to reach one of these states. The watermarks are currently hard-coded as 4% and 9% of the available space. These, and the default of 25% for wantfreevnodes, are too large if the memory size is large. E.g., 9% of 75% of MAXVNODES is more than 566000 vnodes to reclaim whenever vnlru_proc() becomes active.

The vfs.vlru_alloc_cache_src sysctl is removed. New code frees namecache sources as the last chance to satisfy the highest watermark, instead of selecting source vnodes randomly. This provides good enough behaviour to keep vn_fullpath() working in most situations. Filesystem layouts with deep trees, where the removed knob was required, is thus handled automatically.

As the kernel allocates and frees vnodes, it fully initializes them on every allocation and fully releases them on every free. These are not trivial costs: it starts by zeroing a large structure, then initializes a mutex, a lock manager lock, an rw lock, four lists, and six pointers. Looking at vfs.vnodes_created, these operations are being done millions of times an hour on a busy machine.

As a performance optimization, this code update uses the uma_init and uma_fini routines to do these initializations and cleanups only as the vnodes enter and leave the vnode zone. With this change, the initializations are done kern.maxvnodes times at system startup, and then only rarely again. The frees are done only if the vnode zone shrinks, which never happens in practice. For those curious about the avoided work, look at the vnode_init() and vnode_fini() functions in sys/kern/vfs_subr.c to see the code that has been removed from the main vnode allocation/free path.

 Improvements to QLogic HBA driver Alexander Motin mav@FreeBSD.org

The QLogic HBA driver isp(4) received a substantial set of changes. Their primary goal was to make Fibre Channel target role work well with CTL, but many other things were also fixed/improved:

The code is committed to &os; head and stable/10 branches.

iXsystems, Inc.

NVRAM data reading is hackish and requires rework.

FCoE support for 26xx cards was not tested yet.

 Raspberry Pi: VideoCore userland application packaging Mikaël Urankar mikael.urankar@gmail.com Oleksandr Tymoshenko gonzo@FreeBSD.org

The Raspberry Pi SoC consists of two parts: ARM and GPU (VideoCore). Many interesting features like OpenGL, video playback, and HDMI controls are implemented on the VideoCore side and can be accessed from the OS through libraries provided by Broadcom (userland repo). These libraries were ported to &os; some time ago, so Mikaël created the port misc/raspberrypi-userland for them. He also created a port for omxplayer (a low-level video player that utilizes VideoCore APIs) and is working on a port for Kodi (ex-XBMC), a more user-firendly media player software with Raspberry Pi support.

 LXQt on &os; Olivier Duchateau olivierd@FreeBSD.org &os; LXQt Project LXQt Devel Repository

LXQt is the Qt port of and the upcoming version of LXDE, the Lightweight Desktop Environment. It is the product of the merge between the LXDE-Qt and the Razor-qt projects.

The porting effort remains heavily a work in progress: it needs some components of Plasma 5 (the new major KDE's workspace).

Currently, only the 0.10 branch is functional. See our wiki page for a complete list of applications.

We also sent updates for some components of LXDE (required for the LXQt desktop):

Binary packages are available (only for test purposes) which are regularly tested with the KDE development repository.

Port libsysstat to BSD systems.

Fix some issues that need to be resolved, especially the shutdown and reboot commands.

 Node.js Modules Olivier Duchateau olivierd@FreeBSD.org Node.js Modules Repository

Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient — perfect for data-intensive real-time applications that run across distributed devices.

The goal of this project is to make it easy to install the modules available in the npm package registry.

Currently, the repository contains slightly fewer than 300 new ports, in particular:

We have improved the USES framework:

Each port is up-to-date.

Update the pre-draft documentation.

Bring in grunt.js (and modules), the JavaScript task runner.

 Xfce on &os; &os; Xfce Team xfce@FreeBSD.org &os; Xfce Project &os; Xfce Repository

Xfce is a free software desktop environment for Unix and Unix-like platforms, such as &os;. It aims to be fast and lightweight, while still being visually appealing and easy to use.

During this quarter, the team has kept these applications up-to-date:

We also follow the unstable releases (available in our experimental repository) of:

Propose a patch to upstreamto fix Xfdashboard with our version of OpenGL (it currently coredumps).

 Supporting Variants in the Ports Framework Brendan Molloy brendan+freebsd@bbqsrc.net Poudriere PoC with Variants Ports Makefile PoC with Examples

I recently became involved with &os; (as in, the last 2-3 weeks), and found myself quickly involved with Ports development. What quickly struck me was the difficulty in providing a Python package that was depended upon by multiple versions of Python. As it turns out, poudriere can currently only generate one package per port, meaning that a Python version-neutral (compatible with 2.x and 3.x) port cannot simultaneously be packaged for each variant at the same time.

I discussed the issue with &a.koobs;, who suggested that I look into implementing a "variants protocol" within the Ports framework and the necessary changes to poudriere in order to allow a port to generate more than one package.

Support for variants is strongly needed in Ports and provides significant benefits.

For a simple example, editors/vim-lite could be folded into the editors/vim port, while still generating a vim and vim-lite package. For Python, VARIANTS can be derived from the already used USES flags and generate compatible packages. py27-foobar and py34-foobar could now be consistently generated by poudriere without issue.

Fortunately, this is not a wishful thinking piece. I dug in my heels and have implemented a proof-of-concept implementation of variants in the Ports framework, including the necessary modifications to poudriere in order to support it. It was mildly upsetting to find that poudriere is mostly written in Bourne shell scripts, but press on I did nonetheless.

I started with the prototype made by &a.bapt; as a base, and built from there. The poudriere PoC aims to limit changes as much as possible to merely adding support for the new variants flags, while also at the request of &a.koobs; making the logging output more package-centric (as opposed to port-centric) as a result of these changes.

This is a work in progress, and I would love to hear your feedback. I've enjoyed my first few weeks working on &os;, and I hope to stay here for quite some time.

Any constructive feedback on the implementation would be very welcome!

Hopefully the code will be of sufficient quality to be considered for formal review in the coming months.

 + + + New tools to enhance the porting experience + + + + + Brendan + Molloy + + brendan+freebsd@bbqsrc.net + + + + + pytoport: Generate &os; Ports from Python modules on PyPI + bandar: Create Development Overlays for the Ports tree + skog: Generate Visual Dependency Trees for &os; Ports + spdx-lookup: SPDX License List Query Tool + + + +

When I starting working on ports for &os; in the last + couple of weeks, I found that my workflow was not as efficient as + it could be, using just the available tools, so I made a few that + could be useful to the development community at large. All of + these have been added to the Ports tree, or otherwise will soon be + added, so you can play with them today!

+ +

pytoport is a command-line application that + generates a skeleton port for a given PyPI package name. It + attempts to generate the correct dependencies, makes a good + attempt at guessing the license using spdx-lookup, and + generates a pkg-descr. This made generating the fifteen + or so ports I was working on a complete breeze.

+ +

While doing this, however, I noticed that some ports were + bringing in dependencies that I did not expect, and I needed some + way to visualise this. skog builds a dependency tree + from the depends lists output by the Ports framework, and displays + it on the command line (with extra shiny output if you are using + UTF-8). No more pesky example and documentation dependencies + being dragged in when you clearly toggled that + OPTION as far off as it would go.

+ +

While doing all of this, I found it cumbersome to be + copying ports back and forth between my small development tree + living in git and the larger upstream SVN tree I was using in + poudriere. I built a tool called bandar that takes + advantage of the FUSE version of unionfs to easily overlay my dev + tree on the upstream tree, run linting, poudriere and generate + archives with ease.

+ +

I'm very impressed with how easy it was to build more + tooling for &os;. I hope some of these tools will be of some use + to you, and as always, I'd love to hear your feedback!

+ + + + +

Improve skog to support searching a tree for a certain + port.

+ + + +

Get the bandar port completed.

+ + + +

Continue to improve pytoport, adding trove support and better + depedency handling.

+ + + +

Patches welcome for all of the above!

+ + +