Index: head/en_US.ISO8859-1/htdocs/news/status/report-2017-04-2017-06.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2017-04-2017-06.xml (revision 50444) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2017-04-2017-06.xml (revision 50445) @@ -1,937 +1,1024 @@ April-June 2017
Introduction

This is a draft of the April–June 2017 status report. Please check back after it is finalized, and an announcement email is sent to the &os;-Announce mailing list.

This report covers &os;-related projects between April and June 2017. This is the second of four reports planned for 2017.

The second quarter of 2017 was another productive quarter for the &os; project and community. [...]

Thanks to all the reporters for the excellent work!

The deadline for submissions covering the period from July to September 2017 is October 7, 2017.

?>
team &os; Team Reports proj Projects kern Kernel arch Architectures bin Userland Programs ports Ports doc Documentation misc Miscellaneous 64-bit Inode Numbers Gleb Kurtsou gleb@FreeBSD.org Konstantin Belousov kib@FreeBSD.org Kirk McKusick mckusick@FreeBSD.org Phabricator Review

The 64-bit inode project was completed and merged into &os;  12 on May 23, 2017. It extends the ino_t, dev_t, and nlink_t types to be 64-bit integers. It modifies the struct dirent layout to add a d_off field, increases the size of d_fileno to 64 bits, increases the size of d_namlen to 16 bits, and changes the required alignment of the structure. It increases the struct statfs f_mntfromname[] and f_mntonname[] array lengths from MNAMELEN to 1024.

ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and employing various other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat as parameters are broken in backward- and forward-incompatible ways.

The ABI for kinfo sysctl MIBs is changed in a backward-compatible way, but there is no general mechanism to handle other sysctl MIBS which return structures where the layout has changed. It was considered that the breakage is either in management interfaces, where we usually allow ABI slippage, or is not important.

The layout of struct xvnode changed, and no compatibility shims are provided.

For struct xtty, the dev_t tty device member was reduced to be just uint32_t. It was decided that maintaining ABI compatability in this case is more useful than reporting a 64-bit dev_t value, for the sake of pstat.

Updating note: strictly follow the instructions in UPDATING. Build and install the new kernel with the COMPAT_FREEBSD11 option enabled, then reboot, and only then install the new world.

Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kris) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib).

The FreeBSD Foundation (emaste, kib) BSD Meetups at Rennes (France) Mathieu Kerjouan contact@steepath.eu First Event Second Event

Two meetups dedicated to BSD systems were held in Rennes, France. The first one was hosted in the OVH office in Rennes and included presentations on multiple subjects: the non-technical history of FreeNAS (presented by olivier@), how OVH is using ZFS, an introduction to jails, and a use case for BGP/bird on &os;.

The second meetup, also hosted in the OVH office, presented these subjects: how to create a &os; port (presented by jadawin@), how OVH is using Finite State Machines for managing their storage system, network high-availibility with &os;, and a jail tutorial by means of a demonstration running 200 OSFP (using net/bird) routers using jail and vnets on a small PC Engines APU2 system with only 4 CPU cores (1Ghz AMD) and 4GB RAM).

OVH New Port: FRRouting Olivier Cochard-Labbé olivier@cochard.me FRRouting Home Page

FRRouting (FRR), a Quagga fork, is an IP routing protocol suite for Linux and Unix platforms which includes protocol daemons for BGP, IS-IS, OSPF and RIP (LPD and PIM need to be fixed on &os;). FRR is a Linux Foundation Collaborative Project with contributors including 6WIND, Architecture Technology Corporation, Big Switch Networks, Cumulus Networks, LabN Consulting, NetDEF (OpenSourceRouting), Orange, Volta Networks, and other companies.

Orange The Postmaster Team David Wolfskill dhw@FreeBSD.org Larry Rosenman ler@FreeBSD.org Ryan Steinmetz zi@FreeBSD.org Eygene Ryabinkin rea@FreeBSD.org Remko Lodder remko@FreeBSD.org Kurt Jaeger pi@FreeBSD.org The Postmaster Team

Postmaster handles the mail flow for the &os; project.

Clusteradm provides us with four jails: mailman, mailarchive, mx1 and mx2. In addition, there is some part of the setup running on freefall.FreeBSD.org. The system uses postfix, mailman, spamassassin and some other tools from the ports tree to handle the mailflow. We use a very small, non-public Subversion repository for parts of the configuration.

During Q2, Larry Rosenman, Kurt Jaeger, Eygene Ryabinkin, Remko Lodder and Ryan Steinmetz joined the Postmaster Team, and Florian Smeets left the Postmaster Team.

Thanks to Florian for his long service in that role! David Wolfskill is planning to leave the role as soon as the new team members are settled. Vsevolod Stakhov plans to provide us with support to integrate rspamd into the setup.

The workload for the Postmaster Team is not high, but the complexity of the setup has its own demands.

We need to improve our internal documentation of workflows and processes. We should consider adding some monitoring to provide quarterly numbers on the mail flow. &os; Release Engineering Team &os; Release Engineering Team re@FreeBSD.org &os; 11.1-RELEASE Schedule &os; Development Snapshots

The &os; Release Engineering Team is responsible for setting and publishing release schedules for official project releases of &os;, announcing code freezes, and maintaining the respective branches, among other things.

The &os; 11.1-RELEASE cycle started on May 19, and is continuing as scheduled. &os; consumers are urged to test whenever possible to help ensure the reliability and stability of the upcoming second release from the stable/11 branch.

The &os; Foundation Using LLVM's LLD Linker as &os;'s System Linker Rafael Espíndola rafael.espindola@gmail.com Ed Maste emaste@FreeBSD.org &os; lld Wiki Page &os;/LLD Tracking PR (LLVM Bugzilla) Exp-Run Request Using lld as /usr/bin/ld

LLD is the linker in the LLVM family of projects. It is a high-performance linker that supports the ELF, COFF and Mach-O object formats. It is broadly compatible with the common linkers used for each file format. For ELF this is the GNU Binary File Descriptor (BFD) ld and GNU gold. However, LLD's authors are not constrained by strict compatibility where it would hamper performance or desired functionality.

LLD is now used as the default system linker for &os;/arm64 and can link a working kernel, modules, and userland for &os;/amd64. LLD can also link a working kernel and modules (but not userland) for &os;/arm and &os;/i386.

Work is ongoing to address ports that do not build with LLD as the system linker (either by fixing the port, or configuring the port to be linked by GNU ld).

For &os; 12.0 we expect to use LLD as the system linker for the same set of architectures that use Clang by default: 32- and 64-bit arm and x86.

The &os; Foundation Fix libtool to detect LLD and pass the same command line arguments as for GNU ld and gold. Investigate remaining amd64 and arm64 port build failures. Investigate and improve LLD on i386 and arm, before the creation of the stable/12 branch. Investigate and improve LLD on all other architectures. Extensive testing. DTC Emmanuel Vadot manu@FreeBSD.org

The in-tree DTC (Device Tree Compilator) was switched to use the BSD-licensed version by default. (The previous default DTC is licensed under the GPL.) The current version supports overlays and is able to compile every DTS used by the &os; arm releases. The ports GPL version was updated to the latest release (1.4.4). The in-tree GPL version is still present and the goal is to remove it before &os; 12.0.

 DTS Updates Emmanuel Vadot manu@FreeBSD.org

DTS (Device Tree Source) files provide a human-readable source description of the hardware resources for a given computer system (such as ARM- or MIPS-based embedded boards). The DTS source representation must be compiled into a binary format in order to be linked into the kernel and used to locate devices at runtime.

The DTS files in &os; were updated to match the versions from Linux 4.11, to represent more modern devices and provide more accurate representations.

 Updating Port Metadata for non-x86 Architectures Mark Linimon linimon@FreeBSD.org aarch64 Poudriere Machine armv6 Poudriere Machine

I have been analyzing the error logs from ports builds for all non-x86 architectures, including both the logs published on the package build cluster and also other builds of powerpc64 and sparc64.

From this analysis, I have marked almost all the failing ports as either BROKEN or NOT_FOR/ONLY_FOR, as appropriate.

The intent of this work is not to make life harder for anyone, but rather, in fact, the opposite. With these definitions in place, it is possible to scan the poudriere bulk build output (the "Ignored ports" portion, in particular) and see quickly what ports are failing to build and why. Previously, finding the exact reason why a build failed needed some research (portsmon only models failure messages on amd64). Additionally, it is extremely difficult to work through several hundred logs that simply say "failed to compile", "failed to link", and so forth.

This is part of an effort to identify where we need further work to bring sufficient Ports Collection support to, e.g., armv6 and aarch64 to bring them closer to true Tier-1 status.

To further facilitate locating patterns in the Poudriere output, I have begun reworking some existing BROKEN/NOT_FOR/ONLY_FOR messages so that they will sort more easily. This includes sorting the order of the ARCH definitions.

Many people have been doing great work on fixing the individual ports. I hope that my work makes their jobs somewhat easier.

 &os; Driver for the Annapurna Labs ENA Marcin Wojtas mw@semihalf.com Michał Krawczyk mk@semihalf.com Enhanced Networking Guide

ENA (Elastic Network Adapter) is a 25G SmartNIC developed by Annapurna Labs and is based on a custom ARMv8 chip. This is a high performance networking card available in AWS offering. It introduces enhancements in network utilization scalability on EC2 machines under control of various operating systems, in particular &os;.

The goal of &os; enablement is to provide top performance and a wide range of monitoring and management features such as:

The driver is now available in kernel source tree since r318647.

Annapurna Labs — an Amazon company Add RSS configuration from userspace (sysctls). Add support for LLQ mechanisms. Absolute &os;, 3rd Edition Michael Lucas mwlucas@michaelwlucas.com Status as of 30 June Second Edition Trivial Updates

I'm working on a third edition of Absolute &os;. This will be a nearly complete rewrite, thanks to the addition of little details like ZFS, GPT, dma, GELI, new boot stuff, disk labeling, pkg(8), blacklistd, jails, etc..

My current (delusional) plan is to have a first draft finished by the end of October 2017, so we can have print copies for BSDCan 2018.

Write the remaining 75% of the book. pNFS Werver Plan B Rick Macklem rmacklem@FreeBSD.org Testing Instructions

Parallel NFS (pNFS) is an extension to the NFSv4 protocol that allows for file accesses within a single logical mount to be performed against multiple file servers, with the potential for data access to occur in parallel. The pNFS "layout" in use specifies how the division occurs, with metadata operations occurring against the main server, and bulk data operations (read/write/setattr/etc.) occurring via a layout-specific scheme between the client and data servers.

My first attempt at a pNFS server using GlusterFS was a dud. It worked, but performance was so poor that it was not usable. This attempt that I call "Plan B", only uses &os;, with one &os; server handling the metadata operations and multiple &os; servers configured to serve data, is now ready for third party testing. If testing by third parties goes well, I anticipate the code will be merged into &os; head in time for &os; 12. Fairly recent &os; or Linux systems should be usable as pNFS clients for testing. This server supports the File Layout, which is supported by both of these clients.

There is no support for the Flex Files Layout or mirroring at this time. I hope to use the Flex Files Layout to add mirroring support over the next year or so. Striping is not supported, and I have no plans for implementing this at the moment.

The patched &os; sources may now be accessed for testing via either Subversion or download of a gzipped tarball. They consist of a patched kernel plus nfsd daemon and can be used on any &os; 11 or later system.

Testing by others will be needed, now that the code is available. New Xen Handbook Section Benedict Reuschling bcr@FreeBSD.org Handbook Section About &os; as a Xen Host Original Phabricator Review

&os; supports the Xen hypervisor, with DomU support since &os; 8.0 and Dom0 available since &os; 11.0. The &os; Handbook was lacking instructions on how to run a Xen host and VMs. The steps were outlined in the &os; wiki, but needed some extra bits of text from the upstream Xen wiki in order to form a complete guide. The new handbook section briefly explains what Xen is, how it differs from other hypervisors, and what features are currently available in &os;. It then goes on to describe how to set up the Dom0, as well as detailing the guest VM support known as DomU.

Reviewers Nikolai Lifanov, Roger Pau Monné, and Warren Block provided valuable feedback on the initial version in Phabricator. Additional corrections were found by Björn Heidotting while translating the section into German.

More options for the Dom0 and DomU could be provided. People should test these instructions on their hardware and provide feedback. This would also help us get better testing of the Xen port for &os;. Xfce on &os; &os; Xfce Team xfce@FreeBSD.org Olivier Duchateau duchateau.olivier@gmail.com &os; Xfce Project Ports Development Repository

Xfce is a free software desktop environment for Unix and Unix-like platforms such as &os;. It aims to be fast and lightweight, while still being visually appealing and easy to use.

During this quarter, we have kept these applications up-to-date:

We have created new Subversion tag (4.13) in order to follow the unstable releases (due to changes in USES= xfce infrastucture, and not backward compatible new API in xfconf). Ports following unstable release are:

Make the transition to Gtk3 smoother for end users. GNOME on &os; &os; GNOME Team FreeBSD-gnome@FreeBSD.org &os; GNOME Website Development Repository Upstream Build Bot USE_GNOME Porter's Handbook Chapter

The &os; GNOME Team maintains the GNOME, MATE, and CINNAMON desktop environments and graphical user interfaces for &os;. GNOME 3 is part of the GNU Project. MATE is a fork of the GNOME 2 desktop. CINNAMON is a desktop environment using GNOME 3 technologies but with a GNOME 2 look and feel.

After a period of not much activity, this quarter we started a little experiment in how we merge ports from the development repo to the &os; Ports Collection. Instead of merging everything in one big commit, we have been updating the GNOME ports one at a time or in small groups. For example, the GTK+ stack and the Evolution Suite were updated as groups, and all the gnome-games were done in one commit. It might be a bit more work preparing and testing the updates, but on the plus side, it easy to keep track of what is going on, and allows us to pay attention to the details. And it should be easier to commit smaller changes.

This quarter started with the update of GTK+ 3 to 3.22.15, and the underlying libraries to their latest stable versions. After the GTK+ update, work started on getting newer versions of other GNOME applications updated.

The webkit2-gtk3 port was first updated to the 2.14 series and later to 2.16.3, which is the latest stable version. This step was needed because 2.16 couldn't be built on &os; 10.3 without some required framework changes.

harfbuzz-icu was split off from the main harfbuzz port. This drops the heavy icu dependency from the main harfbuzz port.

A longstanding GLib/gio bug was fixed that had previously caused crashes of gnome-shell and other applications when share/applications was modified, as happens on pkg install or deinstall.

Many of these updates are based on work previous.y done in the Gnome development branch by Ruslan Makhmatkhanov, Gustau Perez and Koop Mast.

Porting of Mutter/Gnome-shell/GDM 3.24 is complete. Unfortunately, GDM is blocking the update because of a "handoff" bug to the session after login. Fix the control-center printer sub menu. As a workaround, system-config-printer can be used to configure printers. MATE 1.18 is being QA tested and should arrive in early July. + + + TensorFlow + + + + + Jov + + amutu@amutu.com + + + + + TensorFlow PR + Phabricator Review + Prebuilt Packages + TensorFlow Upstream + + + +

As described on its website, "TensorFlow is an open + source software library for numerical computation using data + flow graphs. Nodes in the graph represent mathematical + operations, while the graph edges represent the + multidimensional data arrays (tensors) communicated between + them. The flexible architecture allows you to deploy + computation to one or more CPUs or GPUs in a desktop, server, + or mobile device with a single API. TensorFlow was originally + developed by researchers and engineers working on the Google + Brain Team within Google's Machine Intelligence research + organization for the purposes of conducting machine learning + and deep neural networks research, but the system is general + enough to be applicable in a wide variety of other domains as + well."

+ +

TensorFlow now is the most popular platform/library for machine + learning and AI. There are official binaries for Linux, Mac, + Windows, and Android, but no official support for &os;. For + the last several months, I have done some work to make + TensorFlow available on &os;. Some notable work:

+ + + +

This port would not be possible without substantial + assistance from bapt@, lwhsu@, mat@, and koobs@ — thank + you for your help! You are very nice and I learned a lot + about &os; and the Ports framework from your help.

+ + + + Review, test, comment, and most importantly, commit to the + Ports Collection. + + Fix the OpenCL support on &os;. + + Port tensorflow-serving, which is a flexible, + high-performance serving system for machine learning models + produced by TensorFlow. + + Set up a CI for TensorFlow on &os; and give early notice + to upstream when they break TensorFlow on &os;. + +