Index: head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml (revision 53332) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml (revision 53333) @@ -1,2414 +1,2700 @@ April-June 2019
Introduction

This quarter our report includes some interesting topics easily accessible to anyone, even if you are not a programmer: we report the link to a presentation of the 2019 FreeBSD survey results at BSDCan 2019 and describe an interesting experience of a 3-person hackaton, which might encourage you to host one yourself, possibly with more participants. We also provide some up to date information about the status of our IRC channels.

For those who have some more technical skills, we give some news about the role of git in the FreeBSD project, describe the status of some tools to hunt bugs or enhance security and announce a clone of sysctl.

Finally, those who are more experienced with programming will probably be interested in the great work that has been done with drivers: in particular, an aknowledgement is due to Alan Somers for having started to bring up to date our FUSE implementation, which was about 11 years behind. Other important improvements include a more user-friendly experience with trackpoints and touchpads enabled by default, much low level work on graphics, many new bhyve features, updates to the linux compatibility layer, various kernel improvements.

Have a nice read!
-- Lorenzo Salvadore

team &os; Team Reports

Entries from the various official and semi-official teams, as found in the Administration Page.

proj Projects

Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.

arch Architectures

Updating platform-specific features and bringing in support for new hardware platforms.

.
third Third-Party Projects

Many projects build upon &os; or incorporate components of &os; into their project. As these projects may be of interest to the broader &os; community, we sometimes include brief updates submitted by these projects in our quarterly report. The &os; project makes no representation as to the accuracy or veracity of any claims in these submissions.

Release Engineering Team FreeBSD Release Engineering Team re@FreeBSD.org FreeBSD 11.3-RELEASE schedule FreeBSD 11.3-RELEASE announcement FreeBSD 12.1-RELEASE schedule FreeBSD development snapshots

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

During the second quarter of 2019, the FreeBSD Release Engineering team started the 11.3-RELEASE cycle, with the code slush starting May 3rd. Throughout the cycle, there were three BETA builds and three RC builds, all of which in line with the originally-published schedule. The final RC build started June 28th, with the final release build targeted for July 5th.

FreeBSD 11.3-RELEASE will be the fourth release from the stable/11 branch, building on the stability and reliability of 11.2-RELEASE.

The FreeBSD Release Engineering Team also published the schedule for the 12.1-RELEASE, targeted to start September 6th. One important thing to note regarding the published schedule is it excludes a hard freeze on the stable/12 branch, as a test run for eliminating code freezes entirely during a release cycle. Commits to what will be the releng/12.1 branch will still require explicit approval from the Release Engineering Team, however.

Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches.

Much of this work was sponsored by the FreeBSD Foundation and Rubicon Communications, LLC (Netgate).

Ports Collection René Ladan portmgr-secretary@FreeBSD.org FreeBSD Ports Management Team portmgr@FreeBSD.org About FreeBSD Ports Contributing to Ports FreeBSD Ports Monitoring Ports Management Team

The following was done during the last quarter by portmgr to keep things in the Ports Tree going:

During the last quarter the number of ports rose to just under 37,000. At the end of the quarter, there were 2146 open PRs and 7837 commits (excluding 499 on the quarterly branch) from 172 committers. This shows a slight decrease in activity compared to previous quarter.

People come and go, last quarter we welcomed Pedro Giffuni (pfg@), Piotr Kubaj (pkubaj@) and Hans Petter Selasky (hselasky@). Pedro and Hans Petter were already active as src committers. We said goodbye to gordon@, kan@, tobez@, and wosch@.

On the infrastructure side, a new USES=cabal was introduced and various default versions were updated: MySQL to 5.7, Python to 3.6, Ruby to 2.5, Samba to 4.8 and Julia gained a default version of 1.0. The web browsers were also updated: Firefox to 68.0 and Chromium to 75.0.3770.100

During the last quarter, antoine@ ran a total of 41 exp-runs to test various package updates, bump the stack protector level to "strong", switch the default Python version to 3.6 as opposed to 2.7, remove sys/dir.h from base which has been deprecated for over 20 years, and convert all Go ports to USES=go.

FreeBSD Core Team FreeBSD Core Team core@FreeBSD.org

The FreeBSD Core Team is the governing body of FreeBSD.

There is a variety of viewpoints within core regarding where and how to host a Git repository, however core feels that Git is the prudent path forward.

Continuous Integration Jenkins Admin jenkins-admin@FreeBSD.org Li-Wen Hsu lwhsu@FreeBSD.org FreeBSD Jenkins Instance FreeBSD CI artifact archive FreeBSD Jenkins wiki freebsd-testing Mailing List freebsd-ci Repository Tickets related to freebsd-testing@ Hosted CI wiki FreeBSD CI weekly report

The FreeBSD CI team maintains continuous integration system and related tasks for the FreeBSD project. The CI system regularly checks the committed changes can be successfully built, then performs various tests and analysis of the results. The results from build jobs are archived in an artifact server, for the further testing and debugging needs. The CI team members examine the failing builds and unstable tests, and work with the experts in that area to fix the code or adjust test infrastructure. The details are of these efforts are available in the weekly CI reports.

The FCP for CI policy is in "feedback" state, please provide any comments to freebsd-testing@ or other suitable lists.

We had a testing working group in 201905 DevSummit

Please see freebsd-testing@ related tickets for more information.

Work in progress:

FreeBSD Graphics Team status report FreeBSD Graphics Team x11@freebsd.org Niclas Zeising zeising@freebsd.org Project GitHub page

The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications.

In the last report, half a year ago, several updates and changes had been made to the FreeBSD graphics stack.

To further improve the user experience, and to improve input device handling, evdev was enabled in the default configuration in late 2018. Building on that, we have enabled IBM/Lenovo trackpoints and elantech and synaptics touchpads by default as well.

The input device library libinput has been updated as the last in a series of updates bringing the userland input stack up to date. This is work that was started in 2018.

We have made several improvements to the drm kernel drivers. A long-standing memory leak in the Intel (i915) driver has been fixed, and several other updates and improvements have been made to the various drm kernel driver components.

A port of the drm kernel drivers using the 5.0 Linux kernel sources has been created and committed to FreeBSD ports as graphics/drm-devel-kmod. This driver requires a recent Linux KPI and is only available on recent versions of FreeBSD CURRENT.

This version of the driver contains several development improvements. The generic drm (drm.ko) driver as well as the i915 (i915kms.ko) driver can now be unloaded and reloaded to ease in development and testing. This causes issues with the virtual consoles, however, so an SSH connection is recommended. To aid debugging i915kms.ko use of debugfs has been improved, but there are still limitations preventing it from being fully functional. Since debugfs is based on pseudofs it is possible that this will prevent a fully functional debugfs in its current state, so we might have to look into adding the required functionality to pseudofs or use another framework.

The new in-kernel drm driver for VirtualBox, vboxvideo.ko has been ported from Linux. Support is currently an experimental work in progress. For example the virtual console won't update after loading the driver, but X- and Wayland-based compositors are working.

Mesa has been updated to 18.3.2 and switched from using devel/llvm60 to use the Ports default version of llvm, currently devel/llvm80.

Several userland Xorg drivers, applications, and libraries have been updated, and other improvements to the various userland components that make up the Graphics Stack have been made.

We have also continued our regularly scheduled bi-weekly meetings, although work remains in sending out timely meeting minutes afterwards.

People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. We are also available in #freebsd-xorg on EFNet.

We also have a team area on GitHub where our work repositories can be found: https://github.com/FreeBSDDesktop

IRC Admin IRC Admin irc@FreeBSD.org

The FreeBSD IRC Admin team manages the FreeBSD Project's presence and activity on the freenode IRC network, looking after:

While the FreeBSD Project does not currently endorse IRC as an official support channel (see here and here), as it has not been able to guarantee a consistent or positive user experience, IRC Admin has been working toward creating a high quality experience, by standardising channel administration and moderation expectations, and ensuring the projects ability to manage all channels within its namespace.

In the last quarter, IRC Admin:

Upcoming changes:

Lastly, and to repeat a previous call, while the vast majority of the broader user community interacts on the freenode IRC network, the FreeBSD developer presence still needs to be significantly improved on freenode.

There are many opportunities to be had by increasing the amount and quality of interaction between FreeBSD users and developers, both in terms of developers keeping their finger on the pulse of the community and in encouraging and cultivating greater contributions to the Project over the long term.

It is critical to have a strong developer presence amongst users, and IRC Admin would like again to call on all developers to join the FreeBSD freenode channels to increase that presence.

Users are invited to /join #freebsd-irc on the freenode IRC network if they have questions, ideas, constructive criticism, and feedback on how the FreeBSD Project can improve the service and experience it provides to the community on IRC.

+ + FreeBSD Foundation + + + + Deb Goodkin + deb@FreeBSDFoundation.org + + + + +

The FreeBSD Foundation is a 501(c)(3) non-profit + organization dedicated to supporting and promoting + the FreeBSD Project and community worldwide. + Funding comes from individual and corporate + donations and is used to fund and manage software + development projects, conferences and developer + summits, and provide travel grants to FreeBSD + contributors. The Foundation purchases and + supports hardware to improve and maintain FreeBSD + infrastructure and provides resources to improve + security, quality assurance, and release + engineering efforts; publishes marketing material + to promote, educate, and advocate for the FreeBSD + Project; facilitates collaboration between + commercial vendors and FreeBSD developers; and + finally, represents the FreeBSD Project in + executing contracts, license agreements, and other + legal arrangements that require a recognized legal + entity.

+ +

Here are some highlights of what we did to help FreeBSD + last quarter:

+ +

We held our annual board meeting in Ottawa on May 14. + Board Director and Officer elections take place + each year at this meeting. Justin Gibbs was + elected as the new President of the Board of + Directors. The new FreeBSD Foundation Board of + Directors includes President and Founder Justin T. + Gibbs, Vice President Benedict Reuschling, + Secretary Philip Paeps, Treasurer Marshall Kirk + McKusick, and Directors Hiroki Sato, George + Neville-Neil and Robert N. M. Watson. You can read + more + about the elections.

+ +

After the elections, our management team gave updates to + the board on their respective areas. We then + discussed the key areas of the Project that need + help, and where we can step in to fill those + holes. We reviewed and updated our 12 month goals, + and identified projects we should support. We then + discussed conferences we are likely to attend, and + went over the latest on our fundraising efforts. + We followed that up with a discussion on how to + get more users to contribute back to the Project. + While discussing how to increase the number of + users and contributors, we talked about methods + for making for more training material available.

+ +

+ Partnerships and Commercial User Support

+

+ We help facilitate collaboration between commercial users + and FreeBSD developers. We also meet with + companies to discuss their needs and bring that + information back to the Project. In Q2, Ed Maste + and Deb Goodkin met with a few commercial users in + Germany. It’s not only beneficial for the above, + but it also helps us understand some of the + applications where FreeBSD is used. Because BSDCan + brings in a high number of commercial users, we + have an excellent opportunity to have similar + discussions about their needs during the four-day + FreeBSD Summit and BSDCan.

+ +

+ Fundraising Efforts

+

+ Our work is 100% funded by your donations. We are grateful + for the generous donations from Intel, NetApp, + VMware and Stormshield last quarter. We are + working hard to get more commercial users to give + back to help us continue our work supporting + FreeBSD. More importantly, we’d like to thank + our individual donors, for making $10-$1,000 + donations last quarter, for a total of $16,000!

+ +

Please consider making a donation to help us + continue and + increase our support for FreeBSD.

+ +

We also have the + Partnership + Program to provide more + benefits for our larger commercial donors. + Find out more information at the link and share with your + companies!

+ +

OS Improvements

+

+ The Foundation improves the FreeBSD operating system by + employing our technical staff to maintain and + improve critical kernel subsystems, add features + and functionality, and fix problems. The + Foundation also provides grants to fund individual + projects.

+ +

There were 243 commits to the FreeBSD base system + repository sponsored by the Foundation during the + quarter. These include improvements to the tmpfs + in-memory, MSDOS, and UFS filesystems, device + driver and hardware compatibility fixes, virtual + memory (VM), tool chain, documentation, and + testing and continuous integration improvements.

+ +

We fixed a number of race conditions and security issues + found by Syzkaller, Google’s + code-coverage-guided system call fuzzer.

+ +

Alan Somers’ work on updating FreeBSD’s support for + FUSE (userspace filesystems) continued during the + quarter; the full details are elsewhere in this + quarterly report. At this point most of the work + has been committed to the project branch but some + bug fixes and improvements have been committed + directly to the FreeBSD development branch.

+ +

Edward Napierala’s Linuxulator project continued through + the quarter, resulting in a number of improvements + to the Linuxulator and linux-specific + functionality such as linsysfs. This work is part + of the path to supporting the Linux strace + debugging tool in order to facilitate debugging + failures of other Linux binaries under the + Linuxulator. Mateusz Guzik continued with + scalability and performance improvements during + the quarter, and Bjoern Zeeb integrated the SDIO + stack (with details elsewhere in the quarterly + report).

+ +

Progress was made on the online RAID-Z expansion project + over the quarter. Matt Ahrens posted an alpha + preview of the feature for further + experimentation and review, and the FreeBSD + Foundation will make an alpha release image + available for testing in the near future.

+ +

Foundation staff contributed to nine FreeBSD security + advisories and errata updates over the quarter, + including CPU vulnerability workarounds. Related + work included improving Intel microcode update + loading.

+ +

Continuous Integration and Quality Assurance

+

+ The Foundation provides a full-time staff member who is + working on improving our automated testing, + continuous integration, and overall quality + assurance efforts.

+ +

During the second quarter of 2019, Foundation staff + continued to improve the project's CI + infrastructure, worked with contributors to fix + the failing build and test cases, and worked with + other teams in the Project for their testing + needs. We hosted a CI-focused working group at + BSDcan and continue to publish the CI weekly + report at freebsd-testing@ + mailing list.

+ +

See the FreeBSD CI section of this report for more + information.

+ +

+ Supporting FreeBSD Infrastructure

+

+ The Foundation provides hardware and support to improve + the FreeBSD infrastructure. Last quarter, we + continued supporting FreeBSD hardware located + around the world.

+ +

FreeBSD Advocacy and Education

+

+ A large part of our efforts are dedicated to advocating + for the Project. This includes promoting work + being done by others with FreeBSD; producing + advocacy literature to teach people about FreeBSD + and help make the path to starting using FreeBSD + or contributing to the Project easier; and + attending and getting other FreeBSD contributors + to volunteer to run FreeBSD events, staff FreeBSD + tables, and give FreeBSD presentations.

+ +

The FreeBSD Foundation sponsors many conferences, events, + and summits around the globe. These events can be + BSD-related, open source, or technology events + geared towards underrepresented groups. We support + the FreeBSD-focused events to help provide a venue + for sharing knowledge, to work together on + projects, and to facilitate collaboration between + developers and commercial users. This all helps + provide a healthy ecosystem. We support the + non-FreeBSD events to promote and raise awareness + of FreeBSD, to increase the use of FreeBSD in + different applications, and to recruit more + contributors to the Project.

+ +

Check out some of the advocacy and education work we did + last quarter:

+ + + +

+ We continued producing FreeBSD advocacy material to help + people promote FreeBSD around the world.

+ +

Read more about our conference adventures in the + conference recaps and trip reports in our + monthly + newsletters.

+ +

We help educate the world about FreeBSD by publishing the + professionally produced FreeBSD Journal. As we + mentioned previously, the FreeBSD Journal is now a + free publication. Find out more and access the + latest issues at + https://www.FreeBSDfoundation.org/journal/.

+ +

You can find out more about + events + we attended and upcoming events.

+ +

We have continued our work with a new website developer to + help us improve our website. Work has begun to + make it easier for community members to find + information more easily and to make the site more + efficient.

+ +

Legal/FreeBSD IP

+

+ The Foundation owns the FreeBSD trademarks, and it is our + responsibility to protect them. We also provide + legal support for the core team to investigate + questions that arise.

+ +

Go to http://www.FreeBSDfoundation.org to find out how we + support FreeBSD and how we can help you!

+ + +
+ bhyve - Live Migration Elena Mihailescu elenamihailescu22@gmail.com Darius Mihai dariusmihaim@gmail.com Mihai Carabas mihai@freebsd.org Github wiki - How to Live and Warm Migrate a bhyve guest Github - Warm Migration branch Github - Live Migration branch

The Migration feature uses the Save/Restore feature to migrate a bhyve guest from a FreeBSD host to another FreeBSD host. To migrate a bhyve guest, one needs to start an empty guest on the destination host from a shared guest image using the bhyve tool with the -R option followed by the source host IP and the port to listen to migration request. On the source host, the migration is started by executing the bhyvectl command with the --migrate or --migrate-live option, followed by the destination host IP and the port to send to the messages.

New features added:

Future tasks:

Matthew Grooms
bhyve - Save/Restore Elena Mihailescu elenamihailescu22@gmail.com Darius Mihai dariusmihaim@gmail.com Mihai Carabas mihai@freebsd.org Github repository for the snapshot feature for bhyve Github wiki - How to Save and Restore a bhyve guest Github wiki - Suspend/resume test matrix Phabricator review - bhyve Snapshot Save and Restore

The Save/Restore for bhyve feature is a suspend and resume facility added to the FreeBSD/amd64's hypervisor, bhyve. The bhyvectl tool is used to save the guest state in three files (a file for the guest memory, a file for the states of various devices and the state of the CPU, and another one for some metadata that is used in the restore process). To suspend a bhyve guest, the bhyvectl tool must be run with the --suspend <state_file_name> option followed by the guest name.

To restore a bhyve guest from a checkpoint, one simply has to add the -r option followed by the main state file (the same file that was given to the --suspend option for bhyvectl) when starting the VM.

New features added:

Future tasks:

Matthew Grooms
ENA FreeBSD Driver Update Michal Krawczyk mk@semihalf.com Maciej Bielski mba@semihalf.com Marcin Wojtas mw@semihalf.com ENA README

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

ENAv2 has been under development for FreeBSD, similar to Linux and DPDK. Since the last update internal review and improvements of the patches were done, followed by validation on various AWS instances.

Completed since the last update:

Todo:

Amazon.com Inc
FUSE Alan Somers asomers@FreeBSD.org

FUSE (File system in USErspace) allows a userspace program to implement a file system. It is widely used to support out-of-tree file systems like NTFS, as well as for exotic pseudo file systems like sshfs. FreeBSD's fuse driver was added as a GSoC project in 2012. Since that time, it has been largely neglected. The FUSE software is buggy and out-of-date. Our implementation is about 11 years behind.

During Q2 I nearly finished the FUSE overhaul that I begain in Q1. I raised the protocol level from 7.8 to 7.23, fixed many bugs (see 199934, 216391, 233783, 234581, 235773, 235774, 235775, 236226, 236231, 236236, 239291, 236329, 236379, 236381, 236405, 236327, 236466, 236472, 236473, 236474, 236530, 236557, 236560, 236647, 236844, 237052, 237181, 237588, and 238565), and added the following features:

I also added the following performance enhancements:

All that remains is to finish merging the branch, and deal with any newly introduced bugs.

The FreeBSD Foundation
Kernel ZLIB Update Yoshihiro Ota ota@j.email.ne.jp Review D19706

Kernel zlib upgrade is in progress.

Xin (delphij@) and I have been working closely for zlib upgrade. We relocated contrib/zlib to sys/contrib/zlib in order for kernel code to access zlib in the tree. We also deleted dead code that depended on zlib and inflate - inflate is a fork of unzip to uncompress gzip files. We also renamed crc.h to avoid conflicts with zlib/crc.h.

Next goal is to compile both old zlib and new zlib into the kernel allowing to switch each zlib user independently.

Linux compatibility layer update Edward Tomasz Napierala trasz@FreeBSD.org

The project aims to improve the Linux compatibility layer, to make it more compatible with recent Linux releases, and also to lower the bar for potential developers who want to start contributing to it.

The initial effort focused on tooling, to make it easier to debug problems and to prevent future regressions. The first part involved making it possible to use Linux strace(1) utility and providing it as linux-c7-strace package. The reason is that while FreeBSD truss(1) and ktrace(1) can trace Linux binaries, they cannot decode Linux-specific flags and structures.

The second part involved providing Linux Test Project binaries as linux-ltp package. There is ongoing work to hook it up to the FreeBSD CI infrastructure http://ci.FreeBSD.org.

There was also a number of improvements and fixes to bugs discovered in the process. One of them (not yet committed) fixes binaries linked against newer version of libc, effectively unbreaking binaries from recent Ubuntu releases.

FreeBSD Foundation
Lock-less delayed invalidation for amd64 pmap Konstantin Belousov kib@freebsd.org

The Virtual Memory machine-dependent layer (pmap) on amd64 needs to track all mappings for the managed physical memory pages, to be able to either destroy all of them (for page-out), or change them from writeable to read-only (e.g. to sync the page content to file, without racing with modifications through user writes). The mappings are accounted by creating pv_entry which records the address space (implicitly, by linking the pv entry to pmap) and the virtual address of the mapping.

Previous work split the lock protecting the pv entries lists from other VM locks into the pvh_global_lock lock, which was global for all address spaces. You can see it in i386 pmap.c still. Later, hashed per-page pv lists locks were introduced, which would reduce contention on pv lists maninulations for different pages, but unfortunately the pvh_global_lock was still needed to guarantee the safety of some operations.

Problem arises because amd64 pmap uses pmap lock to protect page tables and TLB consistency, which is per-pmap locks different from pv lists locks. When updating page table entry, we never drop pmap lock until the necessary TLB invalidation is done globally, including signalling other CPUs with IPI. But pv list locks can be unlocked before the necessary invalidation is done. So for instance when pmap is asked to remove all mappings of the specific page (pmap_remove_all(9)), it checks pv list of the page to find the mappings. The list might appear empty despite other CPUs TLB were not yet invalidated. If such page is reused, other CPUs might change its content using cached TLB entries. Allowing that means allowing both silent data corruption and opening security hole.

So the global pvh lock was held until all pmaps invalidated their TLBs. This mechanism has obvious scalability issues, and instead a generation-count based scheme for handling delayed invalidation (DI) was developed, where each thread that might remove entry from pv list acquired a generation number and marked the page with it, see pmap_delayed_invl_page(9). Then, on e.g. pmap_remove_all(9) or pmap_remove_write(9), pmap code waits for the maximum current thread's invalidation generation number to pass the page's generation, which guarantees that all required TLB invalidations are done.

Original implementation of DI allowed to get rid of pvh_global_lock, and only used a private mutex to handle sequential queueing of the coming and leaving threads, protecting a bounded region. A problem with that appeared e.g. in scalability benchmarks which did massive parallel unmaps, causing most of the threads to contend on DI queueing.

Current implementation of DI switched to lock-less queue algorithm using the approach proposed by T.L. Harris and relying on double-CAS to coalesce generation count and queueing. It uses ifuncs to select either previous locked DI or current lock-less implementation, only old AMD Athlons which did not implemented the CMPXCHG16B instruction falls to the locked implementation by default. Lock-less implementation still blocks the waiting thread on turnstile to avoid priority-inversion issues, but practically the wait occur very rare, typical parallel buildworld generates single-digit number of the events.

The patch got a lot of testing from Peter Holm, continuous reviews by Mark Johnston while I worked out bugs and live-lock problems in the implementation, and additional testing by Mateusz Guzik who helped to identify a priority inversion bug with the wait.

The FreeBSD Foundation
Mellanox Drivers Update Slava Shwartsman Hans Petter Selasky Konstantin Belousov freebsd-drivers@mellanox.com Mellanox OFED for FreeBSD Documentation

The mlx5 driver provides support for ConnectX-4 [Lx], ConnectX-5 [Ex] and ConnectX-6 [Dx] adapter cards. The mlx5en driver provides support for Ethernet adapter cards, whereas mlx5ib driver provides support for InfiniBand adapters and RDMA over Converged Ethernet (RoCE).

Following updates done in mlx5 drivers:

General updates:

Mellanox Technologies
NFSv4.2 client/server implementation for FreeBSD Rick Macklem rmacklem@freebsd.org current sources

NFSv4.2 is a newer minor version of NFSv4, made up of a set of optional operations/features. A majority of these operations are related to the POSIX operations posix_fadvise(2), posix_fallocate(2) and lseek(2)'s support for SEEKHOLE/SEEKDATA. There is also a Copy operation that allows a byte range of a file to be copied to another file locally on the NFS server, avoiding data transfer over the wire in both directions. FreeBSD-current now has a Linux compatible copy_file_range(2) syscall that will invoke this Copy operation on NFSv4.2 mounts. There is also support for MAC labelling, but it requires changes to the RPCSEC_GSS implementation to add V3 support and, as such, may not happen soon.

The implementation of NFSv4.2 (RFC-7862) is progressing nicely. At this time, the LayoutError, IOAdvise, Allocate and Copy operations have been implemented. There is still work to be done on Copy, to add asynchronous support, so that large copies do not result in a long delay for the RPC's reply.

The major operation that will be implemented next is Seek, so that lseek(SEEKHOLE/SEEKDATA) will work for the NFSv4.2 mounts.

It is hoped that this implementation will be ready for FreeBSD-current/head in time for the FreeBSD-13 release.

Testing is always appreciated and can be done by downloading the modified kernel from the svn repository in base/rojects/nfsv42 and then building and testing it on a couple of recent FreeBSD-current systems.

If anyone is conversant with Kerberos and wants to take on the challenge of adding RPCSEC_GSS_V3 support to the kernel RPC, a patch that does that would also be greatly appreciated.

NUMA awareness in the FreeBSD kernel Jeff Roberson jeff@FreeBSD.org Andrew Gallatin gallatin@FreeBSD.org Mark Johnston markj@FreeBSD.org

A set of patches to improve the state of NUMA awareness in the FreeBSD kernel are being developed and refined. This work also aims to generally improve the performance of FreeBSD's memory management subsystem on systems with many CPUs.

FreeBSD 12.0 featured a number of large changes which improve its performance on systems with a non-uniform memory architecture. That is, systems in which memory access latency for a given address varies depending on the CPU. Another round of improvements is being developed and will soon be available in FreeBSD-CURRENT. Short descriptions of some of these patches follow; a few have already been committed to FreeBSD-CURRENT.

In FreeBSD terminology, a memory page whose contents may not be evicted is referred to as "wired." Pages may be wired under different circumstances: for instance, all kernel memory is wired, and userland applications may request that ranges of memory be wired using the mlock(2) and mlockall(2) system calls. FreeBSD has historically defined a system-wide limit on the number of wired pages so as to avoid deadlocks that may arise when too much of a system's memory cannot be reclaimed to satisfy new memory allocations. This limit was applied only to userland wiring requests, but kernel wirings were counted against the limit, so a large source of kernel wirings could cause mlock(2) failures. This occurs frequently with a large ZFS ARC, for example. In FreeBSD-CURRENT this limit has been changed such that only userland wirings are counted against the limit; the kernel contains a number of mechanisms to apply back-pressure to kernel memory usage, so the use of a global limit on all wirings did not provide much benefit. This fixes a common problem on large ZFS systems, and helps enable some other architectural improvements to the code which manages page wirings.

FreeBSD has historically maintained two separate reference counters in the structure which describes a single physical page of memory. These counters initially had quite different properties, but have over time become more and more similar. Some work to merge the two counters has landed in FreeBSD-CURRENT. This does not have any user-visible effects, but it simplifies the page management code and removes a large amount of code which existed solely to transform references of one type to the other. Such code also made use of heavily contended locks, so the simplification improved kernel scalability for some workloads and has enabled further scalability improvements.

UMA is the slab allocator used in FreeBSD's kernel. It is the backend which services virtually all dynamic memory allocations performed in the kernel. The first round of NUMA improvements added NUMA awareness to the "keg" layer of UMA, which allocates and manages slabs. However, the frontend of UMA, which provides several layers of caching for objects, did not provide domain-aware caching, so over time the caches would become "polluted" with objects from different memory domains. However, this caching layer is being modified to ensure that objects from different memory domains are partitioned, helping ensure that consumers can perform domain-local allocations and frees efficiently. This will enable a global "first-touch" allocation policy for UMA-managed objects.

During boot, the FreeBSD kernel allocates a number of static data structures to track physical memory. These structures have historically lived in the lowest available range of physical memory, so they many not inhabit the same NUMA domain as the memory that they track. This is suboptimal when one tries to affinitize a workload to a particular NUMA domain: if while executing the workload the kernel frequently accesses page structures for local memory, and the page structures themselves are not placed in local memory, the kernel will perform many remote memory accesses. Some in-progress work for the amd64 platform creates multiple arrays of page tracking structures, one per NUMA domain, and ensures that each array is local to its domain. This complicates the task of initializing kernel data structures during boot, but can substantially reduce the amount of cross-domain communication that occurs while the kernel is performing useful work. Similarly, some patches to affinitize per-CPU structures are being developed; while most per-CPU memory allocations already return CPU-local memory, some structures allocated during boot are not yet properly placed with respect to the accessing CPU's memory domain.

Netflix
FreeBSD SDIO and Broadcom FullMAC WiFi Support Bjoern Zeeb bz@FreeBSD.ORG FreeBSD Wiki SDIO page

SDIO is an interface designed as an extension to SD Cards to allow attachments of various other peripherals, e.g., WiFi or Bluetooth.

Work has been ongoing by Ilya Bakulin on the MMCCAM stack to provide the infrastructure to be able to have SD cards and SDIO devices attached side-by-side facilitating FreeBSD's CAM framework. Based on this excellent work over the last years, SDIO support was finished earlier this year and committed to FreeBSD HEAD with the intention to merge to 12 at a later time.

Facilitating the newly available SDIO bus, work started to port Broadcom's FullMAC WiFi driver. This work is still in progress and expected to complete later this year. With this WiFi support for the Raspberry Pi and other embedded boards will become available. Likewise drivers for other SDIO devices can be developed now.

The FreeBSD Foundation
BIO_DELETE support for the swap pager Doug Moore dougm@FreeBSD.org Alan Cox alc@FreeBSD.org Mark Johnston markj@FreeBSD.org

An ongoing project aims to teach the swap pager to send SCSI UNMAP or ATA TRIM commands to the swap device when a block of swap space has been freed, for example when the application owning that block is exiting.

SSDs have become commonplace and feature low latency for random I/O requests. This makes them appealing for use as swap devices, since lower latencies mean that applications spend less time blocked while waiting for a page-in from the swap device. To maximize write performance, some SSDs require the operating system to send a notification to the disk when a sector is no longer in use; this helps the disk optimize their usage of NAND flash cells. In FreeBSD such a notification is called a BIO_DELETE.

FreeBSD's UFS and ZFS filesystems have for a long time been able to transmit BIO_DELETE requests to the devices backing the filesystem. For example, for UFS this support is enabled by specifying -t in newfs(8) or tunefs(8)'s parameters. However, FreeBSD has historically not had a corresponding implementation for swap devices.

Thanks to Doug Moore, as of r349286 in -CURRENT and r349930 in stable/12 swapon(8) can send BIO_DELETE to all blocks on the specified device immediately prior to configuring it as a swap device. This is enabled by specifying -E in the swapon(8) parameters, or by adding the "trimonce" option to the swap device's /etc/fstab entry. Some in-progress work on the swap pager implements online block deletion, in which BIO_DELETE is transmitted for blocks as they are freed by applications; this will hopefully be implemented in FreeBSD 13.0.

Fuzzing FreeBSD with syzkaller Mark Johnston markj@FreeBSD.org Andrew Turner andrew@FreeBSD.org Michael Tuexen tuexen@FreeBSD.org Ed Maste emaste@FreeBSD.org syzkaller

See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller.

syzkaller continues to find FreeBSD kernel bugs. A number of such bugs have been fixed in the past quarter, and we continue to investigate and fix bug reports from syzkaller. Work to extend syzkaller's capabilites has progressed: Andrew Turner has implemented support for fuzzing the 32-bit compatibility layer in amd64 kernels, helping illuminate some of the darker corners of the kernel, and it is now possible to use bhyve as a VM backend to syzkaller, so it is now efficient and convenient to fuzz FreeBSD on FreeBSD.

Some planned work includes: enabling the use of ZFS as the base filesystem for fuzzer VMs; extending the range of system calls and ioctls covered by syzkaller; enabling LLVM sanitizers in the kernel so as to catch more issues; and making use of netdump(4) to capture kernel dumps for panics found by syzkaller, making it much easier to diagnose bugs for which syzkaller was unable to find a reproducible test case.

The FreeBSD Foundation
Locking changes for vnodes during execve(2) Konstantin Belousov kib@freebsd.org

The execve(2) family of syscalls replaces the executing image in the current process. The file containing the program text, data, and arbitrary other pre-initialized segments for the newly activated image is usually called the text file. FreeBSD marks the text file as such, the mark is mutually exclusive with any opening of the file for write. In other words, file opened for write cannot be executed, and text file cannot be opened for write.

During the execve(2) syscall processing, kernel needs to lock the text file' vnode. This is done both to satisfy the VFS calls protocol, and to ensure that there is no incompatible parallel changes occuring to the text vnode. A vnode can be locked either in exclusive mode, which is mutually incompatible with any other lock acquisition, or in shared mode, which is only incompatible with exclusive requests, but allows other shared owners.

In principle, there is no reason why would execve(2) need an exclusive vnode lock, since it does not modify neither content nor metadata for the text vnode. The only exception is the marking of the vnode as text, which was done using VV_TEXT flag in v_vflag and protected by the vnode lock. Since we modify v_vflag, the vnode lock protecting the modification should be taken exclusive.

The end result is that execve(2)'s of the same file are serialized. For instance, if user runs parallel build, which executes more than one job for compiling, all invocation of the compiler are serialized during execve(2).

The count of opens for write is contained in other struct vnode member named v_writecount, which was protected by the vnode lock as well. Since text is mutually exclusive with an open for write, I reused v_writecount to indicate text references. Now, negative v_writecount counts the number of text references. The v_writecount content is literally protected by the vnode interlock, but normally all mutators also own vnode lock at least in the shared mode.

This way, we no longer need to acquire exclusive text vnode lock during execve(2), removing the serializing point. Additional positive effect is that we started to account the precise number of text references on the vnode. Before, we cleared VV_TEXT on the last unmap of the text vnode, potentially allowing obscure DoS where mapping the text file while it is executed prevented writes until the mapping is destroyed. Now we mark the mappings for text explicitly in the vm_map_entry and dereference v_writecount by +1 when such entry is unmapped.

The FreeBSD Foundation
Broadcom ARM64 SoC support Michal Stanek mst@semihalf.com Kornel Duleba mindal@semihalf.com Marcin Wojtas mw@semihalf.com

The Semihalf team continued working on FreeBSD support for the Broadcom BCM5871X SoC series

BCM5871X are quad-core 64-bit ARMv8 Cortex-A57 communication processors targeted for networking applications such as 10G routers, gateways, control plane processing and NAS.

Completed since the last update:

In progress:

Todo:

Juniper Networks, Inc
NXP ARM64 SoC support Marcin Wojtas mw@semihalf.com Artur Rojek ar@semihalf.com

The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC

LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with integrated packet processing acceleration and high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications.

Already completed:

In progress:

Todo:

Alstom Group
Aberdeen Hackathon

At BSDCam in Cambridge last year we had a discussion to create a template Hackathon in the same way we have a template for Devsummits. To test out the idea I was convinced (I swear tricked is the correct word) to host a Hackathon in Aberdeen.

As a project I think we benefit a lot from hackathons, but they do take a little organisation. The worst part of this is dealing with getting money from attendees so you can pay for events. I spoke with Deb Goodkin from the foundation at BSDCam and we arranged to use their new EventBrite based system to handle ticketing.

Overall this system made it straight forward for attendees to register and get me their details and requirements. After the event the expenses were then recouped from the foundation. This was much easier than me putting together a custom system or even setting up and using EventBrite myself.

The hackathon went well, you can read in Benedict and Kristof's reports that follow, but it was less well attended than I originally expected.

For hackers planning future hackathons remember to take heed of common national holidays (we could have planned the event to not land at Easter) and expect major geopolitical events to make things unpredictable (we knew Brexit would do something, but not when).

I need to thank the University of Aberdeen for providing the location for the Hackathon and to encourage you to run a hackathon where you are. The next one should be in your home town.

Benedict Reuschling

The hackathon in Aberdeen was happening in the week of Easter at the University of Aberdeen. Although only Kristof Provost (kp@) and myself joined our host Tom Jones, I still consider it a productive week for us. The overall theme of the hackathon was networking and each of us provided something towards that goal (be it PRs, submitting unfinished work, or other bits and pieces). We got together the night of Tuesday, April 16 over dinner and talked about what our plans were for the week.

Kristof and I had talked at AsiaBSDcon when I took his tutorial about Testing in FreeBSD that we should add a chapter about it in the developers handbook. We also used our first meeting to synchronize each other about the latest news in FreeBSD from our developers viewpoint.

The next day, we met up at the Frazer Noble building where the hackathon was taking place. It was one of the newer buildings on campus, nicely integrated into the older houses of the city. Since we were only a handful, we sat in Tom's office for the hackathon, which had plenty of room. He also showed us the room where we are supposed to be having the hackathon if we were more people and Tom gave us a little tour. Working in a university myself, I'm always interested in how other education organizations are structured and the rooms and equipment they provide for learning. Overall, my impression was that there is a good amount of space and equipment available, which we could have used in the hackathon.

After returning, we decided to use a special tag in the commits we would be doing to identify them as coming from this hackathon. We chose "Event:" for it as it is a general enough term to be used at other events like conferences, too. The "Sponsored by:" line we used in the past is more for companies or individuals sponsoring certain features, so I created a review to add this line to the committers guide.

Kristof had a couple of changes to the pf chapter in the FreeBSD handbook for me, so I started going through those. I created a review for him and the commit was made there and then, making use of the short feedback cycle. Originally, we thought about bringing in people via hangouts, but then resolved to contact people via our usual IRC channel if we needed their input.

Kristof and Tom worked on some network specific stuff, whereas I started work on creating an initial draft for the testing chapter. We would occasionally start talking about something and then return to our work in silence. If we needed to coordinate or had questions, we simply asked and could continue once we got our answer. This provided a nice atmosphere to work in. I tackled some doc PRs while Kristof found a bug in pf and fixed it.

The afternoons were spent at different locations within walking distance. Tom made sure we got a good impression on how it is to be a student and that there is both taste and variety of food available. In the evenings, Tom drove us into town to have dinner at various restaurants over the week.

Aberdeen has a lot to offer as a city. Starting from the second day, Kristof and I would meet up at my hotel, which was close to the Aberdeen beach and walk along it to the University. According to Tom, it is possible to see Dolphins when the weather is right and the gulf stream provides the city with enough warmth that the winters aren't as bad as you'd think this far up north.

Tom also gave us a tour of the zoological department of the university, which offered a beautiful garden with various plants and trees, as well as a museum with zoological specimen. This offered a great spot for photographs and to unwind a bit from the technical discussions we've had. Tom also had t-shirts made for the event, which are already rare collectors items.

I had to return on Sunday, so Tom took us on a tour of the Scottish highlands in his car the day before. We stopped at a couple of places to take pictures and Tom would explain at lot to us having lived there all his life. We came to Stonehaven and had fish and chips there from a take-out restaurant that had a lot of awards for sustainable fishing. This was certainly a highlight for the week and even then, we couldn't stop talking about FreeBSD and networking.

Although more people would maybe have produced more output, the three of us were certainly productive as a small group. It also made planning and coordination easier and more flexible. Tom Jones had done a lot of preparation and was an excellent guide. I would encourage him to host another such hackathon in the future and hope that next time, more people will take a trip to Aberdeen to spend some time hacking on FreeBSD

Kristof Provost

While I’d been to Scotland before I’d never seen Aberdeen. It’s a charming city, and I enthusiastically recommend visiting.

I arrived a little while after Benedict, but made it to my hotel easily, and turned up in time to join Benedict and Tom for dinner.

Despite being small (or perhaps because of it), the hackathon was remarkably productive. Benedict and I went through the pf documentation in the handbook, so that Benedict could rework and improve it. (Benedict’s doing the work, but I’m going to take credit anyway.)

Tom and I looked at the GSoC proposals and tried to find potential mentors for two promising proposals. Both of us are candidate mentors as well. We should know soon if our students are awarded slots.

Tom also proposed a patch to eliminate RFC 2675 IPv6 Jumbograms. It has my enthusiastic support.

I managed to look at a couple of open pf issues:

On Saturday Tom took us out to discover some of the pretty bits of Scotland. It turns out there are a lot of them. I can’t really do it justice, but Tom has a promising career at the Scottish tourism board when this computers fad blows over.

On my way home I passed through Oslo, and took the opportunity to meet with (have lunch with) two of the EuroBSDCon local organisers. EuroBSDCon is filling up fast, make sure to register now to secure your place!

Bring more Security Intelligence to FreeBSD Michael Muenz m.muenz@gmail.com Maltrail - distributed Malware detection Wazuh - thread detection and incident response

To bring more Security Intelligence we maintain the FreeBSD port of zmaltail. This open source project based on Python can act as a sensor and/or as a central server. It listens in defined ports or protocols and compares IP addresses and domains against static and dynamic feeds, contributed by the community.

As you can install this piece of software on multiple firewalls and let them send to a central server, you are able to detect attacks and compromises very fast. Within Q2 we updated the port to the latest version and are constantly in contact with the core developer (also co-author of SQLmap) to bring out new features.

The second project we are currently trying to add as a port is Wazuh. Wazuh is a fork of Ossec which is already in the ports tree. Compared to Ossec, Wazuh has some intelligent addition like full ELK-Stack integration with own apps and dashboards.

With Wazuh installed on your webserver, or even on your windows desktop you can monitor file integrity or log files for most kind of attacks. Active response features let you e.g. send API calls to your firewalls to dynamically block this offender.

As Wazuh offers a complete ELK-Stack you can use it also as a central logging solution for better security insights into your network.

m.a.x. Informationstechnologie AG
nsysctl 1.0 Alfonso Sabato Siciliano alfonso.siciliano@email.com gitlab.com/alfix/nsysctl sysutils/nsysctl port Tutorial

The nsysctl utility is a /sbin/sysctl clone, to get or set the kernel state, supporting libxo and extra options.

nsysctl [--libxo=opts [-r tagname]]
[-DdFGgIilmNpqTt[V|v[h[b|o|x]]]Wy]
[-e sep] [-B <bufsize>] [-f filename]
name[=value[,value]] ...
nsysctl [--libxo=opts [-r tagname]]
[-DdFGgIlmNpqTt[V|v[h[b|o|x]]]Wy]
[-e sep] [-B <bufsize>] -A|a|X

You could use nsysctl to explore the sysctl MIB showing the value and the info of an object. The output is explicitly indicated by the options and is printed via libxo in human and machine readable formats, moreover some value is parsed to display it in a structured mode (e.g., vm.phys_free). The support for efi_map_header was added but it is untested, someone could help by trying it via machdep.efi_map.

Please refer to the tutorial for a more thorough description.

libvdsk - QCOW2 implementation Sergiu Weisz sergiu121@gmail.com Marcel Molenaar marcel@freebsd.org Marcelo Araujo araujo@freebsd.org Mihai Carabas mihai@freebsd.org Github - libvdsk repo

Add support for using QCOW in bhyve using the libvdsk library. Libvdsk was used to substitute the regular disk operations from bhyve with a call to libvdsk which will in turn call the disk-specific handler for the operation.

To use this feature one has to install the libvdsk-enabled bhyve version along with libvdsk from the libvdsk repo linked above.

New features added:

Future tasks:

Matthew Grooms