Section Navigation

Introduction

This report will be covering FreeBSD related projects between April and June, and covers a diverse set of topics ranging from kernel updates over userland and ports, as well as third-party work.

Some highlights picked with the roll of a d100 include, but are not limited to, the ability to forcibly unmount UFS when the underlying media becomes inaccessible, added preliminary support for Bluetooth Low Energy, a introduction to the FreeBSD Office Hours, and a repository of software collections called potluck to be installed with the pot utility, as well as many many more things.

As a little treat, readers can also get a rare report from the quarterly team.

Finally, on behalf of the quarterly team, I would like to extend my deepest appreciation and thank you to salvadore@, who decided to take down his shingle. His contributions to not just the quarterly reports themselves, but also the surrounding tooling to many-fold ease the work, are immeasurable.

We hope you find the report as interesting as we have,
Daniel Ebdrup Jensen (debdrup@), on behalf of the quarterly team.

FreeBSD Team Reports

Entries from the various official and semi-official teams, as found in the Administration Page.

FreeBSD Foundation

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and release engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Here are some highlights of what we did to help FreeBSD last quarter:

COVID-19 Impact on the Foundation

Like other organizations, we put policies in place for all of our staff members to work from home. We also put a temporary ban on travel for staff members. We are continuing our work supporting the community and Project, but some of our work and responses may be delayed because of changes in some of our priorities and the impact of limited childcare for a few of our staff members.

Partnerships and Commercial User Support

We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Not surprisingly, the stay at home orders, combined with our company ban on travel during Q2 made in-person meetings non-existent. However, the team was able to continue meeting with our partners and commercial users virtually. These meetings help us understand some of the applications where FreeBSD is used.

Fundraising Efforts

Last quarter we raised $268,400! Thank you to the individuals and organizations that stepped in to help fund our efforts. We’d like to thank Netflix, employees of Nginx, Beckhoff Automation, and Mozilla Foundation for their large contributions last quarter, which helped bring our 2020 fundraising effort to $339k. We hope other organizations will follow their lead and give back to help us continue supporting FreeBSD.

These are trying times, and we deeply appreciate every donation that has come in from $5 to $150,000. We’re still here giving 110% to supporting FreeBSD!

We are 100% funded by donations, and those funds go towards software development work to improve FreeBSD, FreeBSD advocacy around the world, keeping FreeBSD secure, continuous integration improvements, sponsoring BSD-related and computing conferences (even the virtual events!), legal support for the Project, and many other areas.

Please consider making a donation to help us continue and increase our support for FreeBSD.

We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information about the partnership program and share with your companies!

OS Improvements

A number of FreeBSD Foundation grant recipients started, continued working on, or completed projects during the second quarter. These include:

WiFi improvements
Linuxulator application compatibility
DRM / Graphics driver updates
Zstd compression for OpenZFS
Online RAID-Z expansion
if_bridge performance improvements

You can find more details about most of these projects in other quarterly reports.

Staff members also worked on a number of larger projects, including:

Run-Time Dynamic Linker (rtld) improvements
Improved FreeBSD support on Microsoft HyperV and Azure
Fine-grained locking for amd64 pmap
5-level paging structures for amd64
Non-transparent superpages
Migration to a Git repository
Tool chain modernization

Many of these projects also have detailed entries in other quarterly report entries.

Staff members also put in significant effort in many ways other than larger, individual projects. These include assisting with code reviews, bug report triage, security report triage and advisory handling, addressing syzkaller reports, and ongoing maintenance and bug fixes in functional areas such as the tool chain, developer tools, virtual memory kernel subsystem, low-level x86 infrastructure, sockets and protocols, and others.

University of Waterloo Co-op

Foundation co-op students Colin, Tiger, and Yang completed their winter 2020 work term during the second quarter, and continued on with the next school term in their respective programs. Although COVID-19 presented a unique challenge and prompted an abrupt transition to remote work just over half way through the term, all three learned a lot and provided positive contributions to the FreeBSD Project and to the Foundation.

A few projects that were in progress or completed during the work term were committed to the FreeBSD tree in the second quarter.

Continuous Integration and Quality Assurance

The Foundation provides a full-time staff member who is working on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD project.

During the second quarter of 2020, Foundation staff continued improving the Project's CI infrastructure, monitoring regressions and working with contributors to fix the failing build and test cases. The setting up of VM host for CI jobs and staging environment is in progress. We are also working with other teams in the Project for their testing needs. For example, we added jobs for running full tests on non-x86 architectures. We are also working with many external projects and companies to improve their support of FreeBSD.

See the FreeBSD CI section of this report for completed work items and detailed information.

Supporting FreeBSD Infrastructure

The Foundation provides hardware and support to improve FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. We started working on getting the new NYI Chicago colocation facility prepared for some of the new FreeBSD hardware we are planning on purchasing. NYI generously provides this for free to the Project.

FreeBSD Advocacy and Education

A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations.

The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. As is the case for most of us in this industry, COVID-19 has put our in-person events on hold. In addition to attending virtual events, we are continually working on new training initiatives and updating our selection of how-to guides to facilitate getting more folks to try out FreeBSD.

Check out some of the advocacy and education work we did last quarter:

Silver sponsor of BSDCan 2020. The event was held virtually, June 2-6, 2020
Community Sponsor of Rootconf 2020. The event was held virtually, June 19-20, 2020
Annual FreeBSD Day, June 19. This year’s celebration was postponed in support of Juneteeth. However the activities surrounding FreeBSD Day have been transformed into an ongoing series of online sessions. See FreeBSD Fridays below for more information.
Presented 27 Years of FreeBSD and Why You Should Get Involved as part of a Linux Professional Institute series of webinars on June 24, 2020.
Attended and presented at the virtual Open Source Summit 2020.
Announced FreeBSD Fridays: A series of 101 classes designed to get you started with FreeBSD. Find out more in the announcement
Participated as an Admin for Google Summer of Code 2020
Participated in the new FreeBSD Office Hours series including holding our own Foundation led office hours. Videos from the one hour sessions can be found on the Project’s YouTube Channel. You can watch ours here.

In addition to the information found in the Development Projects update section of this report, take a minute to check out the latest update blogs:

Keep up to date with our latest work in our Bi-Monthly newsletters.

Mellanox provided an update on how and why they use FreeBSD in our latest Contributor Case Study.

We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues on the Journal site.

You can find out more about events we attended and upcoming events.

We have continued our work with a new website developer to help us improve our website. Work is nearly complete to make it easier for community members to find information more easily and to make the site more efficient. We look forward to unveiling the refreshed site in Q3.

Foundation Board Meeting

Our annual board meeting was held on Tuesday June 2, 2020. We normally hold this meeting the Tuesday before BSDCan, in Ottawa, Ontario, Canada, but with the company travel ban, and the conference going virtual, our meeting went virtual for the first time. The purpose of the annual board meeting is to hold our board director and officer elections, review work accomplished over the past year, and put together strategic goals for the upcoming 12 months.

The board generally has two all-day board meetings each year, this one, and a more informal one in January, typically held in Berkeley. Both meetings allow us to connect, reevaluate and discuss new ideas, while assessing what we should do to help the Project.

Some of our longer-term goals include Growing User and Developer Communities, Developing Training and OS Course Content, Improving desktop/laptop experience, Promoting FreeBSD (as you can see in all the advocacy work listed above), and Improving Testing Capabilities.

Results of the director and officer elections were:

Justin Gibbs (President)
Benedict Reuschling (Vice President)
Kirk McKusick (Treasurer)
Philip Paeps (Secretary)
Deb Goodkin (Assistant Secretary)
Robert Watson (Director)
Hiroki Sato (Director)
George Neville-Neil (Director)

Find out more about the FreeBSD Foundation Board of Directors on our website.

Legal/FreeBSD IP

The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise.

Go to the FreeBSD Foundation's web site to find out how we support FreeBSD and how we can help you!

FreeBSD Core Team

Contact: FreeBSD Core Team <core@FreeBSD.org>

The FreeBSD Core Team is the governing body of FreeBSD.

The Core Team held 10 meetings during the second quarter of 2020, including a 2020-05-21 joint meeting with members of the FreeBSD Foundation. Here are some highlights from that meeting:

Deb requested guidance on how the Foundation can support the community. Core and Foundation members believe that more developer support is necessary to fill gaps in areas where commercial customers do not provide backing. The clearest example of such a gap is the desktop experience, including graphics and wireless support. What makes this request different from past requests is that rather than support for one-time projects, ongoing positions are necessary for a consistently high-quality desktop experience.

"FreeBSD not being able to run on your laptop is the first step to irrelevance." Ed Maste
Both teams discussed topics for upcoming sessions of FreeBSD Office Hours, informal FreeBSD video conferences that anyone can attend. Everyone agreed that the Office Hours have been a useful way for different parts of the Project to engage with each other and with the wider community. Kudos to Allan Jude for initiating the Office Hours and for everyone who has helped make them a success by hosting or attending sessions.
Both teams agreed that they should meet once per quarter.

The second annual community survey closed on 2020-06-16. The purpose of the survey is to collect data from the public to help guide the Project's efforts and priorities. As an example, last year's survey results helped initiate the Project's conversion to Git. Thank you to all who took the time to respond. The results will be released soon.

The Core-initiated Git Working Group continued to make progress, but there are still some remaining issues to be worked out with the translation from Subversion. Hopefully the new Git src repository will be ready for use this summer. A beta version has been published for people to test and a preliminary version of a Using Git for FreeBSD Development primer will soon be ready to share. Core, the Git Working Group, and Release Engineering are working towards the goal of releasing 12.2 from the new Git repository.

Following the results of a Core-initiated developer survey, The FreeBSD Project has adopted a new LLVM-derived [code of conduct](https://www.freebsd.org/internal/code-of-conduct.html).

The eleventh FreeBSD Core Team was elected by active developers. From a pool of 23, the 9 successful candidates for core.11 are:

Sean Chittenden (seanc, incumbent)
Baptiste Daroussin (bapt)
Kyle Evans (kevans)
Mark Johnston (markj)
Scott Long (scottl)
Warner Losh (imp, incumbent)
Ed Maste (emaste)
George V. Neville-Neil (gnn)
Hiroki Sato (hrs, incumbent)

A new Core Team secretary, Muhammad Moinur Rahman (bofh), was unanimously approved by core.11. The outgoing core team met three times with the new core team to help with the transition. Core.10 wishes core.11 a successful term.

FreeBSD Release Engineering Team

Links
FreeBSD 11.4-RELEASE announcement	URL: https://www.freebsd.org/releases/11.4R/announce.html
FreeBSD 11.4-RELEASE schedule	URL: https://www.freebsd.org/releases/11.4R/schedule.html
FreeBSD development snapshots	URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code slushes, and maintaining the respective branches, among other things.

During the second quarter of 2020, the Release Engineering Team started work on the 11.4-RELEASE cycle, the fifth release from the stable/11 branch. The release cycle went quite smoothly, with both BETA3 and RC3 removed from the schedule. This allowed the final release to occur one week earlier than originally scheduled, which was announced June 16. FreeBSD 11.4-RELEASE is expected to be the final 11.x release.

The FreeBSD Release Engineering Team would like to thank everyone involved in this cycle for their hard work.

Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches.

Much of this work was sponsored by Rubicon Communications, LLC (netgate.com) and the FreeBSD Foundation.

Cluster Administration Team

Links
Cluster Administration Team members	URL: https://www.freebsd.org/administration.html#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following:

Upgrade all x86 ref- and universe-machines
Setup Amsterdam (PKT) mirror
Solve hardware issue for bugzilla and svnweb backend
Setup public beta git server
Decommission CyberOne Data (CYB) mirror
Ongoing systems administration work:
- Accounts management for committers.
- Backups of critical infrastructure.
- Keeping up with security updates in 3rd party software.

Work in progress:

Setup Malaysia (KUL) mirror
Setup Brazil (BRA) mirror
Review the service jails and service administrators operation.
Infrastructure of building aarch64 and powerpc64 packages
- NVME issues on PowerPC64 Power9 blocking dual socket machine from being used as pkg builder.
- Drive upgrade test for pkg builders (SSDs) courtesy of the FreeBSD Foundation.
- Boot issues with Aarch64 reference machines.
New NYI.net sponsored colocation space in Chicago-land area.
Work with git working group
Check new hardware requirement from other teams
Searching for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror

Continuous Integration

Links
FreeBSD Jenkins Instance	URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab	URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive	URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report	URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki	URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki	URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI	URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@	URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository	URL: https://github.com/freebsd/freebsd-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system for the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from the build jobs are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure. The details of these efforts are available in the weekly CI reports.

During the second quarter of 2020, we continue working with the contributors and developers in the project for their testing needs and also keep working with external projects and companies to improve their support of FreeBSD.

Important changes:

All -test jobs will run tests under /usr/tests, previously only x86 architectures were doing this. See the Continuous Integration on !x86 section in this report for more information.
Compression algorithm of disk images on the artifact server has been changed to zstd to speed up compression and decompression.
The build and test results will be sent to the dev-ci mailing list soon. Feedback and help with analysis is very appreciated!

New jobs added:

https://ci.freebsd.org/job/FreeBSD-head-armv7-test/
https://ci.freebsd.org/job/FreeBSD-head-aarch64-test/
https://ci.freebsd.org/job/FreeBSD-head-mips64-test/
https://ci.freebsd.org/job/FreeBSD-head-powerpc64-test/

Work in progress:

Collecting and sorting CI tasks and ideas here
Testing and merging pull requests in the the FreeBSD-ci repo
Setting up a builder dedicated to run jobs using provisioned VMs.
Setting up the CI stage environment and putting the experimental jobs on it
Implementing automatic tests on bare metal hardware
Adding drm ports building tests against -CURRENT
Planning to run ztest and network stack tests
Adding external toolchain related jobs
Improving the hardware lab to be more mature and adding more hardware
Helping more 3rd software get CI on FreeBSD through a hosted CI solution
Working with hosted CI providers to have better FreeBSD support

Please see freebsd-testing@ related tickets for more WIP information, and don't hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Ports Collection

Links
About FreeBSD Ports	URL: https://www.FreeBSD.org/ports/
Contributing to Ports	URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring	URL: http://portsmon.freebsd.org/index.html
Ports Management Team	URL: https://www.freebsd.org/portmgr/index.html

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

There are currently 2,373 open ports PRs of which 526 are unassigned, for a total of 39,628 ports. In the last quarter there were 10,315 commits to HEAD and 476 to the quarterly branch by respectively 178 and 65 committers. Compared to the quarter before, this means a significant increase in commits and also a slight decrease in open PRs.

During the last quarter, we welcomed Hiroki Tagato (tagattie@). We said goodbye to seanc@, zleslie@, gnn@ and salvadore@.

A few default versions got bumped:

Java (new) at 8
Lazarus to 2.0.8

It is now possible to write pkg scripts in Lua instead of sh.

They have two advantages over their sh versions:

they run in a Capsicum sandbox
they respect rootdir, the directory which pkg will use as the starting point to install all packages under.

Some user-facing packages were also updated:

pkg to 1.14.6
Firefox to 78.0.1
Thunderbird to 68.10.0
Chromium to 83.0.4103.116
Ruby to 2.5.8, 2.6.6, and 2.7.1
Qt5 to 5.14.2

During the last quarter, antoine@ ran 55 exp-runs to test port version updates, make liblzma use libmd, flavor devel/scons and Lua ports, add and update library functions in the base system, make malloc.h usable again, remove as(1) from the base system, and augment sed(1) with -f.

FreeBSD Office Hours

Links
Office Hours on the FreeBSD Wiki	URL: https://wiki.freebsd.org/OfficeHours
Poll: What time would you prefer Office Hours be at	URL: https://forms.gle/3HjjRx9KMcM3SL4H7
live.FreeBSD.org: Aggregation of Live streams	URL: https://live.freebsd.org/

Contact: Allan Jude <allanjude@freebsd.org>

Starting on the first of April 2020, the FreeBSD project has started hosting regular video streams to foster greater communication within the wider FreeBSD community. The first of these sessions took the form of a public question and answer session, which drew over 60 participants. A second session was held two weeks later at a time more appropriate for those in Asia, but only drew 20 participants. With the help of the FreeBSD Foundation, we ran a poll to discover what times worked best for the greatest number of people.

On May 13th the FreeBSD Foundation hosted a session where the community could ask questions of or about the foundation. On May 27th many of the candidates for the new FreeBSD Core Team joined an office hours session to answer questions from the community. Finally on June 24th another general question and answer office hours was held.

Each office hours session consists of a video meeting of some FreeBSD developers or other subject matter experts, live streamed along with an IRC chat room for viewers to pose questions to the panel. The stream is recorded and posted to the official FreeBSD youtube channel.

If you would like to host an office hours session, please contact:

Sponsor: ScaleEngine Inc. (video streaming)

Quarterly Status Reports Team

Links
Quarterly status reports	URL: https://www.freebsd.org/news/status/
Git repository	URL: https://github.com/freebsd/freebsd-quarterly

Contact: Quarterly Status Reports <quarterly@FreBSD.org>
Contact: Daniel Ebdrup Jensen <debdrup@FreeBSD.org>

The Quarterly Status Reports Team collects and publishes the reports that you are reading right now.

Many improvements have been done recently and thus we believe it is useful that the Quarterly Status Reports Team submits a report. Not all the changes below are specific to the last quarter, but we list them here anyway since we did not write an entry for earlier reports.

Reports are now built using Makefiles. Among the many advantages, this allows us to easily sort reports logically. Indeed, starting with 2019Q4, all reports are sorted logically, while before they were sorted alphabetically within each category.
The conversion from markdown to docbook was performed using a python script, with some known bugs. Salvadore has rewritten the script using perl fixing most of the bugs. Some features are missing and many improvements are possible, but the script is very unlikely to receive any change since it will become obsolete as soon as the conversion to Hugo/AsciiDoctor is completed.
Another perl script to ease the preparation of the mail version of the reports was written.
One more perl script has been written to allow the quarterly team to send quarterly calls automatically using a cron job. We used it this quarter for the first time.
As you might have noticed, last quarterly calls have been sent to freebsd-quarterly-calls@: this is a new mailing list to which you can subscribe to receive calls for quarterly reports. Please note this is a moderated list, with very low traffic and a high signal to noise ratio.
If you read carefully the last quarterly calls, you should have noticed that we now ask you to send reports to quarterly-submissions@ instead of quarterly@. This was done to help the quarterly team distinguishing internal discussions from submissions. Please keep in mind however that the quarterly team prefers receiving pull requests, as they ease the administrative work.

We would like to thank philip@, from the postmaster team, for having created the freebsd-quarterly-calls@ mailing list and the quarterly-submissions@ address for us.

Projects

Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.

FreeBSD on Microsoft HyperV and Azure

Links
FreeBSD on MicrosoftAzure wiki	URL: https://wiki.freebsd.org/MicrosoftAzure
FreeBSD on Microsoft HyperV	URL: https://wiki.freebsd.org/HyperV

Contact: FreeBSD Integration Services Team <bsdic@microsoft.com>
Contact: Wei Hu <whu@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

HyperV socket for FreeBSD implemented by Wei was checked into FreeBSD head branch on May 20th as r361275. It supports guest/host communications without the need for networking. Some HyperV and Azure features rely on this to be available in guests.

Details of HyperV Socket is available here.

This project is sponsored by Microsoft.

Li-Wen is working on the FreeBSD release code related to Azure for the -CURRENT, 12-STABLE and 11-STABLE branches. The work-in-progress is available here. The 12.1-RELEASE image on Azure Marketplace is published. The work on the 11.4-RELEASE image on Azure Marketplace is in progress.

This project is sponsored by The FreeBSD Foundation.

Git Migration Working Group

Links
Git conversion tooling repo	URL: https://github.com/freebsd/git_conv
FreeBSD-git mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git
Beta doc git repo	URL: https://cgit-beta.FreeBSD.org/doc
Beta ports git repo	URL: https://cgit-beta.FreeBSD.org/ports
Beta src git repo	URL: https://cgit-beta.FreeBSD.org/src

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ulrich Spörlein <uqs@FreeBSD.org>

Work continues on FreeBSD's migration from Subversion to Git. Ulrich has iterated on updates to svn2git in order to improve the fidelity of the conversion, particularly in regards to vendor (contrib) code updates. We believe the conversion is now at an acceptable state, but may make minor adjustments if additional issues are found. We expect to push modifications to the converter every two weeks (first and third Sunday of the month). This means that commit hashes in the beta repo will remain stable for at least two weeks at a time, to allow others to test and experiment with the beta repo.

We are now working on updating FreeBSD processes and documentation. This includes:

Writing a Git Primer, akin to the existing Subversion primer
Updates to the Security Team's tools and processes
Release engineering updates
Ports and packages process updates

Those with an interest in the migration to Git are encouraged to subscribe to the FreeBSD-git mailing list and test out the beta src, ports, and/or doc repositories.

You are also welcome to check out the wiki, issues, README and other documentation at the Git conversion tooling repo.

We expect to be ready for the migration in the next quarter.

Sponsor: The FreeBSD Foundation (in part)

Lua Usage in FreeBSD

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Kyle Evans <kevans@FreeBSD.org>
Contact: Ryan Moeller <freqlabs@FreeBSD.org>

Lua is a small, efficient scripting language that FreeBSD imported before FreeBSD 12.0 for use in the bootloader. Since then, several projects outside of the bootloader have gained some amount of traction with Lua usage:

/usr/libexec/flua is now installed for internal usage
makesyscalls.sh was rewritten in Lua
pkg has gained support for lua scripts
lldb in the base system now supports lua scripting

FreeBSD Lua ("flua") is a version of the lua interpreter that has several modules built-in for convenient usage within the base system. flua is installed with a non-standard name and in a location not included in $PATH so that it is not accidentally found by third-party software or configure scripts. The FreeBSD project makes no guarantees about upgrade cadence or module stability. That said, it is available for use by downstream projects and FreeBSD users aware of those limitations.

Previous work with flua includes, for example, adding libucl support and future work includes libifconfig support for scripting usage.

People interested in working with Lua in FreeBSD are welcome to get in contact to discuss other project ideas. To name a couple of potential projects, some interesting modules that have not been started but could prove useful (listed in no particular order):

libcrypt
libexpat
libjail
libnv
libxo

Linux compatibility layer update

Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>

Earlier Linuxulator work focused on code cleanups and improving diagnostic tools. Work has now shifted from cleanups to fixing actual applications. Current status is being tracked at Linux app status Wiki page. Initial focus is on applications that don't involve X11, mostly because they tend to be easier to test and debug, and the bug fixes are not application-specific.

Example problems fixed include buggy madvise(2) handling, which could break applications linked against jemalloc; uname(2) returning wrong results for 32 bit apps, which caused problems for Steam; recvmsg(2) and accept(2) being broken in some circumstances, which was breaking Redis; and missing support for SO_REUSEPORT, SO_SNDBUFFORCE, SO_RCVBUFFORCE, and SO_PROTOCOL, which spammed the log files when running the Python regression test suite. The default soft open files limit is now automatically adjusted to 1024, as several Linux apps iterate over all the file descriptors up to that limit instead of using closefrom(2).

There's ongoing work on cleanups and the debugging framework for Linux compatibility, such as logging warnings for unrecognized system call parameters, or adding the compat.linux.debug sysctl to turn the warnings off.

The Linux Test Project tests that are being run as part of the the FreeBSD Continuous Integration infrastructure has been upgraded to 20200605 snapshot. This raised the number of test cases from 3670 to 3749, and, predictably, also the number of failures, from 583 to 647.

There's still a lot to do:

There are pending reviews for patches that add extended attributes support, and fexecve(2) syscall, and they require wrapping up and committing
There are over 500 failing LTP tests. Some of them are false positives, some are easy to fix bugs, and some require adding new system calls. Any help is welcome.

Sponsor: The FreeBSD Foundation

NFS over TLS implementation

Contact: Rick Macklem <rmacklem@freebsd.org>

In an effort to improve NFS security, an internet draft which I expect will become an RFC soon specifies the use of TLS 1.3 to encrypt all data traffic on a Sun RPC connection used for NFS.

Although NFS has been able to use sec=krb5p to encrypt data on the wire, this requires a Kerberos environment and, as such, has not been widely adopted. It also required that encryption/decryption be done in software, since only the RPC message NFS arguments are encrypted. Since Kernel TLS is capable of using hardware assist to improve performance and does not require Kerberos, NFS over TLS may be more widely adopted, once implementations are available.

The project to implement this has largely been completed. The code will slowly be merged into head/current and at least the kernel portion should be in FreeBSD-13.

The code is now available for testing. See: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Setting up system(s) for testing is still a little awkward, as explained by the above rough document.

The main limitation in the current implementation is that it uses TLS1.2 and not TLS1.3. This should change once the KERN_TLS rx patch includes TLS1.3 support. Also, testing of pNFS configurations has not yet been done, but will be tested soon.

Third party testing would be appreciated.

Kernel

Updates to kernel subsystems/features, driver support, filesystems, and more.

SoC audio framework and more audio drivers

Links
rk3399_audio	URL: https://github.com/gonzoua/freebsd/tree/rk3399_audio

Contact: Oleksandr Tymoshenko <gonzo@FreeBSD.org>

SoC audio framework made a good progress since last report.
Support for AUX devices was added (devices like auxiliary amplifiers that are not part of main CODEC chip).
To verify the framework design following audio drivers were added: recording support for RT5640 CODEC(Firefly-RK3399), Allwinner I2S, Alwinners sun8i and A64 CODECs (Pine A64+), both recording and playback.
Current work in progress is RK3328 CODEC (Rock64) and ES8316 CODEC (RockPro64 and Pinebook Pro).

bhyve - NVMe emulation improvements

Links
bhyve NVMe reviews	URL: https://reviews.freebsd.org/search/query/xvbcF20W__Km/

Contact: Chuck Tuffli <chuck@freebsd.org>

The University of New Hampshire InterOperability Laboratory (a.k.a. UNH IOL) develops a suite of tests to determine if an NVMe device conforms to the specification and is interoperable with other NVMe products. This quarter, I undertook getting bhyve's emulated NVMe device to pass the mandatory tests. Changes include:

implement Flush command
implement Format NVM command
implement AER support
implement Namespace Identification Descriptor
fix Active Namespace list
fix queue creation and deletion
validate Deallocate range values
handle zero length DSM ranges
fix Get Log Page command
implement SMART data I/O statistics
validate the LBA start and count
add basic Firmware Commit support
add more compliant Get/Set Features
add Feature, Interrupt Vector Config
fix Get Features, Predictable Latency

This was also a good opportunity to restructure parts of the code to make it more modular and easier to enhance. This includes

convert logging statements to parameterized macros
refactor I/O command handling
add locks around queue accesses
consolidate CQ update
base pci_nvme_ioreq size on advertised MDTS

You can help by testing and/or commenting on the code reviews.

Bluetooth Support

Contact: Marc Veldman <marc@bumblingdork.com>

Bluetooth is a wireless technology for creating personal networks, connecting peripherals like keyboards and mice but also speakers and heart rate monitors.

FreeBSD has limited Bluetooth Basic Rate (BR) support and no functional Bluetooth Low Energy (LE) support.

During this quarter many small improvements have been made to help the development of Bluetooth LE support. A number of commands have been added to hccontrol(8), mainly to do LE functions. It is now possible to scan for LE devices within range using hccontrol. A panic that occurred when enabling LE support has also been fixed.

Work is still needed to add Attribute Protocol (ATT) and Generic Attribute Profile (GATT) support.

DRM Drivers Update

Links
drm-kmod	URL: https://github.com/freebsd/drm-kmod/

Contact: Emmanuel Vadot <manu@FreeBSD.Org>

The drm drivers for FreeBSD 13-CURRENT have been updated to match Linux 5.3 This brings us a little bit closer to the last LTS release of Linux (5.4).

The current plan is to first update the driver to match 5.4 and then look at making it work on FreeBSD-12-STABLE to have it ready for the 12.2 release.

Sponsor: The FreeBSD Foundation

DTS Update

Contact: Emmanuel Vadot <manu@FreeBSD.org>

DTS files (Device Tree Sources) were updated to be on par with Linux 5.7 for HEAD and 5.6 for the 12-STABLE branch.

ENA FreeBSD Driver Update

Links
ENA README	URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

Completed since the last update:

Fixes for Rx refill to improve stability on low memory conditions (also released as an errata notice for FreeBSD-12.1)
Upstream of the v2.2.0 driver, introducing:
- Add driver support for the upcoming HW features (reporting Tx drops, disabling meta caching)
- Add sysctl tuneables for IO queue number
- Create IO queues with optional size backoff
- Rework the way of configration of drbr and Rx ring size to be more robust and stable
- New HAL version
- Driver is now marked as epoch ready
- Default RSS key is created using RNG to improve security
- Other minor fixes and improvements
MFC of the ENA v2.2.0 driver to the FreeBSD 11.4

Sponsor: Amazon.com Inc

Forcible Unmount of UFS/FFS Filesystems on Disk Failure

Links
Phabricator Details	URL: https://reviews.freebsd.org/D24088

Contact: Chuck Silvers <chs@freebsd.org>
Contact: Kirk McKusick <mckusick@mckusick.com>

Commit -r361491 on May 25, 2020 enables a UFS file system to do a forcible unmount when the underlying media fails or becomes inaccessible. For example when a USB flash memory card hosting a UFS file system is unplugged.

The rest of this report describes in more detail how forcible unmounts are done. Surprisingly, less than 500 lines of file system code were added or changed.

The strategy for handling disk I/O errors when soft updates are enabled is to stop writing to the disk of the affected file system but continue to accept I/O requests and report that all future writes by the file system to that disk actually succeed. Then initiate an asynchronous forced unmount of the affected file system.

There are two cases for disk I/O errors:

ENXIO, which means that this disk is gone and the lower layers of the storage stack already guarantee that no future I/O to this disk will succeed.
EIO (or most other errors), which means that this particular I/O request has failed but subsequent I/O requests to this disk might still succeed.

For ENXIO, we can just clear the error and continue, because we know that the file system cannot affect the on-disk state after we see this error. For EIO or other errors, we arrange for the geom_vfs layer to reject all future I/O requests with ENXIO just like is done when the geom_vfs is orphaned. In both cases, the file system code can just clear the error and proceed with the forcible unmount.

This new treatment of I/O errors is needed for writes of any buffer that is involved in a dependency. Most dependencies are described by a structure attached to the buffer's b_dep field, but some are created and processed as a result of the completion of the dependencies attached to the buffer.

Clearing of some dependencies require a read. For example if there is a dependency that requires an inode to be written, the disk block containing that inode must be read, the updated inode copied into place in that buffer, and the buffer then written back to disk.

Often the needed buffer is already in memory and can be used. But if it needs to be read from the disk, the read will fail, so we fabricate a buffer full of zeroes and pretend that the read succeeded. This zero'ed buffer can be updated and "written" back to disk.

The only case where a buffer full of zeros causes the code to do the wrong thing is when reading an inode buffer containing an inode that still has an inode dependency in memory that will reinitialize the effective link count (i_effnlink) based on the actual link count (i_nlink) that we read. To handle this case we now store the i_nlink value that we wrote in the inode dependency so that it can be restored into the zero'ed buffer thus keeping the tracking of the inode link count consistent.

Because applications depend on knowing when an attempt to write their data to stable storage has failed, the fsync(2) and msync(2) system calls need to return errors if data fails to be written to stable storage. So these operations return ENXIO for every call made on files in a file system where we have otherwise been ignoring I/O errors.

Sponsor: Netflix

i.MX 8M support

Links
D25274	URL: https://reviews.freebsd.org/D25274

Contact: Oleksandr Tymoshenko <gonzo@FreeBSD.org>

i.MX 8M is the family of application processors from NXP based on Arm® Cortex®-A53 and Cortex-M4 cores. The initial code drop for the platform support includes CCM driver and clock implementation, GPC driver, clock tree for i.MX 8M Quad. Most of the drivers from i.MX 6 can be reused for i.MX 8M systems with relatively minor modifications. Common changes include adding clock support and extending list of FDT compat strings.

With the linked patch FreeBSD successfully booted to multiuser with NFS root on Nitrogen8M SBC.

Intel wireless and 11ac update

Links
Initial project announcement	URL: https://lists.freebsd.org/pipermail/freebsd-wireless/2020-April/009055.html
The freebsd-wireless mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless

Contact: Bjoern A. Zeeb <bz@FreeBSD.org>

The Intel Wireless cards are one of the most commonly used and asked for in FreeBSD notebooks.

This project has three main goals:

newer Intel Wireless device support,
newer WiFi standards support for Intel Wireless,
integration of 802.11ac client support and infrastructure in FreeBSD.

The first one is needed as iwm(4) currently does not support the latest generations of Intel Wireless cards at all. The second is needed as in FreeBSD iwm(4) does not even support 802.11n. The third one we want to catch up and use the improvements the new Wifi standard offers, e.g., speed.

One of the decisions made was: rather than improving iwm(4) this work uses the dual-licensed native Linux driver under BSD license and the linuxkpi framework to stay as close to upstream as possible as a first step. This will give us several advantages, such as, the full support for all cards, quick support for new chipsets, vendor bug fixes, but also the ability to contribute back.

At this point the lower level hardware attachments and the firmware loading and initialisation works. I plan to release a patchset for testing before mid-July, you can see if your currently supported or unsupported hardware will be detected. This first cut will not support any wireless operation yet, which will follow later in the year.

If you want to help testing, please watch the freebsd-wireless list.

Sponsor: The FreeBSD Foundation

amd64 5-Level Paging Structures support

Links
Patch	URL: https://reviews.freebsd.org/D25273
Intel SDM	URL: https://software.intel.com/content/www/us/en/develop/articles/intel-sdm.html
Intel whitepaper	URL: https://software.intel.com/content/www/us/en/develop/download/5-level-paging-and-5-level-ept-white-paper.html

Contact: Konstantin Belousov <kib@FreeBSD.org>

Since its introduction, x86 Long Mode (AKA 64bit execution mode, amd64 in FreeBSD terminology) uses 4-level paging structures, which provides 48 bits of virtual address space (LA48). FreeBSD evenly divides the space between userspace and kernel, giving both 47 virtual address bits.

In near future Intel CPUs will start providing 5-level paging structures, i.e. giving 57 bits for virtual addresses (LA57). This means, with preservation of the existing divide between KVA and UVA, 56 bit for UVA, or 2^9 = 512 times more virtual memory.

The amd64 pmap was modified to support both LA48 and LA57, defaulting to LA57 if hardware supports it. The tunable is provided to force using LA48 even if hardware can do LA57.

The most interesting part of the patch is the switch from boot paging mode to LA57. Loaders, either legacy or UEFI, pass control to the kernel in Long Mode, which implies that the paging is turned on. This necessarily means that it is LA48 mode. SDM states that paging mode cannot be switched while Long Mode is active, so kernel has to create new page table structures, turn Long Mode off, then load new %cr3 and finally re-enable Long Mode.

I decided to only provide the larger virtual address space to usermode for the initial step, leaving KVA layout intact. The main motivation is that changing KVA arrangements requires changing the auto-tuning settings, which deserve separate work. Another argument for it is that most of the kernel memory is non-swappable, so cannot be over-commited. We have 2:1 ratio of useful KVA to physical memory (due to direct map), and until machines get more physical address lines, increasing KVA is not useful.

After this was decided, creating a 5-level paging structure for kernel pmap from existing 4-level one is quite straightforward; we need to add one page for top level, create one PML5 entry to point to existing PML4 page, and create the famous self-referential entry for vtopte()/vtopde().

Care was taken to provide binary compatible layout of UVA for binaries that cannot be executed correctly with larger address space. For instance, programs could have knowledge about used bits in the addresses and used upper bits for other data, or implemented compressed pointers. Even if system runs in LA57 mode, specific binary can request LA48-compatible UVA by procctl(2) or by the flag in the FreeBSD features ELF note.

Since I do not have access to a machine with LA57, development was done using QEMU. It would be interesting to try it on the real hardware.

Sponsor: The FreeBSD Foundation

Not-transparent superpages

Links
Patch	URL: https://reviews.freebsd.org/D24652

Contact: Konstantin Belousov <kib@FreeBSD.org>

FreeBSD already provides excellent support for superpages, in a manner completely transparent to applications. It tries to proactively prevent fragmentation, reserves contiguous runs of the physical pages for linear allocations in managed objects, and auto-promote runs of small pages when they form complete superpage.

The shortcomings of this approach directly follows from its strength: some applications want to get guaranteed superpage mappings, typically because the underlying physical memory is also offloaded into a hardware which also has memory mapping unit. For instance, Infiniband RMDA adapters do memory registration and remapping, which is more efficient with large pages. In such cases transparent (non-guaranteed) support cannot be used.

The extension was developed for POSIX shared memory subsystem to allow the creator request that the shared memory object was backed by physically contiguous pages, with runs of specified size. The mmap(2) syscall is aware of such objects, and if the requested mapping is properly aligned, it will be served by superpages.

The new type of the shared memory objects are backed by a modified physical pager, which only allocates contiguous physical memory. The VM ensures that mappings of the objects are never split (clipped) on a non-superpage boundary. The fault handler is specially optimized to be very fast by quickly installing the superpage PTE, and to avoid touching all small pages constituing it.

Currently the required pmap support is provided for amd64 with 2M and 1G superpage sizes.

Sponsor: The FreeBSD Foundation

NXP ARM64 SoC support

Contact: Marcin Wojtas <mw@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Dawid Gorecki <dgr@semihalf.com>

The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC

LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with integrated packet processing acceleration and high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications.

Completed since the last update:

Improve code in a couple of review cycles and merge following new features to the FreeBSD-HEAD (r361458 - r361464):
- QorIQ platform clockgen driver
- LS1046A clockgen driver
- GPIO support for QorIQ boards
- QorIQ LS10xx AHCI driver
- VF610 I2C controller support
- TCA6416 GPIO expander
- Epson RX-8803 RTC

Todo:

Upstreaming of the QorIQ SDHCI driver - it is expected to be submitted/merged to HEAD in the Q3 of 2020.

Sponsor: Alstom Group

amd64 pmap Fine-grained pv lists locking

Links
Patch	URL: https://reviews.freebsd.org/D24217

Contact: Konstantin Belousov <kib@FreeBSD.org>

FreeBSD kernel Virtual Memory subsystem handles 'normal' application memory, i.e. anonymous or file-backed shared and private mappings, with so called managed pages. Managed page is fully controlled by VM, which tracks it status. In particular, managed page can be made read-only for write-back to the file, or unmapped for reuse (paging).

The machine-dependent VM layer, pmap, must support managed pages, for instance it must provide operations such as pmap_remove_write() to downgrade all mappings to read-only, or pmap_remove_all() to unmap the page from all address spaces. To implement this kind of operations, while not causing the overhead of scanning all page tables, pmap must track existing mappings of the page. The tracking is done by allocating a small data structure 'pv entry' per mapping, and linking all pv entries for the given page into pv list.

Since pv entries come from context of different address spaces, pmap must provide synchronization to guarantee correctness of the list structures. Current pmap allocates one mutex per one 2M physical superpage in NUMA configurations, and MAXCPU == 256 locks hashed by the page physical address for non-NUMA. The end result is often undeserved lock aliasing causing pv list locks contention, since all 4k pages in the 2M superpage share the same lock, and reservations typically cause adjusted pages to come from the same superpage.

The proposed patch creates a new kernel synchronization primitive called one byte mutex, which is embedded into the currently unused padding in machine-dependent portion of the struct vm_page. This way each page gets dedicated pv list lock without using any more memory. In the ever-important buildkernel benchmark on non-NUMA config, this change provides 2x reduction of the system time.

One complication is that old locking distribution scheme made a natural fit for superpages promotion and demotion, since all embedded small pages shared the same pv list lock, and the operations basically fold/unfold corresponding pv entries. Now the promotion and demotion operations require taking all locks for constituent small pages, which provides small but measurable impact on them. It is possible to optimize it further by providing the 'superlock' on the first page from the superpage run, but the affected operations are relatively rare so that it is not even obvious that implementing the optimization would not slow down other pathes.

Another important nuance of the pv entries handling is that sometimes pv entries allocator must not fail. Typically this is required when kernel makes a call to pmap_enter() which must establish new mapping, and for managed page this includes allocating the pv entry if existing cannot be reused. If allocator cannot get a fresh page from the vm_page_alloc(9), it opts to destroy some other managed mapping to either get a reusable pv entry from current pmap, or destroy enough managed mappings from some other pmap to free whole page.

To do the reclamation, currently all pages from which with pv entries are allocated, are linked in the global pv chunk list, which is protected by global (per-NUMA domain) mutex. Any allocation or free of pv entry has to lock the mutex, which is apparently a contention point for large machines.

Patch removes the global list of chunks, instead linking all pmaps in the global list like it is done on i386 (but for different reason). Now the global lock is only taken for pmap creation and free, which corresponds to fork/exec and exit of a process, and when pv allocator starts reclaiming from other pmaps (which is normally does not happen).

Sponsor: The FreeBSD Foundation

Lockless routing lookups and scalable multipath

Links
Implementation of scalable multipath	URL: https://reviews.freebsd.org/D24141#change-ZOjdMqgDgUr7

Contact: Alexander Chernikov <melifaro@FreeBSD.org>

The primary goal of this work is to bring scalable routing multipath implementation, enabled by default. Another goal is enabling high-performance routing lookups.

Multipath will close long-standing feature gap that modern networking OS must have. Lockless routing lookups will remove lookup bottlenecks, improve both dataplane and control plane performance for the setups with large number of routes.

Background

The initial routing kpi was introduced back in 1980. It was a nice generic approach back then, as no one knew how the protocols would evolve. It has been enormously successful as it was able to survive for 20+ years.

Unfortunately, this kpi does not try to protect subsystem internals from the outside users, resulting in tight coupling with other subsystems. As a result, making changes is hard, leading to compromises and piling technical debt.

Implementation overview

Most changes are based on introduction of the concept of nexthops. Nexthops are separate datastructures, containing all necessary information to perform packet forwarding such as gateway, interface and mtu. They are shared among the routes, providing more pre-computed cache-efficient data while requiring less memory. Interested reader can find more detailed description in D24141. Another overview can be found in Nexthop object talk describing Linux implementation.

Multipath implementation extends nexthop concept further by introducing nexthop groups.

Each route has a pointer to either nexthops or a nexthop group, decoupling lookup algorithm from the routing stack internals. Both nexthops and nexthop groups are immutable and use epoch(9)-backed reclamation.

A pre-requisite for lockless routing lookup is the introduction of modular lookup framework, allowing to attach any longest-prefix-match algorithm implementation to any IPv4/IPv6 fib.

Currently there are plans to use modified DIR-24-8 algorithm from DPDK for both IPv4 and IPv6 families as an example of base lockless implementation.

Status

Nexthop objects [ DONE ]

Introduction of nexthop objects [ DONE ]
Conversion of old KPI users to the new one [ DONE ]
Conversion of route caching to nexthop caching [ DONE ]
Conversion of struct rtentry field access to nhop field access [ DONE ]
Eliminating old lookup KPI and hiding struct rtentry [ DONE ]

Multipath [ IN PROGRESS ]

Switch control plane consumers to use (rtentry, nhop) pairs instead of rtentry to allow multipath changes happen transparently [ 90% DONE ]
Introduce nexthop group objects
Add mutipath support for the rib (routing information base) manipulation functions
Add flowid generation for outbound traffic to enable load balancing

Modular longest-prefix-match lookup algorithm [ IN PROGRESS ]

Design control plane framework for attaching algorithms [ 90% DONE ]
Port IPv6 lockless lookup algorithm [ DONE ]
Port IPv4 lockless lookup algorithm

ZSTD Compression in ZFS

Contact: Allan Jude <allanjude@freebsd.org>

Zstandard (ZSTD) is a modern high-performance compression algorithm designed to provide the compression ratios of gzip while offering much better performance. ZSTD has been adopted in FreeBSD for a number of other uses, including compressing kernel crash dumps, as a replacement for gzip or bzip for compressing log files, and for future versions of pkg(8).

This effort to complete the integration of ZSTD into ZFS is sponsored by the FreeBSD Foundation.

Integrating ZSTD into ZFS will further extend the transparent compression feature of ZFS by offering higher compression ratios without the performance penalty associated with gzip. ZSTD offers compression levels ranging from 1 (low compression) to 22 (maximum compression), plus ZSTD-Fast levels that offer less compression but even greater speed. This will allow the storage administrator to select the performance-vs-compression tradeoff that best suits their needs.

Tasks remaining to be completed:

Extend ZFS to support compression algorithms with large numbers of levels
Solve issues around the inheritence of compression settings
Restore compression level when reading blocks from disk
Create a future-proofing scheme to handle changing versions of ZSTD
Extend ZFS replication to handle backwards compatibility with pools that do not yet support ZSTD
Resolve issues around backwards compatibility when ZSTD is configured but not used

Sponsor: The FreeBSD Foundation

CheriBSD 2020 Q2

Links
CHERI-CPU	URL: http://www.cheri-cpu.org
DARPA FETT Bug Bounty Program	URL: https://fett.darpa.mil

Contact: Alex Richardson <arichardson@FreeBSD.org>
Contact: Andrew Turner <andrew@FreeBSD.org>
Contact: Brooks Davis <brooks@FreeBSD.org>
Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>
Contact: Jessica Clarke <jrtc27@FreeBSD.org>
Contact: John Baldwin <jhb@FreeBSD.org>
Contact: Robert Watson <rwatson@FreeBSD.org>
Contact: Ruslan Bukin <br@FreeBSD.org>

CheriBSD extends FreeBSD to implement memory protection and software compartmentalization features supported by the CHERI instruction set extensions.

Support for CHERI-RISC-V in CheriBSD has continued to mature this quarter in tandem with refinements to the CHERI-RISC-V architecture. We have recently made CheriBSD's "pure capability" (CheriABI) process environment the default ABI rather than a compatibility layer. It has grown support for:

dynamically linked binaries (previously only statically-linked binaries were supported)
C++ including exceptions
sealed return address and function pointer capabilities ("sentries") which provide additional CFI protection
initial MMU protections for loading and storing tags

At this point, CHERI-RISC-V support in CheriBSD is generally on par with support for CHERI-MIPS.

Much of this effort has been focused on preparing CheriBSD on CHERI-RISC-V for inclusion as a demonstrator system in DARPA's Finding -Exploits to Thwart Tampering (FETT Bug Bounty -program). +Exploits to Thwart Tampering (FETT) Bug Bounty +program.

In addition, work has begun this quarter on porting CheriBSD to Arm's Morello SoC. Morello is a prototype demonstrator board which adds CHERI extensions to ARMv8-A.

We've recently switched to a dev-branch model where active work takes place on the dev branch and we periodically merge to master with synchronization between CheriBSD and dependencies like CHERI-LLVM.

For those interested in what it's like to program for CHERI, we've recently released a CHERI C/C++ Programming Guide.

Architectures

Updating platform-specific features and bringing in support for new hardware platforms.

Continuous Integration on !x86

Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>

For quite a while the FreeBSD CI infrastructure has been running FreeBSD builds and regression tests, making it easy to spot regressions. While CI was building images for all architectures, the regression tests were only run on amd64 and i386, which means they couldn't detect architecture-specific runtime breakage on non-x86 architectures. This poses a problem not only for FreeBSD itself, but also for people working on FreeBSD forks for !x86, such as the CHERI project at University of Cambridge and SRI International.

The goal of this project is to run regression tests on the remaining architectures supported by FreeBSD: ARM, ARM64, MIPS, POWER, and RISC-V. The tests are being run using common, mostly machine-independent scripts. Those required some changes to make it possible to use QEMU in addition to Bhyve, the hypervisor used for x86 tests. The sysutils/u-boot-qemu-arm and sysutils/u-boot-qemu-arm64 ports were added to the Ports Collection. Finally, each of the architectures required some tweaks, to account for different configuration requirements - for example the MIPS kernel doesn't support VirtIO disks, or even AHCI, whereas on ARM64 the U-Boot gets confused with more than one VirtIO disk.

On ARM, we're currently at 52 failures and 590 skipped tests, of 5925 tests ran (https://ci.freebsd.org/job/FreeBSD-head-armv7-test/lastCompletedBuild/testReport/). On ARM64 it's 19 failures and 160 skipped (https://ci.freebsd.org/job/FreeBSD-head-aarch64-test/lastCompletedBuild/testReport/). On MIPS it's 172 failures and 734 skipped (https://ci.freebsd.org/job/FreeBSD-head-mips64-test/lastCompletedBuild/testReport/). For POWER, and RISC-V the results are not available yet.

Remaining work:

Failing regression tests need to be fixed.
The tests are quite slow on QEMU, for example the ARM64 run takes about five hours. Running them automatically after each commit would quickly overload the CI cluster. A solution would be to e.g. run them daily.
Some of the jobs still fail to produce results: powerpc64 deadlocks at the end of regression test suite due to an unkillable process, riscv64 panics randomly, and on mips64 kyua(1) often crashes on jemalloc assertion. Those might be fixed by an upcoming QEMU port update.

Sponsor: DARPA

FreeBSD/RISC-V Project

Links
Wiki	URL: https://wiki.freebsd.org/riscv

Contact: Mitchell Horne <mhorne@FreeBSD.org>

Contact: freebsd-riscv Mailing List
Contact: IRC #freebsd-riscv channel on freenode

The FreeBSD/RISC-V project is providing support for running FreeBSD on the RISC-V Instruction Set Architecture. Since RISC-V is still a young and evolving platform, one of our goals is to have FreeBSD be a well-supported option for users as RISC-V hardware increases in availability.

This quarter saw a number of improvements to the boot process.

The physmem interface used by arm and arm64 to enumerate physical memory resources was moved to machine-independent code and adopted on RISC-V. As a result, the kernel is now able to detect and exclude physical memory reserved by devices or firmware. A bug that prevented the kernel from using physical memory below its load address was fixed. This typically did not manifest in much waste, as the kernel is loaded 2MB after the start of physical memory by default. In future boot configurations, the impact would have been much larger.

Our port for OpenSBI was updated to v0.8, bringing several new features and fixes. In particular, it brought the Hardware State Management (HSM) extension, which can be used to start and stop CPUs. FreeBSD will now use this extension whenever it detects that it is available.

There has also been a lot of work done to port FreeBSD's standard bootloader, loader(8), to RISC-V. This has big advantages in terms of boot flexibility, and brings us closer to what's needed to produce official FreeBSD/RISC-V release images. By leveraging UEFI support from u-boot, loader.efi can be used in a manner similar to arm and arm64. Next quarter will likely bring u-boot ports for RISC-V targets, pending the v2020.07 release. Booting the kernel directly via SBI firmware will continue to be supported.

Userland Programs

Changes affecting the base system and programs in it.

Import of new implementation of bc and dc

Links
Official repository	URL: https://git.yzena.com/gavin/bc
Repository mirror on GitHub	URL: https://github.com/gavinhoward/bc/

Contact: Stefan Esser <se@FreeBSD.org>
Contact: Gavin D. Howard <yzena.tech@gmail.com>

A new version of bc and dc has been imported into FreeBSD-CURRENT and enabled by default. An import into 12-STABLE is planned before the end of July, but it will not be enabled by default (will require "WITH_GH_BC=yes" to be set in /etc/src.conf).

This version has been developed by Gavin D. Howard with the goal to provide a highly portable and POSIX compatible implementation. It offers GNU bc compatibility and should be a drop-in replacement for the bc in FreeBSD, except for standard-violating behavior of the bc currently in FreeBSD (e.g., the modulo operator).

Additional features:

High performance (up to more than a factor of 100 faster than the current FreeBSD implementation in some tests)
support of message catalogs with a large number of languages supported in the current release (contributions of further translations are welcome).
Extra built-in functions and operators.
Extended library of advanced math functions
Detailed man-page explaining standard conformant and extended features
One shared binary for bc and dc (bc is not just a pre-processor that relies on dc for the actual computations)

The only dc feature not supported by the dc is the execution of sub-processes, since the author considers it a security hazard for a calculator to have.

This code is also available as a port and package (math/gh-bc or gh-bc), e.g. for use with a FreeBSD binary release.

Binutils Retirement

Links
GPL in Base wiki page	URL: https://wiki.freebsd.org/GPLinBase

Contact: Ed Maste <emaste@FreeBSD.org>

We have been working on migrating to a modern and copyfree or permissively licensed toolchain for quite some time. In the last quarter we retired two obsolete GNU bintuils: objdump, and as.

Many uses of objdump can be replaced with readelf, and llvm-objdump is also available in the base system. Ports that depend on objdump have been updated to rely on the GNU binutils port or package.

The GNU as utility was used by both the base system and by ports. As with objdump, ports that require GNU as have generally been updated to depend on binutils. One file in the base system (skein_block_asm.s) proved troublesome during earlier attempts to migrate to using Clang's integrated assembler (IAS). However, after the update to Clang 10 (and with some trivial modifications to the source) IAS can assemble the file.

Both GNU as and objdump have been removed from FreeBSD-CURRENT and will be absent from FreeBSD 13.0.

TODO

The final task in the binutils retirement project is to remove GNU GDB 6.1. It is currently retained for crashinfo(8).

Sponsor: The FreeBSD Foundation

Run-Time Dynamic Linker improvements

Contact: Konstantin Belousov <kib@FreeBSD.org>

Rtld gets some number of small bug fixes and improvements.

RTLD_DEEPBIND dlopen(3) flag was implemented, despite being a strange and even unsafe idea, for compatibility with glibc.

Several improvements to the direct execution mode were made. Most interesting are perhaps the '-v' switch to report some configuration parameters for rtld, the ability to specify argv0 different from the executed binary name, and fixes to properly set osrel/ABI for the directly executed binary.

The link_map structure that is used by tools that need to know the list of loaded shared objects (like gdb and wine) was made more compatible with glibc, while keeping existing FreeBSD ABI intact.

In the course of the link_map work, it become apparent that rtld sometimes needs to report presence of features that cannot be deduced by just a runtime test for symbol presence or for function behavior. For that, a scheme of reporting features with uniformingly named symbols was designed - see the rtld(1) man-page (in CURRENT) for an explanation.

Position-independent (PIE) binaries on FreeBSD are now marked with the DF_1_PIE DT_FLAG1 flag. Otherwise, such binaries are just ET_DYN objects and it is quite hard to distinguish proper dynamically shared object (DSO) from PIE binary. The problem is that for binaries, the static linker strips some information which is required for proper loading as a DSO, and additonally, binaries contains relocations like copy-relocations that cannot be handled for non-main binaries at all.

With the flag addition, rtld properly detects binaries and refuses to load them with dlopen() or as DT_NEEDED dependency. ldd(1) also misdetected PIE vs. DSO, and required a fix to parse dynamic segments to not try to dlopen() them.

Sponsor: The FreeBSD Foundation

VHDX support in mkimg(1)

Contact: Oleksandr Tymoshenko <gonzo@FreeBSD.org>

VHDX is the successor of Microsoft's VHD virtual drive file format. It increases maximum capacity of the virtual drive to 64TB and introduces features to better handle power/system failures.
VHDX is the required format for 2nd generation Hyper-V VMs.

Ports

Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves.

Bastille

Links
Bastille GitHub	URL: https://github.com/bastillebsd/bastille
Bastille Templates	URL: https://gitlab.com/bastillebsd-templates
Bastille Website	URL: https://bastillebsd.org

Contact: Christer Edwards <christer.edwards@gmail.com>

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.

Bastille Templates automate container setup allowing you to easily reproduce containers as needed.

Bastille is available in ports at sysutils/bastille.

Q2 2020 Status

In Q2 2020 Bastille merged some exciting new features into GitHub. Changes include:

experimental support for new Bastillefile template syntax
added mount and umount sub-commands
added a default Vagrantfile for simple testing
experimental support for empty containers
improvements to VNET DHCP support
cosmetic bugfixes in error output
extended config file documentation
updated bastille help output
option to (-f) force destroy container

sysutils/bastille was updated to 0.6.20200414 (latest).

New Bastille templates added this quarter include:

Percona database server
Asterisk SIP server
dnsmasq DNS/DHCP server (VNET required)
nginx pkg server for poudriere

Everything mentioned here was done under COVID-19 quarantine. Special thanks to everyone that contributed during this time.

KDE on FreeBSD

Links
KDE FreeBSD	URL: https://freebsd.kde.org/
KDE Community FreeBSD	URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project packages the software produced by the KDE Community for FreeBSD. The software includes a full desktop environment KDE Plasma, IDE KDevelop, a PIM suite Kontact and hundreds of other applications that can be used on any FreeBSD desktop machine.

This quarter has been an ever-so-peculiar one. While we are used to working remotely, collaborating over the internet to update the ports tree, it's qualitatively different when the whole world locks down. Meanwhile, software continues to be released, so this quarter the kde@ team:

Restored a patch that closes down a remote TCP held by X11 applications that use the ICE library. Thanks to Colin Percival for reporting it. It went missing in one of the port updates. To prevent this in the future, the patch has been upstreamed. PR 229772.
Chased KDE-adjacent software like CMake, Cutelyst, Latte-dock and Nheko through new releases. In particular CMake takes a lot of effort every time because it is a build-time dependency of over 2000 ports.
graphics/poppler was updated to the latest upstream release. This is a low-level dependency for many document-viewing applications, and like CMake requires chasing a lot of other software. Poppler is one of the components shared between various software stacks (and "desktop environments") under the desktop@ group, in which kde@ participates.
KDE Frameworks release like clockwork, reaching KDE Frameworks 5.70 mid-may.
KDE Applications -- the KDE release service, really, which delivers libraries, applications, and add-ons -- had one large release, with 20.04.1 landing in the ports tree also mid-may and its monthly update 20.04.2 in mid-june.
Some new Wayland support for KDE Plasma -- we have not tested this on FreeBSD -- has appeared and has been duly packaged.
A great deal of preparation has gone into Qt 5.15. Many ports have been pre-emptively patched for this new -- and last -- LTS release of Qt 5. The update itself has not yet landed, pending a few last bits of fallout.

The kde@ team would like to thank Antoine for many exp-runs, mikael@ for useful tips, swills@ for patience and kai@ for dealing with WebEngine.

The next big round of updates for the KDE stack is slated: CMake 3.18, Qt 5.15 LTS, and KDE Frameworks 5.71.

Haskell on FreeBSD

Links
Haskell language homepage	URL: http://www.haskell.org/
Ports development repo	URL: https://github.com/freebsd/freebsd-ports-haskell

Contact: Gleb Popov <haskell@FreeBSD.org>

Haskell is a general-purpose strictly-typed pure functional language. The Haskell on FreeBSD projects strives to provide the up-to-date Haskell toolchain as well as various application written in this language.

This quarter brought the long-awaited GHC update, which is now at version 8.8.3. Along the compiler, the Haskell build system frontend, cabal-install, was also upgraded to 3.0.2.0. During this update, numerous Haskell ports were updated too.

All existing ports of Haskell applications were migrated to USES=cabal, which implements Go-style build proccess - all dependencies are compiled as part of the build. As a consequence, ports for Haskell libraries have been deprecated and removed.

Upgrading GHC became a tedious task for a single person, so a new GitHub repository was created under the FreeBSD organization - freebsd-ports-haskell. Right now, work is being done on preparing another GHC upgrade in the ghc-upgrade-810 branch. Any contributions are welcome.

rtsx - Porting driver for Realtek SD card reader from OpenBSD

Links
rtsx	URL: https://github.com/hlh-restart/rtsx
PR204521	URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204521

Contact: Henri Hennebert <hlh@restart.be>

The rtsx driver for Realtek SD card reader has been ported from OpenBSD. Its development snapshot is available via the sysutils/rtsx-kmod port.

From March to May 2020, the code has been completed with the help of Gary Jennejohn (gj@) and Jesper Schmitz Mouridsen (jsm@). Some tweaks have been imported from the Linux counterpart.

The driver has been successfully tested with:

RTS5209 under head (Lenovo ThinkPad L520)
RTS5227 under stable/11 and releng/12.1 (HP ProBook 430 g2, Lenovo ThinkPad T450/T450s)
RTS5229 under releng/12.1 (Lenovo IdeaPad 120S-14IAP)
RTS522A under releng/12.1 and head (Intel NUC8i5BE, Lenovo ThinkPad P50s)
RTS525A under releng/12.1 (Dell Latitude E5570)
RTL8411B under stable/12 (Acer Aspire E 15 E5-576-77W6)

The driver should also work for Realtek RTS5249, RTL8402 and RTL8411.

More tests are welcome, especially for the devices not yet tested. These devices may require more tweaks.

PR204521 contains the bulk of exchanges for completion of the code.

Valgrind updates

Links
Patch for valgrind	URL: https://reviews.freebsd.org/D25452

Contact: Paul Floyd <paulf@free.fr>
Contact: Kyle Evans <kevans@FreeBSD.org>

A large amount of work has been done to rebase FreeBSD support on top of Valgrind 3.17.0, and to address numerous test suite failures. Currently, almost all of the regression tests pass on amd64. This is a major improvement over the current state of affairs, in which the Valgrind is quite out of date and is missing important functionality. Some follow-up work aims to make FreeBSD an officially supported target platform for Valgrind.

The devel/valgrind-devel port is in the process of being updated to point at the new work.

Documentation

Noteworthy changes in the documentation tree, in manpages, or in external books/documents.

FreeBSD Translations on Weblate

Links
Translate FreeBSD on Weblate wiki	URL: https://wiki.freebsd.org/DocTranslationOnWeblate
FreeBSD Weblate Instance	URL: https://translate-dev.freebsd.org/

Contact: Danilo G. Baio <dbaio@FreeBSD.org>
Contact: Edson Brandi <ebrandi@FreeBSD.org>

This quarter was improved the renderization of RTL (Right-to-left) languages on the FreeBSD Documentation. We faced this issue after the first RTL language joined the translations effort (fa_IR).

We are looking forward to receive more languages and translators to the project as well.

Q2 2020 Status

10 languages (No new languages)
80 registered users (33 new users since last quarter)

Languages

Chinese (Simplified) (zh_CN)
Chinese (Traditional) (zh_TW)
French (fr_FR)
German (de_DE)
Italian (it_IT)
Norwegian (nb_NO)
Persian (fa_IR)
Portuguese (pt_BR)
Spanish (es_ES)
Turkish (tr-TR)

We want to thank everyone that contributed, translating or reviewing documents. And please, help promote this effort on your local user group, we always need more volunteers.

Miscellaneous

Objects that defy categorization.

FreshPorts

Links
FreshPorts	URL: http://freshports.org/
FreshPorts blog	URL: http://news.freshports.org/

Contact: Dan Langille <dan@langille.org>

FreshPorts, and its sister site, FreshSource, have reported upon FreeBSD commits for 20 years. They cover all commits, not just ports.

FreshPorts tracks the commits and extracts data from the port Makefiles to create a database of information useful to both port developers and port users.

For example, https://www.freshports.org/security/acme.sh/ shows the history of this port, back to its creation in May 2017.

git

Work on git started back in September. It was ignored for a while and started back in mid-June with the creation of new git-specific jails for commit ingress (commit processing gitdev) and for the website.

Serhii (Sergey) Kozlov created a script to transform GIT commit entries into XML digestible by FreshPorts. This was a huge step foward for the effort.

The next step include:

incorporate a that script the automated processes of FreshPorts
migrate to new test & stage versions of FreshPorts
test
get ready for prod

Help wanted

git is not far away now. I could use helpers to

review code
watch the commits on the devgit websites
catch missing stuff

Thank you

Packages

FreshPorts now displays the packages version available from the repo sources. This covers all primary tiers (e.g. FreeBSD:12:amd64) and all secondary tiers (e.g. FreeBSD:13:powerpc64). This helps uses know what versions they can expect and when then repo was last built.

Dependency lines

Some things are easiest done via copy/paste. If you are working on a port Makefile and need to add a new dependency, FreshPorts shows the dependency line for that port. For example:


 acme.sh>0:security/acme.sh

Libraries are also covered by this feature.

Python ports were recently adjusted to display


  ${PYTHON_PKGNAMEPREFIX}virtualenv>0:devel/py-virtualenv

instead of


 py37-virtualenv>0:devel/py-virtualenv

You can read more about this change in [issue #73](https://github.com/FreshPorts/freshports/issues/73).

Watch ports I maintain

The search page has long had the "Ports I Maintain" button (if you are logged in). This feature recently branched out to a new automated watch list option: Watch ports I maintain.

This report subscription will notify you of any commits to the ports you maintain. Your email address on FreshPorts must match the value in the MAINTAINER field of the port. This is always a daily report.

From time to time, an infrastructure change will occur which touches your port. This feature ensures you know about that change.

Repology links

Repology links were requested. This allows you to see what versions of that port are in the repositories of other systems. A link to repology.org appears on every port page.

Javascript help wanted

We recently upgraded some outdated Javascript modules. This broke our [JavaScript based graphs](https://www.freshports.org/graphs2.php). We could use some help on fixing that please. The starting points are listed on that URL. If you need a working website to play with, please contact me with a ssh public-key.

Sponsor: hardware provided by iXsystems

PCI passthrough with bhyve on Intel and for OpenBSD guests

Links
bhyve Intel bug report	URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229852
bhyve OpenBSD bug report	URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245392
PCI passthrough with bhyve (FreeBSD wiki article)	URL: https://wiki.freebsd.org/bhyve/pci_passthru

Contact: Anatoli <me@anatoli.ws>
Contact: Callum <callum@aitchison.org>
Contact: Peter Grehan <grehan@freebsd.org>

bhyve(8) is a hypervisor that supports running a variety of guest operating systems in virtual machines. bhyve(8) includes support for PCI devices passthru, a technique to pass host PCI devices to a virtual machine for its exclusive control and use.

For some years, PCI passthrough (ppt) in bhyve was not working on some Intel systems and for OpenBSD guests due to two bugs. The first one was crashing FreeBSD host when bhyve was started with ppt on Intel processors with two VT-d translation units (IOMMU), included in most Skylake and newer Intel processors.

The second bug was preventing correct interrupts handling for OpenBSD guests. As a result, OpenBSD guests running on bhyve were not able to use any PCI devices passed through to them from the host.

During the last 2 months the second bug was identified and fixed and they both were backported to 12.1-RELEASE (p7). So now it's possible to fully take advantage of PCI passthrough (ppt) with bhyve in a production-ready RELEASE version.

The most typical case for ppt is to pass to the guest network adapters for its complete control, but you can also pass through USB devices (including external HDDs). Note though, passthrough of VGA and GPU devices is not supported yet (for more details see the 3rd link).

A particularly interesting case for ppt is to use OpenBSD guest as a firewall and a router for a FreeBSD server.

With ppt you can achieve this all inside a single server. You could pass to the OpenBSD guest a network adapter connected to the internet and it would take a complete control of it. After filtering the traffic, it could pass good packets via virtual network interfaces to other guests or to the host.

Once a network adapter is passed through, a FreeBSD host not only doesn't see it and hence doesn't handle the network traffic, it doesn't even have to initialize the adapter (e.g. in case of a WiFi card, it's the guest that loads the firmware).

In simple terms the host only passes the device interrupts to the guest as they come from the hardware. Everything related to the device management happens inside the guest so there's no danger that some network traffic exploits some issue in the host's network stack and causes the host to crash or misbehave in other ways.

SageMath

Links
SageMath site	URL: https://www.sagemath.org/

Contact: Thierry Thomas <thierry@FreeBSD.org>

SageMath is a free open-source mathematics software system licensed under the GPL. It builds on top of many existing open-source packages: NumPy, SciPy, matplotlib, Sympy, Maxima, GAP, FLINT, R and many more. Thanks to SageMath it is possible to access their combined power through a common, Python-based language or directly via interfaces or wrappers.

The goal is creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab.

This is a complex port, with a lot of dependencies, and it has been broken for some time. Upstream is working on easing its packaging, and many previously bundled applications can now be replaced by external packages.

If you are interested, it would be nice to create a team of maintainers

to maintain some of the dependencies;
to maintain SageMath itself and prepare the next release (9.2 is coming!).

Third-Party Projects

Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions.

chaifi - a tool to simplify joining public WiFi networks

Links
chaifi	URL: https://github.com/gonzoua/chaifi

Contact: Oleksandr Tymoshenko <gonzo@FreeBSD.org>

chaifi is a TUI (text UI) utility aimed at simplifying the process of joining public WiFi networks in places like coffee shops or libraries. It replaces the process of scanning and manually editing wpa_supplicant.conf with an interactive dialog. The utility is in no way a replacement for full-featured network managers in major desktop environments. Still, if you're working from a console or using a tiling window manager, it may save you some seconds (or in worst case minutes) of your time.

MixerTUI

Links
mixertui	URL: https://gitlab.com/alfix/mixertui

Contact: Alfonso Sabato Siciliano <alfonso.siciliano@email.com>

MixerTUI is a volume mixer with a Terminal User Interface built on the FreeBSD sound system. It can show the current Sound Driver configuration and select an audio device: to get its information, to change the volume or to set it as default, the last feature allows to switch easily audio from/to laptop and hdmi, headphones and speakers, and so on.
MixerTUI can be installed via the audio/mixertui port.

I would like to thank the FreeBSD community for the tips, feedbacks and patches to improve this project.

Potluck - Flavour & Image Repository for pot

Links
Potluck Repository & Project	URL: https://potluck.honeyguide.net/
Potluck on github	URL: https://github.com/hny-gd/potluck
pot project	URL: https://pot.pizzamig.dev

Contact: Stephan Lichtenauer <sl@honeyguide.eu>

pot is a jail management tool that also supports orchestration through nomad.

Potluck aims to be to FreeBSD and pot what Dockerhub is to Linux and Docker: A repository of pot flavours and complete images for usage with pot.

This should simplify setting up complex software with many packages and ports in comparison to manual configuration: Potluck aims to provide a content library as an additional layer of abstraction, on top of existing infrastructure like pkg, that pot has to offer.

Pot "flavour" files are provided on [Github]((https://github.com/hny-gd/potluck)) and fed into a Jenkins instance. On the Potluck Repository, for each flavour, detailed descriptions as well as ready-made images to be imported by pot are provided.

The initial project has been set up, and three simple flavours, along with a complete Jitsi Meet instance in a jail has been created as a Proof of Concept that should allow running a fully-fledged video conference system with just a few easy commands within a few minutes.

As only the initial versions have been set up and implemented so far, general feedback, tests, as well as additional, useful flavours are very welcome!

News Home | Status Home

Skip site navigation (1) Skip section navigation (2)

Header And Logo

Donate to FreeBSD

Peripheral Links

Site Navigation

Section Navigation

Introduction

This report covers FreeBSD related projects for the period between July and September, and is the third of four planned reports for 2020.

This quarter brings a good mix of additions and changes to the FreeBSD Project and community, from a diverse number of teams and people covering everything from architectures, continuous integration, wireless networking and drivers, over drm, desktop and third-party project work, as well as several team reports, along with many other interesting subjects too numerous to mention.

As the world is still affected by the epidemic, we hope that this report can also serve as a good reminder that there is good work that can be done by people working together, even if we're apart.

We hope you'll be as interested in reading it, as we've been in making it.
Daniel Ebdrup Jensen, on behalf of the quarterly team.

FreeBSD Team Reports

FreeBSD Foundation
FreeBSD Release Engineering Team
Cluster Administration Team
Continuous Integration
Ports Collection
FreeBSD Office team - 3rd quarter 2020 report
FreeBSD Graphics Team status report

Kernel

DRM Drivers Update
DTS Update
DesignWare Ethernet adapter driver improvements
Google Summer of Code’20 Project - eBPF XDP Hooks
ENA FreeBSD Driver Update
IPSec Extended Sequence Number (ESN) support
NXP ARM64 SoC support
Addition of PowerPC64LE Architecture
ure - USB 3.0 Gigabit Ethernet Driver update
Stateless hardware offloads for VXLANs
Wireless updates
ZSTD Compression in ZFS

Architectures

CheriBSD 2020 Q3
FreeBSD/RISC-V Project

Ports

Update to grub-bhyve
KDE on FreeBSD

Documentation

DOCNG on FreeBSD

Third-Party Projects

Potluck - Flavour & Image Repository for pot
Puppet

FreeBSD Team Reports

Entries from the various official and semi-official teams, as found in the Administration Page.

FreeBSD Foundation

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

Here are some highlights of what we did to help FreeBSD last quarter:

COVID-19 Impact to the Foundation

Partnerships and Commercial User Support

We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Not surprisingly, the stay at home orders, combined with our company ban on travel during Q3 made in-person meetings non-existent. However, the team was able to continue meeting with our partners and commercial users virtually. These meetings help us understand some of the applications where FreeBSD is used.

We are currently scheduling Zoom company meetings for Q4, please reach out if you would like to schedule a meeting with us.

Fundraising Efforts

Last quarter we raised $192,874.43! Thank you to the individuals and organizations that stepped in, to help fund our efforts. We'd like to thank Arm for their large contribution last quarter, which helped bring our 2020 fundraising effort to $521k. We hope other organizations will follow their lead and give back to help us continue supporting FreeBSD.

These are trying times, and we deeply appreciate every donation that has come in from $5 to $150,000. We're still here giving 110% to supporting FreeBSD!

Please consider making a donation to help us continue and increase our support for FreeBSD.

We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information about the partnership program and share with your companies!

OS Improvements

A number of FreeBSD Foundation grant recipients started, continued working on, or completed projects during the third quarter. These include:

Ongoing WiFi and Linux KPI layer improvements.
Linuxulator application compatibility.
DRM / Graphics driver updates.
Zstd compression for OpenZFS.
Online RAID-Z expansion.
Modernized LLDB target support for FreeBSD.

You can find more details about most of these projects in other quarterly

reports.

Staff members also worked on a number of larger projects, including:

Run-Time Dynamic Linker (rtld) and kernel ELF loader improvements.
Rewritten UNIX domain socket locking.
Build infrastructure.
Open system call path handling support for O_BENEATH, O_RESOLVE_BENEATH.
arm64 support.
Migration to a Git repository.

Many of these projects also have detailed entries in other quarterly report

entries.

University of Waterloo Co-op

With the transition to working from home, the Foundation decided to again take on three University of Waterloo Co-op students for the Fall 2020 term (September to December). Tiger returns for a second term, joined by new students Yang and Zac. Projects for the term include more work on ELF Tool Chain, application of Capsicum to additional utilities, testing and integration of FreePBX and Asterisk VOIP software, pkgbase, and exploring containerization tooling.

Continuous Integration and Quality Assurance

The Foundation provides a full-time staff member and funds projects on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD project.

During the third quarter of 2020, Foundation staff continued improving and monitoring the Project's CI infrastructure, and working with experts to fix the failing builds and the regressions found by tests. The setting up of dedicated VM host for running tests is completed. New feature developments and the CI staging environment is in progress. We are also working with other teams in the Project for their testing needs. For example, tests of non-x86 architectures now run periodically, and improve the CI of the embedded systems. We are also working with many external projects and companies to improve the CI between their products and FreeBSD.

See the FreeBSD CI section of this report for completed work items and detailed information.

Supporting FreeBSD Infrastructure

The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. We coordinated efforts between the new NYI Chicago facility and clusteradm to start working on getting the facility prepared for some of the new FreeBSD hardware we are planning on purchasing. NYI generously provides this for free to the Project. We also worked on connecting with the new owners of the Bridgewater site, where most of the FreeBSD infrastructure is located.

Some of the purchases we made for the Project last quarter to support infrastructure includes:

Spamhaus spam filtering software to limit the amount of spam on the mailing lists.
5 application servers to run tasks like bugzilla, wiki, website, cgi, Phabricator, host git, etc.
1 server to replace the old pkg server and provide a lot more IOPS to avoid the slowdowns seen during peak times of the day where the disks just cannot keep up with the request volume.
1 server for exp-runs to make them faster.
1 server to build packages more frequently.

FreeBSD Advocacy and Education

Check out some of the advocacy and education work we did last quarter:

Launched our FreeBSD Fridays series of 101 classes. Topics included an Introduction to FreeBSD, FreeBSD Installfest, Introduction to Security, Introduction to ZFS and more. Videos of the past sessions and a schedule of upcoming events can be found here.
Attended and presented at OSI's State of the Source conference. The event was held virtually, September 9-11, 2020.
Launched the redesign of the FreeBSD Foundation Website.
Announced the 20th Anniversary of the FreeBSD Foundation.
Participated as an Admin for Google Summer of Code 2020
Continued to promote the FreeBSD Office Hours series including holding our own Foundation led office hours. Videos from the one hour sessions can be found on the Project's YouTube Channel. You can watch ours here.
Interviewed members of the outgoing FreeBSD Core Team to get their thoughts on their term.
Began working with the FreeBSD Vendor Summit planning committee on the November 2020 Vendor Summit.
Promoted the Foundation's 20th Anniversary and our work to support the FreeBSD Project in the It's FOSS Article. FreeBSD Foundation Celebrates 20 Years of Promoting and Supporting FreeBSD Project.
Authored a Beginners Guide to FreeBSD for Fosslife.
Committed to sponsoring All Things Open as a media Sponsor.
Committed to sponsoring the OpenZFS Developers Summit at the Bronze level.
Became an International RISC-V Member.
Committed to giving a FreeBSD talk at the nerdear.la conference on October 20th.

Keep up to date with our latest work in our

monthly newsletters.

Netflix provided an update on how and why they use FreeBSD in our latest Contributor Case Study.

You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/.

Legal/FreeBSD IP

The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. We updated our Trademark Usage Terms and Conditions on July 1, 2020.

Go to the FreeBSD Foundation's web site to find out how we support FreeBSD and how we can help you!

Other

We welcomed Andrew Wafaa and Kevin Bowling to our board of directors, to help govern the Foundation and guide us with our strategic direction. We have more information about our new board members on our website.

FreeBSD Release Engineering Team

Links
FreeBSD 12.2-RELEASE schedule	URL: https://www.freebsd.org/releases/12.2R/schedule.html
FreeBSD 12.2 test builds	URL: https://www.freebsd.org/where.html#helptest
FreeBSD development snapshots	URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

During the third quarter of 2020, the Release Engineering Team started work on the 12.2-RELEASE cycle, the third release from the stable/12 branch.

As of this writing, two BETA builds have been released, with the expectation there will be a third BETA build currently remaining on the schedule.

The 12.2-RELEASE cycle will continue throughout October, with two RC builds currently planned, and RC3 scheduled on an as-needed basis. The 12.2-RELEASE is so far scheduled for final release on October 27.

In addition to the 12.2-RELEASE, Glen Barber of the Release Engineering Team finished work to the release build tools and scripts to prepare for the conversion from Subversion to Git for the 13.0-RELEASE cycle. There are no plans to merge these changes to stable branches at this time; as discussed within the Git working group, we feel such a change on a stable branch would be too intrusive to our user base as well as downstream FreeBSD consumers. Development snapshot builds for 13.0-CURRENT have recently been built from the Git tree within the project, and further snapshot builds for 12.x and 11.x will continue to be built from Subversion.

Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches.

Finally, the Release Engineering Team would like to thank Marius Strobl for his time serving on the team; he had recently stepped down from the Deputy RE Lead role due to constraints on his time. The Team welcomes Colin Percival, who has accepted fulfilling this role.

Much of this work was sponsored by Rubicon Communications, LLC (netgate.com) and the FreeBSD Foundation.

Cluster Administration Team

Links
Cluster Administration Team members	URL: https://www.freebsd.org/administration.html#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

Work with the FreeBSD Foundation on hardware update for web services, mirror and package building servers.
Disable directory indexing on the package mirrors to resolve performance issues of the machine.
- This was later relaxed to allow indexing of the parent directories but still disallow the large package directories.
Ongoing systems administration work:
- Accounts management for committers.
- Backups of critical infrastructure.
- Keeping up with security updates in 3rd party software.

Work in progress:

Setup Malaysia (KUL) mirror.
Setup Brazil (BRA) mirror.
Review the service jails and service administrators operation.
Infrastructure of building aarch64 and powerpc64 packages.
- NVMe issues on PowerPC64 POWER9 blocking dual socket machine from being used as pkg builder.
- Drive upgrade test for pkg builders (SSDs) courtesy of the FreeBSD Foundation.
- Boot issues with Aarch64 reference machines.
New NYI.net sponsored colocation space in Chicago-land area.
Work with git working group for the git repository.
Searching for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror.

Continuous Integration

Links
FreeBSD Jenkins Instance	URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab	URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive	URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report	URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki	URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki	URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI	URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@	URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository	URL: https://github.com/freebsd/freebsd-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the codes or adjust test infrastructure. The details of these efforts are available in the weekly CI reports.

During the third quarter of 2020, we continued working with the contributors and developers in the project to fulfill their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD.

Important changes:

All !x86 -test builds now trigger a new build on 22:00 UTC daily; this was not running very often because running all the tests in qemu takes lots of time. The work on improving the test execution speed and parallelism is in progress. The following is a list of the jobs affected:
The build and test results will be sent to the dev-ci mailing list soon. Feedback and help with analysis is very appreciated!
- A builder dedicated to run jobs using provisioned VMs is setup, this improves the stableness and reduces the execution time.
- The result of FreeBSD-head-amd64-test_zfs is changed after OpenZFS importing; we encourage everyone to check and fix the failing and skipped test cases.

New jobs added:

CI build for FreeBSD HEAD on PowerPC64LE.

Work in progress:

Collecting and sorting CI tasks and ideas here.
Testing and merging pull requests in the the FreeBSD-ci repo.
Designing and implementing pre-commit CI building and testing,
Reduce the procedures of CI/test environment setting up for contributors and developers.
Setting up the CI stage environment and putting the experimental jobs on it.
Setting up public network access for the VM guest running tests.
Implementing automatic tests on bare metal hardware.
Adding drm ports building tests against -CURRENT.
Planning to run ztest and network stack tests.
Adding more external toolchain related jobs.
Improving the hardware lab to be more mature and adding more hardware.
Helping more 3rd software get CI on FreeBSD through a hosted CI solution.
Working with hosted CI providers to have better FreeBSD support.

Please see freebsd-testing@ related tickets for more WIP information, and don't hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Ports Collection

Links
About FreeBSD Ports	URL: https://www.FreeBSD.org/ports/
Contributing to Ports	URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring	URL: http://portsmon.freebsd.org/index.html
Ports Management Team	URL: https://www.freebsd.org/portmgr/index.html

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

We passed the landmark of 40,000 ports in the Ports Collection and are now around 40,400 ports. The last quarter saw 9335 commits to the HEAD branch and 481 commits to the 2020Q3 branch by respectively 167 and 63 committers. There are currently 2525 open problem reports of which 595 are unassigned. Compared to last quarter, this means a slight decrease in activity and also a slight increase in open PRs.

During the last quarter we welcomed Rainer Hurling (rhurlin@) and said goodbye to Kevin Lo (kevlo@) and Grzegorz Blach (gblach@).

The last three months saw new default versions for Perl (5.32), PostgreSQL (12) and PHP (7.4). Various packages also got updated: Firefox to 81.0.1, Chromium to 84.0.4147.135, Gnome to 3.36, Xorg to 1.20.9, Qt5 to 5.15.0, Emacs to 27.1, KDE Frameworks to 5.74.0 and pkg itself to 1.15.8.

Never tired, antoine@ ran 30 exp-runs to test port version updates, on such diverse matters as:

Updating byacc in base to 20200330.
Check balancing of sed "y" command.
Use of brackets.
Removing the now redundant "port" argument from USES=readline.

FreeBSD Office team - 3rd quarter 2020 report

Links
The FreeBSD Office project	URL: https://wiki.freebsd.org/Office

Contact: FreeBSD Office team ML <office@FreeBSD.org>
Contact: Dima Panov <fluffy@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The FreeBSD Office team works on a number of office-related software suites and tools such as OpenOffice and LibreOffice.

Work during this quarter focused on providing the latest stable release of LibreOffice suite and companion apps to all FreeBSD users.

Alongside with updating old stable branch to latest 6.4.x releases, current ports-tree now have a full-featured cutting-edge 7.0.1 bundle.
Conservative users can keep 6.4.x stable version by switching to use all-in-one editors/libreoffice6 port and even with i18n language pack (off by default). It will be kept updated at least till 7.1.0 version is released.

We are looking for people to help the project.

All unstable work with LibreOffice snapshots is staged in our WIP repository.
The open bugs list contains all filed issues which need some attention. Patches, comments and objections are always welcome in the mailing list and bugzilla.

FreeBSD Graphics Team status report

Links
Project GitHub page	URL: https://github.com/FreeBSDDesktop

Contact: FreeBSD Graphics Team <x11@freebsd.org>
Contact: Niclas Zeising <zeising@freebsd.org>

The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications.

There have been several updates to the FreeBSD graphics stack and related libraries since the last report.

Most notably, MESA related ports were changed to use the meson build system, instead of the autotools based one. This was needed since mesa upstream has deprecated and removed the autotools build system, and this paved the way for further mesa updates. While there was a need for a few minor corrections after the initial update, this update has been successful and made it possible to further update and improve the FreeBSD mesa port.

There have also been several security fixes for xorg-server and libX11, so these ports have been updated to fix these issues.

During the period, FreeBSD 12 was changed to improve the compatibility with input devices using udev/evdev and libinput. This change removes the need for local configuration and makes most mice, touchpads and keyboards work out of the box. This change will be in the upcoming FreeBSD 12.2 release.

There have also been several updates to various libraries, both in the graphics and input stacks, and several userland drivers have been updated. Libraries such as libdrm and libevdev have been updated to include new FreeBSD support, developed by team members and added upstream.

There has also been ongoing work to keep the various drm-kmod ports and packages up to date, mostly in response to changes in various FreeBSD versions.

We have also continued our regularly scheduled bi-weekly meetings.

People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our gitter chat. We are also available in #freebsd-xorg on EFNet.

We also have a team area on GitHub where our work repositories can be found.

Projects

Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.

FreeBSD on Microsoft HyperV and Azure

Links
Microsoft Azure article on FreeBSD wiki	URL: https://wiki.freebsd.org/MicrosoftAzure
Microsoft HyperV article on FreeBSD wiki	URL: https://wiki.freebsd.org/HyperV

Contact: FreeBSD Integration Services Team <bsdic@microsoft.com>
Contact: Wei Hu <whu@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

Li-Wen is working on the FreeBSD release code related to Azure for the -CURRENT, 12-STABLE and 11-STABLE branches. The work-in-progress is available here. The 11.4-RELEASE image on Azure Marketplace is published. We are testing the releng/12.2 branch and 12.2-RELEASE image will be published to Azure Marketplace soon after released.

This project is sponsored by The FreeBSD Foundation, with resources provided by Microsoft.

Building FreeBSD on non-FreeBSD hosts

Links
Wiki	URL: https://wiki.freebsd.org/BuildingOnNonFreeBSD

Contact: Alex Richardson <arichardson@freebsd.org>

Until recently FreeBSD could only be built on a FreeBSD host. However, many popular free CI tools only allow building on Linux or macOS and therefore can not be used for building the FreeBSD base system. Furthermore, it is sometimes useful to cross-build FreeBSD for a remote machine or an emulator even if the build machine is not running FreeBSD. The goal of this project is to allow building the base system on Linux and macOS hosts.

I started this project in 2017 to allow building CheriBSD on the Linux servers and desktops that many of us working on the CHERI project use. The first few patches were upstreamed in 2018 (see the 2018q3 report) and I merged the full set of patches to CheriBSD shortly after. Over the past two years I have slowly been upstreaming the remaining patches and finally committed the last required change in time for this report.

As of September 2020 it should be possible to use the buildworld and buildkernel make targets to build a fully-functional FreeBSD installation on macOS and Linux hosts. We use this in our continuous integration system to build and test CheriBSD disk images for multiple architectures. I have also committed a GitHub Actions configuration upstream that takes approximately 10 minutes to build an amd64 kernel. This will ensure that changes that break crossbuilding from Linux/macOS can be detected easily.

Upstreaming the crossbuilding changes has resulted in various build system cleanups. For example, we now no longer need to use lorder.sh when building libraries which speeds up the linking step a bit. The portability and bootstrapping changes should also make it easier to upgrade from older versions since we no longer rely on host headers in /usr/include matching those of the target system (e.g. when bootstrapping localedef, etc.).

While this support for building on Linux and macOS should still be considered experimental, it should work in many cases. If you would like to give it a try, the following command line should successfully build an amd64 world on Linux and macOS systems that have packages for LLVM 10 (or newer) installed: MAKEOBJDIRPREFIX=/somewhere ./tools/build/make.py TARGET=amd64 TARGET_ARCH=amd64 buildworld Builds must be performed using the ./tools/build/make.py wrapper script since most Linux and macOS systems do not ship an appropriate version of bmake. Please let me know if you encounter any issues.

Sponsor: DARPA

Git Migration Working Group

Links
Git conversion tooling repo	URL: https://github.com/freebsd/git_conv
FreeBSD-git mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git
Beta doc git repo	URL: https://cgit-beta.FreeBSD.org/doc
Beta ports git repo	URL: https://cgit-beta.FreeBSD.org/ports
Beta src git repo	URL: https://cgit-beta.FreeBSD.org/src

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ulrich Spörlein <uqs@FreeBSD.org>

Work continues on FreeBSD's migration from Subversion to Git. Ulrich has addressed all known issues with svn2git and has been able to work around the inconsistent metadata and forced commit issues in the Subversion history.

We still have additional documentation to write, and need to finish installing commit hooks (e.g. restricting branch creation, or ensuring appropriate data exists on cherry-pick commits).

We expect to open the beta repository to test commits before the end of October. This is to allow testing of the commit hooks, and to allow developers to test access and become familiar with git operation. Commits in this repository will be deleted and the repository will be recreated at least once prior to the final migration.

Those with an interest in the migration to Git are encouraged to subscribe to the FreeBSD-git mailing list and test out the beta src, ports, and/or doc repositories.

You are also welcome check out the wiki, issues, README and other documentation at the Git conversion tooling repo.

We currently expect to transition the src and doc repositories in mid-November. Additional investigation and experimentation with the ports repository is still underway.

Sponsor: The FreeBSD Foundation (in part)

Linux compatibility layer update

Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>
Contact: Mark Johnston <markj@FreeBSD.org>

Earlier Linuxulator work focused on code cleanups and improving diagnostic tools. Work has now shifted from cleanups to fixing actual applications. Current status is being tracked at Linux app status Wiki page. Initial focus was on applications that don't involve X11, mostly because they tend to be easier to test and debug, and the bug fixes are not application-specific.

Foundation-sponsored work during this quarter included implementing a devfs(5) workaround to fix gettynam(3) inside jail/chroot, and workaround for the missing splice(2) syscall, which caused problems for grep and autotools. The Linux version reported to userspace was bumped to 3.10.0, which matches the kernel shipped with RHEL 7 and is neccessary for IBM's DB2 database installation to succeed. The BLKPBSZGET ioctl neccessary for Oracle database is supported now. There is now support for kcov(4), neccessary for syzcaller; as well as a number of fixes for issues reported by syzcaller, such as futex lock leaks. There were also more cleanups, including moving some Linuxulator-specific functionality related to error handling off from the syscall's fast code paths. The sysutils/debootstrap port, which provides an easy way to create Debian or Ubuntu jail, was updated to version 1.0.123. Finally there were some improvements to the documentation.

Most of those changes have been merged to FreeBSD 12-STABLE, in order to ship with 12.2-RELEASE.

There is increased involvement from other developers; this includes termios performance fixes, improved memfd support, implementing CLOCK_MONOTONIC_RAW required for Steam, madvise improvements, new compat.linux.use_emul_path sysctl. There is also ongoing work on tracking down the causes of failures related to Steam and WebKit, with fixes being first implemented in linuxulator-steam-utils.

Sponsor: The FreeBSD Foundation

LLDB Debugger Improvements

Links
Moritz Systems Project Description	URL: https://www.moritz.systems/blog/lldb-debugger-improvements-for-freebsd/
Git Repository	URL: https://github.com/moritz-systems/llvm-project

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

FreeBSD includes LLDB, the debugger in the LLVM family, in the base system. At present it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. It relies on an obsolete plugin model in LLDB that causes growing technical debt. This project aims to bring LLDB closer to a fully featured replacement for GDB, and therefore for FreeBSD to feature a modern debugger for software developers.

The legacy monolithic target supports the executed application being debugged in the same process space as the debugger. The modern LLDB plugin approach, used on other supported targets, executes the target process under a separate lldb-server process. This improves reliability and simplifies the process / thread model in LLDB itself. In addition, remote and local debugging will both be performed using the same approach.

After the migration to the new process model is complete, the project will include reviewing the results of LLDB's test suite and fixing tests as time permits. The work is expected to be complete in 2020.

The project schedule is divided into three milestones, each taking approximately one month:

1. Introduce new FreeBSD Remote Process Plugin for x86_64 with basic support and upstream to LLVM. 2. Ensure and add the mandated features in the project (process launch, process attach (pid), process attach (name), userland core files, breakpoints, watchpoints, threads, remote debugging) for FreeBSD/amd64 and FreeBSD/i386. 3. Iterate over the LLDB tests. Detect, and as time permits, fix bugs. Ensure bug reports for each non-fixed and known problem. Add missing man pages and update the FreeBSD Handbook.

We are nearing the completion of the first milestone. The new plugin is getting into shape, and it can already run simple single-threaded programs. The supported features include single-stepping, breakpoints, memory and register I/O on amd64. Both plugins are supported simultaneously. The new plugin is used if FREEBSD_REMOTE_PLUGIN environment variable is set to any value, or if lldb-server is spawned directly. Otherwise, the old plugin is used for compatibility. Once the new plugin matures, we are planning to enable it unconditionally on the architectures that it is ported to.

Sponsor: The FreeBSD Foundation

Lua usage in FreeBSD

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Kyle Evans <kevans@FreeBSD.org>
Contact: Ryan Moeller <freqlabs@FreeBSD.org>

During this quarter, flua (FreeBSD Lua) was taught where to find base .lua modules in order to support require of .lua modules to be provided by the base system. flua also gained support for require of binary modules.

A review for libjail bindings has also been submitted, pending review. libjail is an essential component if one wants to be able to write jail management utilities in flua.

libcrypt
libexpat
libnv
libxo

There is also a small list of scripts that would do well with a port to flua:

certctl(8)

NFS over TLS implementation

Contact: Rick Macklem <rmacklem@freebsd.org>

In an effort to improve NFS security, an internet draft which I expect will become an RFC soon specifies the use of TLS 1.3 to encrypt all data traffic on a Sun RPC connection used for NFS.

The coding for this project has now been completed. All required changes to the NFS and kernel RPC code have been committed to -CURRENT. The daemons are now believed to be complete, but will remain in base/projects/nfs-over-tls until -CURRENT has an OpenSSL library with the kernel TLS support incorporated in it. If this does not happen for FreeBSD-13, hopefully the patched OpenSSL and the daemons can become ports.

While setting up system(s) for testing is still a little awkward, the documentation is now available for those who want to help with testing.

The main limitation in the current implementation is that it uses TLS1.2 and not TLS1.3. This should change once the KERN_TLS rx patch includes TLS1.3 support.

Third party testing would be appreciated.

syzkaller on FreeBSD

Contact: Mark Johnston <markj@FreeBSD.org>

See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller.

syzkaller, especially the public syzbot instance, continues to find bugs in the FreeBSD kernel. A number of these bugs have been fixed in subsystems such as the VFS name cache, the TCP and SCTP stacks, pf(4), the unix domain socket implementation, and the Linuxulator.

The FreeBSD Foundation sponsored some work to enable cross-OS fuzzing. This makes it possible to fuzz the Linuxulator using syzkaller's Linux target. This effort quickly found several bugs; once the support is committed upstream we will hopefully be able to leverage syzbot to gain continuous testing of the Linux system call interface in addition to the native and 32-bit compatibility interfaces.

Some work was also done to enable running syzkaller in a FreeBSD jail, with the eventual aim of making it easy to distribute binary images containing everything required to immediately start running syzkaller on a new host. Currently a number of setup steps are required, making deployment somewhat painful.

Sponsor: The FreeBSD Foundation

Kernel

Updates to kernel subsystems/features, driver support, filesystems, and more.

DRM Drivers Update

Links
drm-kmod	URL: https://github.com/freebsd/drm-kmod/

Contact: Emmanuel Vadot <manu@FreeBSD.Org>

The drm drivers for FreeBSD 13-CURRENT have been updated to match Linux 5.4.62 Then graphics/drm-current-kmod have been updated to follow this LTS release of Linux.

For now graphics/drm-devel-kmod is also tracking this release but will be updated to a later revision of Linux drm drivers in the near future.

A lot of linuxkpi code was removed from the ports or replaced with a BSD licenced implementation.

Sponsor: The FreeBSD Foundation

DTS Update

Contact: Emmanuel Vadot <manu@FreeBSD.org>

DTS files (Device Tree Sources) were updated to be on par with Linux 5.8 for HEAD and 5.6 for the 12-STABLE branch.

DesignWare Ethernet adapter driver improvements

Links
WIP branch	URL: https://github.com/gonzoua/freebsd/tree/rk_eth

Contact: Oleksandr Tymoshenko <gonzo@FreeBSD.org>

DesignWare Ethernet adapter IP is used in Rockchip and Allwinner SoCs. The driver was updated with following fixes:

Initialize clocks instead of relying on u-boot to do the right thing.
Sense media type and adjust controller configuration accordingly.
Add support for RMII PHY mode.

Yet uncommitted changes include performance optimisation by adding

support for multi-segment mbuf transmission. The next step is to try to get more performance boost by using interrupt coalescence.

Google Summer of Code’20 Project - eBPF XDP Hooks

Links
Github diff link	URL: https://github.com/Ankurk99/freebsd/tree/ebpf-import
Project wiki	URL: https://wiki.freebsd.org/SummerOfCode2020Projects/eBPFXDPHooksl

Contact: Ankur Kothiwal <ankur@freebsd.org>

The eBPF eXpress Data Path (XDP) allows eBPF programs to be run to filter received packets as early as possible, avoiding unnecessary processing overhead before the filter is run. The goal of this project is to extend an existing FreeBSD network driver (a virtual NIC like a VirtIO if_vtnet) to be able to call into an eBPF program when processing a newly received packet. In short, with XDP the driver must PASS (accept and process normally), DROP, TX or REDIRECT the packet as specified by the program. eBPF helper functions and maps for aiding in packet filtering will also be implemented.

Implemented:

Register a eBPF probe when an interface is registered with pfil.
Activating eBPF probe.
Create hooks and link them to the pfil head when the eBPF XDP probe is activated and successfully list the XDP probes.
Create a xdp_rx function which will pass the received packets to the eBPF program where the packets can be further processed. This function will return XDP actions: DROP and PASS.
Register the xdp hook and link it to the pfil head.
Write an eBPF program to process (currently drop and pass) ICMP traffic - This is to test that the hook is working properly.
Write a loader function to load the ICMP filter program to the kernel.

Future Work:

Currently we can only attach the XDP hook to PASS and DROP the packets - The work on detaching the hook is left.
The XDP action to “TX” and “REDIRECT” the packets.

Final Deliverables:

Implemented XDP hook to pass and drop packets.
Created a loader program to attach the eBPF program to the kernel.
A test program to DROP ICMP filter.

This code was done under the Google Summer of Code 2020 under the guidance

of Ryan Stone (rstone@). The eBPF implementation for FreeBSD is still a work in progress and FreeBSD doesn’t support eBPF yet. The basic implementation for eBPF was a GSoC’18 project, and is still under development. This project is based on that implementation so the XDP implementation for FreeBSD can only be merged into the FreeBSD source code once it supports eBPF.

Currently this code is a work in progress and is merged to Ryan Stone’s branch with support for the eBPF implementation.

Sponsor: Google Summer of Code

ENA FreeBSD Driver Update

Links
ENA README	URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

Completed since the last update:

Fix ENA compilation in case it is integrated into the kernel binary.
MFC of the ENA v2.2.0 driver to the FreeBSD 12.2.

Work in progress:

Add feature that allows reading extra ENI (Elastic Network Interface) metrics about exceeding BW/pps limits.
Introduce full kernel RSS API support.
Allow reconfiguration of the RSS indirection table and hash key.
Evaluation and prototyping of the driver port to the iflib framework.

Sponsor: Amazon.com Inc

IPSec Extended Sequence Number (ESN) support

Contact: Grzegorz Jaszczyk <jaz@semihalf.com>
Contact: Patryk Duda <pdk@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

Extended Sequence Number (ESN) is IPSec extension defined in RFC4303 Section 2.2.1. It makes possible to implement high-speed IPSec implementations where standard, 32-bit sequence number is not sufficient. A key feature of the ESN is that only low order 32 bits of sequence number are transmitted over the wire. High-order 32 bits are maintained by sender and receiver. Additionally high-order bits are included in the computation of Integrity Check Value (ICV) field.

Extended Sequence Number support contains following:

Modification of existing anti-replay algorithm to fulfil ESN requirements.
Trigger soft lifetime expiration at 80% of UINT32_MAX when ESN is disabled.
Implement support for including ESN into ICV in cryptosoft engine in both encrypt and authenticate mode (eg. AES-CBC and SHA256 HMAC) and combined mode (eg. AES-GCM).
Implement support for including ESN into ICV in AES-NI engine in both encrypt and authenticate mode and combined mode.

Completed since the last update:

Adjust implementation of crypto part to the reworked Open Crypto Framework.
Move the core ESN implementation from the crypto drivers to netipsec layer.
Make use of the newly introduced crp_aad mechanism for combined modes.
Introduce minor fixes and improvements.

TODO:

Complete review process in Phabricator and merge patches in the tree.

Sponsor: Stormshield

NXP ARM64 SoC support

Contact: Marcin Wojtas <mw@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Dawid Gorecki <dgr@semihalf.com>

The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC

Completed since the last update:

Upstreaming of the QorIQ SDHCI driver (r365054).

With above the current Semihalf upstreaming activity is complete.

The major out-of-tree supported components:

DPAA network controller support.
QSPI controller support.

They work on 11.2-RELEASE, but still require significant

effort to adopt to FreeBSD-CURRENT.

Sponsor: Alstom Group

Addition of PowerPC64LE Architecture

Links
Early notes	URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-August/012043.html
Announcement	URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-September/012098.html

Contact: Brandon Bergren <bdragon@freebsd.org>

As of r366063, experimental support for little-endian PowerPC64 (PowerPC64LE) is available in -CURRENT for POWER8 and POWER9 machines.

In 2010, when FreeBSD was ported to PowerPC64, the average user would have been using a G5 PowerMac, a purely big-endian machine.

While, at the time, a 32-bit PowerPC machine could run in little-endian, as well as POWER6 and POWER7, in practice, the complexities involved in managing it at the kernel level and lack of firmware support made it infeasible to support.

When IBM designed POWER8, one main focus was to improve little-endian support, and bring it up to parity with big-endian.

This improved support makes it practical to support a little-endian operating environment on what is traditionally a primarily big-endian platform.

In 2020, with POWER9 being affordable for many users thanks to the Raptor Blackbird, semi-easy access to surplus POWER8 hardware, IBM having a major future focus on POWER little-endian, and the decay of big-endian support in modern video cards and graphical environments, there is demand for a little-endian version of FreeBSD on POWER.

With FreeBSD/PowerPC64's transition in 2019 to the ELFv2 ABI as part of the 2019q4 PowerPC on Clang effort, the last major barrier to a little-endian port was eliminated.

Since nobody else was working on it, and I had the skillset required to do the port, I decided to experiment one weekend with a little-endian kernel to see how difficult it would be to port.

It turned out to be a lot more trivial than I was expecting. Three days later I had console support in qemu, and after another week of debugging, I had it fully up and running on hardware.

FreeBSD PowerPC64LE is now an experimental MACHINE_ARCH in base, and is continuing to evolve at a rapid pace.

Big-endian PowerPC64 is still the preferred platform for the foreseeable future, and will not be deprecated.

Sponsor: Tag1 Consulting, Inc.

ure - USB 3.0 Gigabit Ethernet Driver update

Links
svn commit: r365648	URL: https://svnweb.freebsd.org/changeset/base/365648
FreeBSD-SA-20:27.ure	URL: https://www.freebsd.org/security/advisories/FreeBSD-SA-20:27.ure.asc
D25809 major update to if_ure	URL: https://reviews.freebsd.org/D25809

Contact: John-Mark Gurney <jmg@FreeBSD.org>

The ure is a driver for handling the RealTek ethernet adapters, including the RTL8153 USB 3.0 Gigabit ethernet adapters. It is used in many ethernet dongles and docking stations.

Previous to this update, the driver was limited in speed. In my testing, I was only able to get ~91Mbps. This limit was due to one packet per USB transfer. USB has a limit of 8000 transfers per second (1500 bytes/pkt * 8000 pkts/sec * 8bits/byte == 96 Mbps). This was acceptable for fast ethernet (RTL8152, 100Mbps), but with the additional support for Gigabit ethernet, it became a bottleneck.

The updates add sending and receiving multiple packets in a single USB transfer, VLAN hardware tagging, and enable TCP and UDP checksum offloading. This increased the speed on gigabit ethernet to ~940 Mbps.

In doing this work, a security vulnerability was discovered in the driver. Due to improper setting of a device register, on some devices, it caused packets to be fragmented when they shouldn't be and the driver was unable to handle them correctly. This allowed an attacker, who could generate large frames (say, ping packets, or large TCP transfers), to inject arbitrary packets into the network stack. This could allow the attacker to spoof traffic from other machines, and bypass VLAN protections. See the SA for more information.

As part of this work, a script was created to run tests to validate that basic functionality of the driver (w/o options) work properly, and then iterate over each option to make sure that they function properly. This will be released at some point in the future.

If you're interested in helping out, or testing it, let me know.

Stateless hardware offloads for VXLANs

Links
r365867	URL: https://svnweb.freebsd.org/base?view=revision&revision=r365867
r365868	URL: https://svnweb.freebsd.org/base?view=revision&revision=r365868
r365869	URL: https://svnweb.freebsd.org/base?view=revision&revision=r365869
r365870	URL: https://svnweb.freebsd.org/base?view=revision&revision=r365870
r365871	URL: https://svnweb.freebsd.org/base?view=revision&revision=r365871
RFC6935	URL: https://tools.ietf.org/html/rfc6935

Contact: Navdeep Parhar <np@FreeBSD.org>

VXLAN (Virtual eXtensible LAN) is a tunneling protocol in which Layer 2 traffic for a virtual LAN is encapsulated in UDP and transferred over Layer 3 networks between VTEPs (VXLAN Tunnel End Points). Traffic on the wire has two sets of networking headers: the headers for the encapsulation and the headers of the traffic being encapsulated. VXLANs are supported by if_vxlan(4) on FreeBSD.

Modern NICs commonly support header checksum insertion and verification, TSO (TCP Segmentation Offload) on transmit, and RSS for load distribution on receive. But the default is to operate on the outermost headers. Some NICs can operate on the inner encapsulated frames as well. The commits listed above allow if_vxlan(4) to take advantage of such NICs.

r365867 and r365868 add new mbuf checksum flags and ifnet capabilities. r365870 implements the kernel parts of the new capabilities and updates if_vxlan(4) to make use of them. r365871 implements driver support for the new capabilities in cxgbe(4).

VXLAN and other tunneling protocols that use UDP explicitly allow zero checksum in the outer UDP header, even with IPv6. r365869 adds support for configuring one UDP/IPv6 port where zero checksums are allowed.

This work was sponsored by Chelsio Communications and was implemented and tested using T6 (Terminator 6) NICs supported by cxgbe(4). It is available in 13.0-CURRENT (head) right now and will be available in 12-STABLE in the future.

VXLANs can be created as usual and will automatically have checksum and TSO capabilities if the underlying physical interface supports VXLAN stateless offloads. Use ifconfig to list, disable, and enable checksum capabilities on the VXLAN interface. Use https://bugs.freebsd.org/bugzilla/ to report bugs.

Future work:

Direct call into a vxlan input routine from the driver's receive routine.
LRO support in if_vxlan(4).
GENEVE support.

Sponsor: Chelsio Communications

Wireless updates

Links
The freebsd-wireless mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless
athp github repository	URL: https://github.com/erikarn/athp

Contact: Adrian Chadd <adrian@FreeBSD.org>
Contact: Bjoern A. Zeeb <bz@FreeBSD.org>

The following works happened in FreeBSD HEAD (some already in Q2) and were merged for 12.2-BETA2 and include net80211 and driver updates for better 11n and upcoming 11ac support.

In more detail, this includes an ath(4) update, some run(4) 11n support, 11n for otus(4), A-MPDU, A-MSDU, A-MPDU+A-MSDU and Fast frames options, scanning fixes, enhanced PRIV checks for jails, restored parent device name printing, improvements for upcoming VHT support, lots of under-the-hood infrastructure improvements, new device IDs, and debug tools updates.

If you have a chance please test before the release.

Atheros 11ac driver athp

In the last three months the athp(4) port of the ath10k driver has progressed well. Adrian reports the following important changes:

Per-node transmit buffering was implemented, required for correct hostap and QCA6174 behaviour.
Issues with ignoring sending some management frames got fixed; null-data frames were being filtered out and this caused undesirable hostap behaviour.
Transmit path refactoring reduced code duplication.
A fix on firmware start / VAP running tracking no longer stops the first VAP from coming active after VAP creation / ifconfig up.
Correcting hostap mode PHY configuration now allows non-VHT stations to associate and correctly exchange data with a VHT AP.
Addition of a crypto key configuration cache in the driver ensures the ieee80211_key details are available after the key is deleted; net80211 would reuse or free the state before the driver task would finish the firmware command.

Newer Intel Wireless device support

Initial work was done to integrate net80211 support in the LinuxKPI compat layer to get the wireless parts going. In addition, upstreaming code changes and working through problems and review started on two sides. One was trying to get mostly compile time changes upstreamed to the iwlwifi driver. The other is sorting out conflicting LinuxKPI changes to not break the DRM graphics drivers. Bjoern hopes that with some of that sorted out, he can soon go back to focus on the wireless parts and produce a new snapshot.

rtw88 and brcmfmac

As the Intel driver port and LinuxKPI advance, both the rtw88, and to a lower degree the brcmfmac, ports benefit from that. Bjoern lately also got a brcmfmac PCIe card and started to port support for that. This for the moment remains a free-time project.

Work by Bjoern was sponsored by: Rubicon Communications, LLC (d/b/a "Netgate") and The FreeBSD Foundation

ZSTD Compression in ZFS

Contact: Allan Jude <allanjude@freebsd.org>

This effort to complete the integration of ZSTD into ZFS is funded by the FreeBSD Foundation.

During the third quarter the integrating of ZSTD into OpenZFS was completed in the upstream OpenZFS repository, and the new OpenZFS 2.0 codebase was imported into 13-CURRENT. Completed milestones in this project:

Importing ZSTD 1.4.5 into OpenZFS, using the recent upstream zstd features that make it easier to embed zstd in other projects.
Changing the way compression levels are tracked and inherited.
Save and restore the compression level via an embedded block header.
Also store the version of zstd used in the embedded block header, for future-proofing. The checksum of a block may not match if zstd is upgraded, since it may compress the block more.
Add tests to ensure zstd compression and metadata survive ZFS replication.
Resolve possible negative interactions with L2ARC and ZFS Native Encryption.
Fix bug with L2ARC if the Compressed ARC feature is disabled.
Improve the ZFS feature activation code, so that zstd cannot create pools that will panic older versions of ZFS.

With these changes, upgraded pools can compress data with zstd

or zstd-fast, across a wide range of different compression levels. This will allow the storage administrator to select the performance-vs-compression tradeoff that best suits their needs.

Tasks remaining to be completed:

Add a section to the FreeBSD Handbook ZFS chapter about zstd
Create more documentation around selecting a suitable compression level
Finish support for ZSTD in the FreeBSD boot loader (Warner Losh imp@freebsd.org)

Sponsor: The FreeBSD Foundation

Architectures

Updating platform-specific features and bringing in support for new hardware platforms.

CheriBSD 2020 Q3

Links

Contact: Alex Richardson <arichardson@FreeBSD.org>
Contact: Andrew Turner <andrew@FreeBSD.org>
Contact: Brooks Davis <brooks@FreeBSD.org>
Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>
Contact: George Neville-Neil <gnn@FreeBSD.org>
Contact: Jessica Clarke <jrtc27@FreeBSD.org>
Contact: John Baldwin <jhb@FreeBSD.org>
Contact: Robert Watson <rwatson@FreeBSD.org>
Contact: Ruslan Bukin <br@FreeBSD.org>

CheriBSD extends FreeBSD to implement memory protection and software compartmentalization features supported by the CHERI instruction-set extensions. There are three architectural implementations of the CHERI protection model: CHERI-MIPS, CHERI-RISC-V, and Arm's forthcoming experimental Morello processor (due late 2021). CheriBSD is a research operating system with a stable baseline implementation into which various new research features have been, or are currently being, merged:

Arm Morello - We are preparing to open source our adaptation of CheriBSD to Arm's Morello architecture. The Morello branch is being updated to the most recent CheriBSD baseline, and patches are in review for upstreaming to our open-source repository. CheriBSD currently boots and runs statically linked CheriABI binaries on the Morello simulator, and dynamic linking support is in progress, with OS and toolchain bugs being worked on. We aim to make a first CheriBSD/Morello snapshot available alongside other open-source Morello software in mid-October 2020, however, our target for a more mature and usable implementation is December 2020.
Kernel spatial memory safety (pure-capability kernel) - The current CheriBSD kernel is a hybrid C program where only pointers to userspace are CHERI capabilities. This ensures that the kernel follows the intent of the application runtime and cannot be used to defeat bounds on application pointers. We have developed and will soon merge a pure-capability kernel where all pointers in the kernel are appropriately bounded capabilities. This vastly reduces the opportunity for buffer overflows. This spatial memory safety lays the groundwork for future work such as device driver compartmentalization and kernel temporal safety.
Userspace heap temporal memory safety (Cornucopia) - CHERI capabilities provide the necessary features to enable robust and efficient revocation of freed pointers. With Cornucopia we have implemented a light-weight revocation framework providing protection from use-after-reallocation bugs with an average cost below 2%. We aim to bring these overheads down further over the next year and merge this functionality into the mainline CheriBSD.
We have been working on updating the arm64 bhyve from Politehnica University of Bucharest to have it committed to FreeBSD. We have been upstreaming initial changes to help support this.
Baseline FreeBSD improvements - We are upstreaming (to FreeBSD) various bug fixes and tweaks for PCIe support, and support for the System MMU (SMMU) that will be present on the N1SDP and Morello SoCs. We have upstreamed support for cross-building FreeBSD from macOS and Linux (with some limitations; see separate entry on crossbuilding). We have also fixed implementation bugs in the RISC-V ABI.

CHERI Documentation and Exercises

We have released [Capability Hardware Enhanced RISC Instructions: CHERI - Instruction-Set Architecture (Version 8)](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-951.pdf). +
We have released Capability + Hardware Enhanced RISC Instructions: CHERI Instruction-Set Architecture (Version 8). Notable changes include promotion of CHERI-RISC-V to non-experimental and discussion of Arm's Morello prototype.
We have developed a set of [Adversarial CHERI Exercises and - Missions](https://ctsrd-cheri.github.io/cheri-exercises) to introduce security +
We have developed a set of Adversarial + CHERI Exercises and Missions to introduce security researchers to CHERI protections.

FreeBSD/RISC-V Project

Links
Wiki	URL: https://wiki.freebsd.org/riscv

Contact: Mitchell Horne <mhorne@FreeBSD.org>

Contact: freebsd-riscv Mailing List
Contact: IRC #freebsd-riscv on freenode

The FreeBSD/RISC-V project is providing support for running FreeBSD on the RISC-V Instruction Set Architecture.

This quarter saw several important bug fixes. A number of hangs in the system were identified and addressed, and a bug in QEMU's implementation of the Platform Level Interrupt Controller was fixed. This fix is included in the new devel/qemu50 and devel/qemu-devel ports.

The end result of these fixes is that the test suite can now be reliably run to completion in QEMU. The entire run takes several hours, so CI has been configured to run the job once a day. There is active effort into reducing the time it takes to run the entire test suite.

A new u-boot port was created: sysutils/u-boot-qemu-riscv64. This variant can be used as a secondary bootloader alongside OpenSBI to load and launch FreeBSD's loader(8) from an EFI System Partition.

Next quarter will likely bring further fixes to address some of the failing test cases.

Ports

Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves.

Update to grub-bhyve

Links
grub-bhyve Git Repository	URL: https://gitlab.com/ctuffli/grub

Contact: Chuck Tuffli <chuck@freebsd.org>

bhyve is the hypervisor included in FreeBSD and other operating systems used to run virtual machines. When not using a boot ROM (i.e. UEFI), the user must load the guest operating system for bhyve. For non-FreeBSD guests, the loader is a version of GNU GRUB (a.k.a the GNU GRand Unified Bootloader) modified to interface with bhyve. This work is an effort to both update the base GRUB code to the latest version as well as improve the usability on FreeBSD.

The current grub-bhyve is based on an older version of GRUB (circa 2015) and thus is missing more recent additions such as XFS file system and syslinux support. With the update, installing CentOS, for example, now does not require the extra step of changing the default file system to something other than XFS.

Internally, the code has been restructured to be its own "platform" which should make it easier to keep in sync with upstream development. The major improvement is the ability to automatically find and load the GRUB configuration file from the guest disk image. With this change, it is not necessary to create a device map file or specify which Linux kernel or initrd image to use. More importantly, if the guest image updates its GRUB configuration, for example after updating the kernel, no changes are needed when invoking grub-bhyve. Note, this feature requires a new "disk" option:

# grub-bhyve --disk=/zroot/vms/u18-mini/disk0.img --vm=u18-mini

The automatic configuration file detection works with both GRUB configuration files (e.g. CentOS, Ubuntu) as well as syslinux configuration (e.g. Alpine). For the adventurous, there is experimental support for Fedora's BootLoaderSpec (a.k.a. blscfg) on the blscfg branch of the grub-bhyve Git repository.

The code has been tested on a few Linux variants, but it would benefit from wider testing (and bug reports!). The new version does not have a Port but is easily built on FreeBSD. After cloning / downloading the source, run:

$ PYTHON=python3.7 ./bootstrap $ MAKE=gmake ./configure --with-platform=bhyve $ gmake

The resulting binary, grub-bhyve, will be in the grub-core/ directory. If you have success or troubles with it, please let me know.

KDE on FreeBSD

Links
KDE FreeBSD	URL: https://freebsd.kde.org/
KDE Community FreeBSD	URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, an IDE with the name KDevelop, a PIM suite known as Kontact and hundreds of other applications that can be used on any FreeBSD machine.

With the continuation of the ever-so-peculiar era of almost-only-online, the KDE community has shifted gears and also gone for online events. The yearly conference, Akademy, was conducted online over video calls. Meanwhile, software continues to be released, so this quarter the kde@ team:

Put the beta of the next version of KDE Plasma, scheduled for official release in October 2020, into the Area51 development tree. Area51 is a fork of the FreeBSD ports tree where new development for KDE ports happens.
The monthly regular updates to the KDE Plasma desktop landed on-time and safely.
With three months in a quarter, there were also three releases of KDE Frameworks 5, including a new framework for handling DAV jobs.
The June applications update and its .1 release landed a bit late, but brings with it the usual raft of updates to KDE applications and libraries,
A new Digikam release, which arrived in the ports tree on the day of its release.
A new KDevelop release arrived a day after its release. This update contains a number of crash fixes for refactoring support.
Qt was updated to Qt 5.15, the last in the Qt5 series and an LTS version. Bugfix releases are expected, but the next major Qt will be Qt 6.

On the infrastructure front, August saw some minor updates to CMake and ninja.

As usual, kde@ continues to support the work of xorg@ and gnome@ in maintaining the Free Desktop stack on FreeBSD, including XOrg, poppler, and xdg-utils. A new MAINTAINER group, desktop@, has been created to provide shared ownership of that shared stack.

With Python2 deprecation looming, the build system for QtWebEngine -- itself a fork of Chromium -- is becoming a pressing issue in Q4 and will no doubt chew up a lot of time in the coming months.

Documentation

Noteworthy changes in the documentation tree, in manpages, or in external books/documents.

DOCNG on FreeBSD

Links
DOCNG Website Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-website
DOCNG Documentation Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-documentation
DOCNG Share Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-data

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

The Doc New Generation project aims to convert the website and all existing documentation to Hugo/AsciiDoctor. Right now almost everything is converted as you can see in the repositories.

The objective of using Hugo and AsciiDoctor is to reduce the learning curve and let people to start quickly with our documentation system. Other benefits of using Hugo is that we can use other technologies aside from AsciiDoctor, like MarkDown, RST, Pandoc, etc.

The remaining tasks include:

Finish the conversion of some books to AsciiDoctor.
Get some tweaks in the CSS to be responsive.
Add AsciiDoctor extensions to create an index of tables and figures.
Make a general review.

The dates for making the migration have yet to be discussed.

Patches, comments and objections are always welcome.

Third-Party Projects

Potluck - Flavour & Image Repository for pot

Links
Potluck Repository & Project	URL: https://potluck.honeyguide.net/
Potluck on github	URL: https://github.com/hny-gd/potluck
pot project	URL: https://pot.pizzamig.dev

Contact: Stephan Lichtenauer <sl@honeyguide.eu>

pot is a jail management tool that also supports orchestration through nomad.

Potluck aims to be to FreeBSD and pot what Dockerhub is to Linux and Docker: A repository of pot flavours and complete images for usage with pot.

In the last quarter, an initial set of Nomad, Consul and Traefik images has been created that are sufficient to run a simple virtual datacenter out of the box.
A three-part article series explaining how to set this up is also available now.

Furthermore, ready-made images suitable for scheduling via Nomad and Consul in such an environment have been created, e.g. a BackupPC or a Postfix Backup MX service.

Future plans include additional images and exposing more configuration options in the existing images to allow a more flexible usage.

Beside general feedback and tests, additional flavours and patches are very welcome!

Sponsors: Honeyguide GmbH & Honeyguide Group (Pty) Ltd

Puppet

Links
Puppet	URL: https://puppet.com/docs/puppet/latest/puppet_index.html
Puppet's FreeBSD slack channel	URL: https://puppetcommunity.slack.com/messages/C6CK0UGB1/
Bolt	URL: https://puppet.com/docs/bolt/latest/bolt.html
Choria	URL: https://choria.io/

Contact: Puppet Team <puppet@FreeBSD.org>

Since out last status report a few years ago, the puppet@ team regularly updated the various Puppet ports to follow upstream releases of Puppet 4, Puppet 5 and Puppet 6. Puppet 4 was removed when it reached EOL.

More recently, an effort was made to enhance Facter 4 so that it can be used as a drop-in replacement of Facter 3 on FreeBSD. Facter 4 is a Ruby rewrite of Facter 3, the C++ rewrite of Facter 2 which was initially in Ruby. As a consequence we have two ports for Facter: sysutils/facter is the C++ implementation (Facter 3) and sysutils/rubygems-facter is the Ruby implementation (updated from Facter 2 to Facter 4 a few weeks ago). The Puppet 5 and Puppet 6 ports already allow to choose which version of Facter to use. Facter 4 will be the default version of Facter with Puppet 7 which is expected to be released soon.

We are getting ready to add a port for Puppet 7 as sysutils/puppet7 when it is available, along with PuppetServer 7 (sysutils/puppetserver7), and PuppetDB 7 (databases/puppetdb7).

Regarding orchestration, most Marionette Collective ports have been deprecated for a long time, and the last component sysutils/mcollective is expected to be deprecated soon: Marionette Collective was not shipped anymore with Puppet 6 and Bolt has been made available as a lightweight replacement.

Bolt is already available in the ports tree as sysutils/rubygems-bolt), but if you are using Marionette Collective, you are invited to look into Choria which will reach the ports tree soon as sysutils/choria. Choria is a direct evolution of Marionette Collective allowing a smooth transition from MCollective. Once Choria is available in the ports tree, Marionette Collective will be deprecated.

News Home | Status Home

Skip site navigation (1) Skip section navigation (2)

Header And Logo

Donate to FreeBSD

Peripheral Links

Site Navigation

Section Navigation

Introduction

This report covers FreeBSD related projects for the period between October and December, and is the fourth of four planned reports for 2020.

This quarter had quite a lot of work done, including but certainly not limited to, in areas relating to everything from multiple architectures such as x86, aarch64, riscv, and ppc64 for both base and ports, over kernel changes such as vectored aio, routing lookups and multipathing, an alternative random(4) implementation, zstd integration for kernel dumps, log compression, zfs and preparations for pkg(8), along with wifi changes, changes to the toolchain like the new elfctl utility, and all the way to big changes like the git migration and moving the documentation from DocBook to Hugo/AsciiDoctor, as well as many other things too numerous to mention in an introduction.

This report with 42 entries, which don't hold the answer to life, the universe and everything, couldn't have happened without all the people doing the work also writing an entry for the report, so the quarterly team would like to thank them, as otherwise, we wouldn't have anything to do.

Please note that the deadline for submissions covering the period between January and March is March 31st.

We hope you'll enjoy reading as much as we enjoyed compiling it.
Daniel Ebdrup Jensen, on behalf of the quarterly team.

FreeBSD Team Reports

FreeBSD Foundation
FreeBSD Release Engineering Team
Cluster Administration Team
Continuous Integration
Ports Collection
Office Hours

Projects

GPL in Base
Git Migration Working Group
Linux compatibility layer update
LLDB Debugger Improvements
Upstreaming NetApp Changes
NFS over TLS implementation
OpenBSM Synchronisation
Tool Chain

Kernel

ENA FreeBSD Driver Update
Intel wireless update
Fenestras X random(4)
pf performance improvement
IP Routing lookup improvements
Scalable routing multipath support
Thunderbolt3/USB4 stack
Vectored AIO
ZSTD Compression in ZFS

Architectures

arm64 platform updates
FreeBSD/RISC-V Project

Userland Programs

Dual-stack ping command

Ports

KDE on FreeBSD
FreeBSD Office team
Ports On Non-x86 Architectures
Python 2.7 removal from Ports
Xfce on FreeBSD

Documentation

FreeBSD Translations on Weblate
DOCNG on FreeBSD

Miscellaneous

Prometheus NFS Exporter

Third-Party Projects

FreeBSD Aarch64 under VMWare ESXi-ARM Fling
Bastille
CheriBSD
Embedded Lab Project
FreshPorts
helloSystem
K8S-bhyve
Puppet

FreeBSD Team Reports

Entries from the various official and semi-official teams, as found in the Administration Page.

FreeBSD Foundation

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

Here are some highlights of what we did to help FreeBSD last quarter:

COVID-19 Impact to the Foundation

Like most organizations, we transitioned all of our staff to work from home. We also put a temporary ban on travel for staff members, which didn't affect our output too much, since most conferences went virtual. We continued supporting the community and Project, even though some of our work and responses may have been delayed because of changes in some of our priorities and the impact of limited childcare for a few of our staff members.

Partnerships and Commercial User Support

We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Not surprisingly, the stay at home orders, combined with our company ban on travel during Q4 made in-person meetings non-existent. However, the team was able to continue meeting with our partners and commercial users virtually. These meetings help us understand some of the applications where FreeBSD is used.

An event we help plan and organize, that helps with vendor/developer engagement, is the annual Bay Area Vendor Summit. We weren't going to let a pandemic stop us from holding this invaluable yearly event, so we went virtual! From the feedback we received from the vendor community on how we should run this, so it would be beneficial for them, we decided to hold this over 3 half days in November. One unexpected result was that more commercial users from around the world attended. Since a Vendor/Developer Summit is typically invitation only, we opened this up to FreeBSD contributors from around the world to watch the livestream. Because of the success and excitement of this event, we are planning to hold another one around June or July.

Fundraising Efforts

We want to take a moment to say thank you to all the individuals and corporations that stepped up to help fund our efforts last year. As of this writing, we raised $1,235,926, and will have the final tally by mid-January. The companies that gave generous financial contributions include Arm, NetApp, Netflix, Juniper Networks, Beckhoff, VMware, Stormshield, Tarsnap, and Google. We also want to say thank you to the Koum Family Foundation for awarding us a large grant, and to the employees of Nginx who also made generous financial contributions.

We truly appreciate these large contributions, which makes the most impact on how much we can contribute back to the Project. However, it's the individual donations that have the most meaning to us. Those are the folks who are giving because they trust we will invest their personal donations, whether large or small, into improving the operating system and Project. As stewards of your donations, we want to thank you for your trust in us and your commitment to making FreeBSD the best platform for products, education, research, computing, and more.

You'll find out how we used your donations for Q4 in our report, as well as in individual reports throughout this status report.

Though we know this is a Q4 status report, we are excited about our plans for 2021, including growing our software development team! We'll be posting two job descriptions for a Senior Software Developer and Project Coordinator soon.

Please consider making a donation to help us continue and increase our support for FreeBSD in 2021: https://www.FreeBSDfoundation.org/donate/.

We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information at https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/ and share with your companies!

OS Improvements

The Foundation provided many project grants over the last quarter, and you can read about OpenZFS Zstd support, Linuxulator application compatibility improvements, LLDB target support, test lab infrastructure, and WiFi projects in other entries in this quarterly report.

The Foundation hired six co-op students from the University of Waterloo during the 2020 fall term, as well as one intern. Former co-op student Tiger returned, and new students Yang and Zac joined us for the first time.

Tiger worked on improvements to the code-coverage guided kernel fuzzing tool Syzkaller, adding new system call definitions so that Syzkaller can expand the code it tests. A number of FreeBSD kernel bug fixes have already resulted from this work. Tiger also contributed a number of improvements to the ELF Tool Chain set of binary utilities, and worked on tooling to run tests from other tool suites against ELF Tool Chain.

Zac worked on an improvement to the pkg package management tool, investigating and upstreaming patches for FreeBSD support in FreePBX, and investigating compiler support for addressing the stack clash vulnerability.

Yang investigated and fixed a compilation bug with the kernel's Skein-1024 assembly implementation (used by ZFS), and then a number of projects related to Capsicum: applying Capsicum to sort(1), implementing a Capsicum service to execute utilities, and finally working with developers of the Game of Trees (got) version control system to adapt it for Capsicum support.

Our intern Ka Ho focused on improving the desktop experience of the FreeBSD. He fixed and improved many items of OBS (Open Broadcaster Software) on FreeBSD, worked on FreeBSD native audio support on Firefox, adding a facility that user-space audio programs could make use of to enumerate a list of audio devices. He also ported the fcitx5 input method framework.

The five Foundation staff members continued contributions in 2020 in both ongoing operational tasks (including the Git working group and security team) and software development for a number of projects.

Staff members responded to reported security vulnerabilities and release errata, prepared patches, and participated in the security advisory process. We also worked on proactive security vulnerability mitigations. Syzkaller also provided many reports of kernel issues that resulted in Foundation-sponsored bug fixes. We worked on several issues relating to FreeBSD/arm64 to move it along the path of being a Tier-1 architecture.

We participated in code reviews and supported community members in integrating changes into FreeBSD, and triaged incoming bug reports.

We contributed enhancements to many kernel and userland subsystems, including the x86 pmap layer, ELF run-time linker and kernel loader, the Capsicum sandboxing framework and Casper services, the threading library, some RISC-V changes, the build system, tool chain and freebsd-update, network stack stability improvements, machine-dependent optimizations, new kernel interfaces, DTrace bug fixes, documentation improvements, and others.

Continuous Integration and Quality Assurance

The Foundation provides a full-time staff member and funds projects on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD Project.

During the fourth quarter of 2020, Foundation staff continued improving and monitoring the Project's CI infrastructure, and working with experts to fix the failing builds and the regressions found by tests. The work was focused on pre-commit tests and development of the CI staging environment. The other main working item is working on the VCS migration to change the src and doc source from Subversion to Git. There are also many work-in-progress tasks like analysis and improve the tests of non-x86 platforms.

See the FreeBSD CI section of this report for completed work items and detailed information.

Supporting FreeBSD Infrastructure

The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. We coordinated efforts between the new NYI Chicago facility and clusteradm to start working on getting the facility prepared for some of the new FreeBSD hardware we are planning on purchasing. NYI generously provides this for free to the Project. We also worked on connecting with the new owners of the NYI Bridgewater site, where most of the existing FreeBSD infrastructure is located.

Some of the purchases we made for the Project last quarter to support infrastructure includes:

5 application servers to run tasks like bugzilla, wiki, website, cgi, Phabricator, host git, etc.
1 server to replace the old pkg server, which will provide a lot more IOPS to avoid the slowdowns seen during peak times of the day where the disks simply cannot keep up with the request volume.
1 server for exp-runs and to make them faster.
1 server to build packages more frequently.

FreeBSD Advocacy and Education

While we were still unable to attend in-person meetings due to COVID-19, we were able to attend virtual events at new venues and facilitate the first online FreeBSD Vendor Summit. In addition to attending and planning virtual events, we are continually working on new training initiatives and updating our selection of how-to guides to facilitate getting more folks to try out FreeBSD.

Check out some of the advocacy and education work we did last quarter:

Continued our FreeBSD Fridays series of 101 classes. Topics included an Introduction to Capsicum, Introduction to Bhyve, Introduction to DTrace, and more. Videos of the past sessions can be found here. We'll be back with new sessions in early 2021.
Gave a FreeBSD talk at the nerdear.la conference on October 20th.
Participated in the podcast: What the Dev: A Dive into the FreeBSD Foundation on its 20th Anniversary
Promoted the Foundation's 20th Anniversary in the FossBytes article: 20 Years of The FreeBSD Foundation
Continued to promote the FreeBSD Office Hours series. Videos from the one hour sessions can be found on the Project's YouTube Channel. See the Office Hours section of this report for more information.
Added two new How-To Guides: Contributing FreeBSD Documentation and How to Submit a Bug Report.
Worked with the organizing committee to host the November 2020 Vendor Summit
Promoted the use of FreeBSD in regards to CHERI and ARM's Morello Processor
Authored a Beginners Guide to FreeBSD for Fosslife.
Sponsored All Things Open as a Media Sponsor.
Sponsored OpenZFS Developers Summit at the Bronze level.
Applied for a virtual stand at FOSDEM 2021.
Committed to attend the online Apricot 2021.

Keep up to date with our latest work in our newsletters: https://www.freebsdfoundation.org/news-and-events/newsletter/

Netflix provided an update on how and why they use FreeBSD in our latest Contributor Case Study.

You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/.

Legal/FreeBSD IP

The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise.

Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you!

FreeBSD Release Engineering Team

Links
FreeBSD 12.2-RELEASE schedule	URL: https://www.freebsd.org/releases/12.2R/schedule.html
FreeBSD 13.0-RELEASE schedule	URL: https://www.freebsd.org/releases/13.0R/schedule.html
FreeBSD development snapshots	URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team <re@FreeBSD.org>

During the fourth quarter of 2020, the Release Engineering Team completed work on 12.2-RELEASE, the third release from the stable/12 branch, released on October 27. Thank you to all involved for the hard work that went into this release.

Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches. Development snapshot builds for 13.0-CURRENT have recently been built from the Git tree within the project, while further snapshot builds for 12.x and 11.x will continue to be built from Subversion. As we approach the end of 2020, continued preparations are being put in place for the upcoming 13.0 release, which will be the first release from Git.

Much of this work was sponsored by Rubicon Communications, LLC (netgate.com) and the FreeBSD Foundation.

Cluster Administration Team

Links
Cluster Administration Team members	URL: https://www.freebsd.org/administration.html#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

Finished setting up the Malaysia mirror site, generously hosted by the Malaysian Research & Education Network. Traffic from Oceania and parts of Asia is now going to this mirror instead of farther away sites like Japan and California.
Upgraded the package building machines to a version of head from mid-October 2020.
Upgraded developer machines in the cluster (freefall, ref* and universe*) to a version of head from mid-October 2020.
Installed eight new x86 servers in our New Jersey site: five application servers, two package builders and one mirror server.
- The new mirror server is in production (pkg0.nyi.freebsd.org).
- The two package builders are in production.
- Two of the application servers have been put into production as the Git source of truth and the cgit web frontend, respectively.
Installed two new aarch64 servers in our New Jersey site. Both are now building aarch64 packages.
Fixed package mirror synchronisation for powerpc64 packages.
Rebuilt the ZFS pool on the UK mirror server (pkg0.bme.freebsd.org) for better I/O parallelism. This should improve download performance especially at peak times.
Ongoing systems administration work:
- Accounts management for committers.
- Backups of critical infrastructure.
- Keeping up with security updates in 3rd party software.

Work in progress:

Hardware refreshing for web services, backup version control system in NYI
Upgrading production machines in the FreeBSD cluster to 12.2
- Most machines have been upgraded as of mid-December 2020
- Remaining machines will be decommissioned / repurposed after services migrate to newer hardware
Supporting Git migration and infrastructure setup
powerpc pkgbuilder/ref/universal machines
Preparations for a new mirror site in Australia, to be hosted by IX Australia.
Setup Brazil (BRA) mirror.
Review the service jails and service administrators operation.
Searching for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror.

Continuous Integration

Links
FreeBSD Jenkins Instance	URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab	URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive	URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report	URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki	URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki	URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI	URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@	URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository	URL: https://github.com/freebsd/freebsd-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure. The details of these efforts are available in the weekly CI reports.

During the fourth quarter of 2020, we continued working with the contributors and developers in the project to fulfil their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD.

Important changes:

doc jobs were changed to use git to follow VCS migration:
- https://ci.freebsd.org/job/FreeBSD-doc-main/
- https://ci.freebsd.org/job/FreeBSD-doc-main-igor/ Thanks Brandon Bergren (bdragon@)
head and stable/12 build environment have been upgraded to 12.2-RELEASE

New jobs added:

LINT kernel of head on riscv64

Work in progress:

Follow VCS migration, change src jobs to use Git - PRs are available Thanks Brandon Bergren (bdragon@)
Collecting and sorting CI tasks and ideas here
Testing and merging pull requests in the the FreeBSD-ci repo
Designing and implementing pre-commit CI building and testing
Reducing the procedures of CI/test environment setting up for contributors and developers
Setting up the CI stage environment and putting the experimental jobs on it
Setting up public network access for the VM guest running tests
Implementing automatic tests on bare metal hardware
Adding drm ports building tests against -CURRENT
Planning to run ztest and network stack tests
Adding more external toolchain related jobs
Improving the hardware lab to be more mature and adding more hardware
Helping more software get FreeBSD support in their CI pipeline Wiki pages: 3rdPartySoftwareCI, HostedCI
Working with hosted CI providers to have better FreeBSD support
The build and test results will be sent to the dev-ci mailing list soon. Feedback and help with analysis is very appreciated!

Please see freebsd-testing@ related tickets for more WIP information, and don't hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Ports Collection

Links
About FreeBSD Ports	URL: https://www.FreeBSD.org/ports/
Contributing to Ports	URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring	URL: http://portsmon.freebsd.org/index.html
Ports Management Team	URL: https://www.freebsd.org/portmgr/index.html
Ports Tarball	URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

For the last quarter the dashboard looks like:

41500 ports (including flavors)
2516 open PRs of which 625 are unassigned
8715 commits to the HEAD branch by 164 committers
420 commits to the 2020Q4 branch by 59 committers

Compared to the third quarter, the PR statistics mostly stayed the same. There

were slightly fewer commits by the same number of people. The number of ports again grew steadily, this time by almost 4 percent.

During the last quarter, we welcomed Juray Lutter (otis@) as a new ports committer and said goodbye to cpm, jadawin, knu, araujo, mmokhi and scottl.

Traditionally merges to the quarterly ports branches, which are more conservative versions of the HEAD tree, required approval of either the Ports Security Team (ports-secteam@) or portgmr@. There were already a number of blanket approvals for tested commits, ranging from fixing typing mistakes to upgrading web browsers to their latest version. As of last December, all ports committers are free to merge on their own, lessening the burden on ports-secteam@.

Patent limitations have been disconnected from the license framework, given that patents are a complex topic with implications varying from one jurisdiction to another.

The last quarter saw a number of updates to default versions of ports:

librsvg2: "rust" on supported platforms, "legacy" otherwise
Mono: 5.10
FPC switched to 3.2.0
GCC switched to 10 for powerpc64le
Lazarus switched to 2.0.10
Ruby switched to 2.7.X
Samba switched to 4.12

During the last quarter, a new virtual category was added: "education" for ports

that for instance help the user to learn about a certain topic or help facilitating examinations.

The @shell and @sample keywords have been rewritten in Lua which makes root-dir compliant (see pkg -r) and ensures they are Capsicum-sandboxed.

The last quarter also saw updates to several user-facing ports:

Firefox 84.0.1
Firefox-esr 78.6.0
Chromium 87.0.4280.88
Ruby 2.7.2
Qt5 5.15.2
XFce 4.16

As always, antoine@ was busy running exp-runs, 37 this quarter, testing:

various ports upgrades
changing sys/cdefs.h in base
adding "set pipefail" to most framework scripts to catch errors earlier
changing the default locale to C.UTF-8 in base
using bsdgrep as /usr/bin/grep

Office Hours

Contact: Allan Jude <allanjude@freebsd.org>
Contact: Ed Maste <emaste@freebsd.org>

During the final quarter of 2020 three office hours sessions were held.

The first was hosted by the core team in a time slot conducive to Asia and Australia, covering topics including the transition to git, recruiting for project teams, and core's todo list.

The second was hosted by the git transition team, and answered attendee questions about the transition to git and how it would impact the project's workflows.

The third session was hosted by bhyve maintainers Peter Grehan and John Baldwin to present recent development efforts and answer questions about bhyve.

The project is looking for volunteers to host future office hours sessions, as well as taking topic suggestions. We also hope to improve the system to allow people to submit questions ahead of time, so that we can take maximum advantage of subject matter experts when we have them for these calls.

You can find the schedule for future office hours, and videos of past office hours on the FreeBSD Wiki

Sponsor: ScaleEngine Inc.

Projects

Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.

GPL in Base

Links
GPL Software in the Base System	URL: https://wiki.freebsd.org/GPLinBase

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Kyle Evans <kevans@FreeBSD.org>
Contact: Baptiste Daroussin <bapt@FreeBSD.org>

A long-standing goal of the FreeBSD project is for the base system to migrate to modern, copyfree or more permissively licensed components. In this quarter, the following components have been successfully removed or replaced:

gdb (removed in favor of lldb in base or devel/gdb in ports)
gnugrep (replaced with bsdgrep)
libgnuregex (removed)

The following component(s) have yet to be claimed. Some replacement prospects

may be listed on the above-linked wiki page. Interested parties are welcome to evaluate the options to restart the discussion:

dialog
gcov (kernel)

The following component(s) have a principal investigator to coordinate work.

Note that partial completion likely means that a component is partially compatible, but could use evaluation and patches to bring parity with the component that it is replacing.

diff3 (Contact bapt@ if interested)

Git Migration Working Group

Links
src (base system) git repo	URL: https://cgit.FreeBSD.org/src
doc git repo	URL: https://cgit.FreeBSD.org/doc
Beta ports git repo	URL: https://cgit-dev.FreeBSD.org/ports
Warner's git documentation repo	URL: https://github.com/bsdimp/freebsd-git-docs
FreeBSD-git mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git
Git conversion tooling repo	URL: https://github.com/freebsd/git_conv
Game of Trees	URL: http://gameoftrees.org/
gitup	URL: https://github.com/johnmehr/gitup

Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Ulrich Spörlein <uqs@FreeBSD.org>

The Git working group largely completed the migration of the doc and src (base system) trees from Subversion to Git in December 2020. We are currently working on some minor outstanding issues and preparing for the ports tree migration.

We set up new hosts to serve as the Git repositories and mirrors, and developed commit hooks for restrictions on commits to various branches, generation of commit mail, and similar needs.

The doc tree migration occurred on December 8th and 9th. After the conversion some minor changes to the documentation build infrastructure were necessary.

The src tree migration occurred between December 20th and 23rd for the main branch; some additional tasks occurred over the next week or so. These included enabling the stable branches, vendor (contrib) code updates, and the git->svn gateway. We are translating stable branch commits to Subversion for the stable/11 and stable/12 branches and associated release branches. This allows FreeBSD users who follow stable branches or releases to continue using existing processes and tooling.

An experimental Git conversion of the ports tree is available at the link above. There are some unique challenges in the ports tree (that do not impact the doc or src repos in the same way), so additional work is ongoing. The window for migrating the ports tree is immediately prior to a quarterly branch, so we anticipate a migration at the end of March 2021. Over the next few months testing of the experimental ports repo is very welcome.

Process documentation for developer and user interaction with FreeBSD's repositories is currently available in Warner's GitHub repository at the link above. It will be moved to the FreeBSD developer's handbook and/or other suitable locations following the documentation project's asciidoc conversion.

The working group is experimenting with two permissively-licensed tools that are compatible with Git servers or repositories. Game of Trees is a version control system that is compatible with Git repositories. It is being developed by Stefan Sperling along with some OpenBSD developers and other contributions.

John Mehr's gitup is a minimal, dependency-free program that clones and synchronizes a local tree with a remote repository. It is intended for use cases that would otherwise be served by tools like portsnap.

Sponsor: The FreeBSD Foundation (in part)

Linux compatibility layer update

Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>

Linuxulator improvements have been ongoing for the last two years, with support from the FreeBSD foundation over a few distinct project grants as well as contributions from the community. The goal of this project is to improve FreeBSD's ability to execute unmodified Linux binaries. Current status is being tracked at Linux app status Wiki page. The work has now shifted from command-line apps to desktop applications.

There wasn't much Foundation-sponsored work done during this quarter, apart from extending fuse(4) to make it possible to run Linux FUSE servers, which is one of the things required to run AppImages. The Foundation-sponsored effort will continue into the first quarter of 2021 in order to make sure the 13.0-RELEASE ships with Linuxulator in a good shape.

There was a very significant contribution from Conrad Meyer in the form of SO_PASSCRED setsockopt(2) support, PR_SETDUMPABLE and PR_GETDUMPABLE prctl(2) flags, and also CLONE_FS and CLONE_FILES handling. This, along with some more cleanups and improvements, leads to working Linux Chromium; it has been tested with Netflix and Spotify clients. It still requires three flags (--no-sandbox --no-zygote --in-process-gpu) to be passed on the command line to work around missing functionality, though. Also, the name_to_handle_at(2) and open_by_handle_at(2) syscalls are now supported. There are also much better debug messages for unrecognized socket options.

Sponsor: The FreeBSD Foundation

LLDB Debugger Improvements

Links
Moritz Systems Project Description	URL: https://www.moritz.systems/blog/lldb-debugger-improvements-for-freebsd/
FreeBSD Foundation Blog	URL: https://freebsdfoundation.org/blog/guest-blog-foundation-sponsors-freebsd-lldb-improvements/
Git Repository	URL: https://github.com/moritz-systems/llvm-project

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

The LLDB project builds on libraries provided by LLVM and Clang to provide a great modern debugger. It uses the Clang ASTs and the expression parser, LLVM JIT, LLVM disassembler, etc so that it provides an experience that “just works”. It is also blazing fast and more permissively licensed than GDB, the GNU Debugger.

LLDB is the default debugger in Xcode on macOS and supports debugging C, Objective-C, and C++ on the desktop and iOS devices and the simulator.

FreeBSD includes LLDB in the base system. At present, it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. It used to rely on an obsolete plugin model in LLDB that was a growing technical debt. This project aimed to bring LLDB closer to a fully featured replacement for GDB, and therefore for FreeBSD to feature a modern debugger for software developers.

The legacy monolithic target support executed the application being debugged in the same process space as the debugger. The modern LLDB plugin approach, used on other supported targets, executes the target process under a separate lldb-server process. This improves reliability and simplifies the process / thread model in LLDB itself. In addition, remote and local debugging is now performed using the same approach.

After the migration to the new process model on 32 and 64-bit x86 CPUs, the project focused on reviewing the results of LLDB’s test suite and fixing tests as time permits.

During the Moritz Systems work, the FreeBSD Project gained numerous important improvements: in the kernel, userland base libraries (the dynamic loader) and the LLVM toolchain FreeBSD support.

The introduced changes are expected to be shipped with LLDB 12.0, and where applicable in FreeBSD 13.0.

The overall experience of FreeBSD/LLDB developers and advanced users on this rock solid Operating System reached the state known from other environments. Furthermore, the FreeBSD-focused work also resulted in generic improvements, enhancing the LLDB support for Linux and NetBSD.

Sponsor: The FreeBSD Foundation

Upstreaming NetApp Changes

Links
Klara Inc.	URL: https://klarasystems.com/freebsd-development/

Contact: Alexander Sideropoulos <Alexander.Sideropoulos@netapp.com>
Contact: Allan Jude <allan@klarasystems.com>

NetApp has started an effort to upstream bug fixes and other improvements from the ONTAP code line into FreeBSD. These changes benefit the FreeBSD community by providing many fixes that NetApp has made over the past few years, while allowing NetApp to reduce the number of customizations needed when bringing in the latest FreeBSD changes back into the ONTAP tree.

NetApp has partnered with Klara to facilitate this project, to help identify interesting and useful changes to send upstream, to rework and generalize those changes as required to make them suitable for upstreaming, and to shepherd them through the FreeBSD code review process.

During the fourth quarter, Klara has made 40 upstream fixes in the FreeBSD kernel in various subsystems including geom, dev, amd64, net, kern, netinet, and several other areas of the tree on behalf of NetApp.

NetApp intends to continue to sponsor this effort throughout 2021.

Sponsor: NetApp

NFS over TLS implementation

Contact: Rick Macklem <rmacklem@freebsd.org>

In an effort to improve NFS security, an Internet Draft titled "Towards Remote Procedure Call Encryption By Default" specifies use of TLS 1.3 to encrypt all data traffic on a Sun RPC connection used for NFS.

The coding for this project has now been completed. All required changes to the NFS and kernel RPC code have been committed to the head/current kernel and will be in FreeBSD13. The daemons can now be built from a port that depends upon the security/openssl-devel port of Openssl3 that includes patches for support of ktls. The port for the daemons is called sysutils/nfs-over-tls and should be committed to the ports framework soon. In the meantime, the port can easily be fetched, as described in https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt.

To support clients such as laptops, the daemons that perform the TLS handshake may optionally handle client X.509 certificates from a site local CA. There are now exports(5) options to require client(s) to provide a valid X.509 certificate. The case where a "user" name is stored in the certificate and is used to map all RPC credentials to that user is probably in violation of the Internet Draft. This is only enabled when the "-u" command line option is provided to rpc.tlsservd(8).

The code is now available for testing. See: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt

Setting up system(s) for testing still requires building a custom kernel with "options KERN_TLS" from recent head/FreeBSD13 sources plus installing the port for the daemons, as explained by the above document.

The main limitation in the current implementation is that it uses TLS1.2 and not TLS1.3, as required by the Internet Draft. This should change once the KERN_TLS rx patch includes TLS1.3 support.

Third party testing would be appreciated.

OpenBSM Synchronisation

Links
TrustedBSD / OpenBSM	URL: http://www.trustedbsd.org/openbsm.html
OpenBSM Github Sources	URL: https://github.com/openbsm/openbsm
Synchronisation with macOS Catalina	URL: https://github.com/openbsm/openbsm/commit/54a0c07cf8bac71554130e8f6760ca68e5f36c7f
Apple OpenSource	URL: https://opensource.apple.com

Contact: Gordon Bergling <gbe@FreeBSD.org>

OpenBSM is a crucial part of FreeBSD, which provides auditing features for the operating system. OpenBSM is incorporated into FreeBSD and macOS. Both Apple and FreeBSD have currently made changes to the OpenBSM framework, which weren't upstreamed. This small project aims to consolidate these changes and upstream them to the OpenBSM github repository, so that both development efforts can be merged to FreeBSD later on. The tricky part of this project is the manual comparison, since Apple doesn't provide any changelogs.

I am currently working on the macOS Catalina sources and hopefully Apple will release the sources of macOS Big Sur in time for FreeBSD 13.

Tool Chain

Links
ELF Tool Chain homepage	URL: https://sourceforge.net/p/elftoolchain

Contact: Dimitry Andric <dim@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>

In October Clang/LLVM was updated to 11.0.0, followed by a number of bug fixes from upstream, including improvements for a number of Tier-2 architectures. We also enabled the -fstack-clash-protection flag to enable compiler mitigation for the "stack clash" vulnerability and are coordinating with upstream.

Upstream LLDB support for FreeBSD improved substantially over the last quarter, as detailed elsewhere in this report. These improvements will make it into the FreeBSD base system early in 2021 when LLVM is next updated to 12.0. As also mentioned elsewhere, we removed the obsolete copy of GDB 6.1.1.

The ELF Tool Chain received a number of bug fixes, as well as support for readelf -z (handling compressed ELF debug sections) and an improvement to addr2line to report based on labels when other debug information is not available. We are working to upstream these changes to the ELF Tool Chain project.

There are a number of open issues and opportunities for improvements in various ELF Tool Chain components. Contributions in these areas are very welcome,

Sponsor: The FreeBSD Foundation (in part)

Kernel

Updates to kernel subsystems/features, driver support, filesystems, and more.

ENA FreeBSD Driver Update

Links
ENA README	URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

Completed since the last update:

MFC of the ENA v2.3.0 driver to the FreeBSD 11-STABLE branch
MFC of the ENA v2.3.0 driver to the upcoming FreeBSD 12-STABLE branch
Add feature that allows reading extra ENI (Elastic Network Interface) metrics about exceeding BW/pps limits
Add SPDX license tag to the ENA driver files
Add Rx offsets (hardware feature) support for the ENA driver
Fix completion descriptors alignment for the ENA device - on some of the platforms ENA needs alignment to 4k

Work in progress:

Introduce full kernel RSS API support.
Allow reconfiguration of the RSS indirection table and hash key
Prototype the driver port to the iflib framework

Sponsor: Amazon.com Inc

Intel wireless update

Links
The freebsd-wireless mailing list	URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless

Contact: Bjoern A. Zeeb <bz@FreeBSD.org>

The Intel Wireless driver update project aims to bring support for newer chipsets and also get station side to 11ac in a first step.

During the last months connection code between net80211 and the Linux driver KPI was implemented and scanning is working. Currently the focus is on sending and driving one state machine from the other and syncing state between net80211 and the Linux compat code.

In addition the driver and firmware was updated from upstream sources to include support for the AX210 hardware generation, which was already tested to attach.

The hope is that by the time the status report gets published authentication and association are working and basic data packet passing will work soon.

Sponsor: The FreeBSD Foundation

Fenestras X random(4)

Links
SVN revision 1/3	URL: https://svnweb.freebsd.org/base?view=revision&revision=366620
SVN revision 2/3	URL: https://svnweb.freebsd.org/base?view=revision&revision=366621
SVN revision 3/3	URL: https://svnweb.freebsd.org/base?view=revision&revision=366622
FX Design (PDF)	URL: https://aka.ms/win10rng
Fortuna Design	URL: https://www.schneier.com/academic/fortuna/

Contact: Conrad Meyer <cem@FreeBSD.org>
Contact: FreeBSD CSPRNG group <csprng@FreeBSD.org>

Since FreeBSD 11, the default random(4) implementation is based on the Fortuna (2003) design by Ferguson and Schneier. At a high level, Fortuna accumulates entropy into a series of pools, and reseeds a single generator from some of these pools according to some criteria.

In 2019, Ferguson (at Microsoft) published a whitepaper on the design of the Windows 10 system random number generator. Fenestras X is a random(4) implementation based on the published Windows 10 design.

The Fenestras X / Windows 10 design is similar to Fortuna, so it is probably most interesting to describe their differences:

Fenestras X has per-CPU generators seeded from a root generator. Fortuna only has the root generator. This change eliminates lock contention between random(4) readers running on multiple cores.
Generators in Fenestras X form a tree from the root RNG. When read, generators efficiently check if their parent generator has been seeded with newer entropy. If so, child generators reseed themselves before serving the read operation. This is integrated with arc4random(9), as well as userspace arc4random(3).
Fenestras X generators are buffered. Requests smaller than some arbitrary threshold (currently 128 bytes) are served from the buffer. Bytes read from the buffer are securely erased when they are consumed. The buffer is refreshed if the request consumes more bytes than were available in the buffer. This amortizes the cost of rekeying and generating output from a cryptographic CTR-mode cipher, which is especially slow with AES.

There are other important differences, and readers interested in system CSPRNGs

should read Ferguson's whitepaper. It is short and accessible. For more information on the FreeBSD implementation, please see the SVN commit messages — especially r366620.

The Fenestras X implementation is available in CURRENT, but disabled by default. (The default remains Fortuna.) At this time, you must set the RANDOM_FENESTRASX option in your custom kernel configuration and rebuild your kernel to use the new design. There are no known bugs or weaknesses relative to the Fortuna implementation.

Future work and call to action:

Additional design review, implementation review, and testing is welcome.
Additional entropy sources: we could use implementations of some of the sources described in the whitepaper, in both Fortuna and Fenestras X. In particular, we're missing a jitter entropy source.

pf performance improvement

Links
First commit	URL: https://cgit.freebsd.org/src/commit/?id=1c00efe98ed7d103b9684ff692ffd5e3b64d0237
D27707	URL: https://reviews.freebsd.org/D27707
D27756	URL: https://reviews.freebsd.org/D27756
D27757	URL: https://reviews.freebsd.org/D27757
D27758	URL: https://reviews.freebsd.org/D27758
D27759	URL: https://reviews.freebsd.org/D27759
D27760	URL: https://reviews.freebsd.org/D27760
D27761	URL: https://reviews.freebsd.org/D27761
D27762	URL: https://reviews.freebsd.org/D27762
D27763	URL: https://reviews.freebsd.org/D27763
D27764	URL: https://reviews.freebsd.org/D27764

Contact: Kristof Provost <kp@freebsd.org>

The performance of pf was not as good as it could be. Some investigation with the invaluable hwpmc tooling eventually pointed to very poor cache behaviour. The longest_lat_cache.miss event was very informative.

This turned out to be due to pf doing packet and byte counting in states, rules and interfaces.

The pf code took the very straightforward approach of having a simple uint64_t variable and incrementing it for every packet. The downside of this is that when multiple cores do it simultaneously the CPU ends up having to write this to memory very often, slowing packet processing down greatly. Happily the counter(9) framework is designed for this exact situation.

One additional complication is that pf uses the same structure definitions for its internal data as it uses for configuration from user space. To avoid breaking user space these data structures have been decoupled. That is, where pf_rule used to be used both to set rules via the ioctl() interface and to evaluate rules while processing packets we now only use pf_rule for configuration. The new pf_krule structure is used when evaluating packets. This allows us to change the pf_krule structure, to change uint64_t to counter_u64_t, without affecting user space.

Olivier Cochard-Labbé tested the full set of changes, and found (depending on hardware) substantial improvements in throughput.

Sponsor: Orange Business Services

IP Routing lookup improvements

Links
Add modular routing lookup framework.	URL: https://reviews.freebsd.org/D27401

Contact: Alexander Chernikov <melifaro@FreeBSD.org>

This work adds a fib lookup framework, allowing to attach custom IP lookup algorithms to any routing table on the fly. It allows to use more performant and efficient lookup algorithms, dynamically selected based on the number of routes in the routing table. Finally, it provides an implementation of modified DIR-24-8 for IPv4/IPv6, speeding IP lookups for the large-fib use case.

This work is a part of a larger effort to modernise the routing subsystem.

Background

FreeBSD runs diverse workloads on both low-end and high-end devices, resulting in different networking/memory requirements for each case. Small boxes with a couple of routes are different from routers with full-view. IPv4 lookups are different from IPv6 ones. Conditions can change dynamically: one may easily reconfigure a system to receive full view instead of a default route.

Currently, FreeBSD uses radix (compressed binary tree) to perform all unicast route manipulations, including routing lookups. Radix implementation requires storing key length in each item, allowing to use sockaddrs, transparently supporting virtually any address family. This flexibility comes at a cost: radix is relatively slow, cache-unfriendly and adds locking to the hot path. Finally, radix is closely coupled to the rest of the system, making it hard to switch to something else.

Implementation overview

Overview

Modular fib IP lookup framework has been designed to address flexibility and performance requirements.

It keeps system radix as the "control plane" source of truth, simplifying actual algorithms implementation. It allows dynamic load new algorithms as the kernel modules and abstracts most OS-specific details, reducing algorithm "glue" code. It automatically adapts to the current system state by picking the best matching algorithm for the routing table on-the-fly.

The following algorithms are provided by default.

IPv4:

bsearch4 (lockless binary search in a specially-crafted IP array), tailored for small-fib (less than 16 routes)
radix4_lockless (lockless immutable radix, re-created on every routing table change), tailored for small-fib (less than 1000 routes)
radix4 (base system radix backend)
dpdk_lpm4 (DPDK DIR24-8-based lookups), lockless datastructure optimised for large-fib ( D27412 )

IPv6:

radix6_lockless: lockless immutable radix, re-created on every routing table change, tailored for small-fib (less than 1000 routes)
radix6: wrapper around existing system radix
dpdk_lpm6: DPDK DIR24-8-based lookups, lockless datastructure optimised for large-fib ( D27412 )

Performance changes

Micro benchmarks (i7-7660U, single-core lookups, 2048 destinations, benchmark code in D27604).

IPv4:

8 routes: radix4: ~20mpps, radix4_lockless: ~25mpps, bsearch4: ~69mpps, dpdk_lpm4: ~67 mpps
700k routes: radix4_lockless: 3.3mpps, dpdk_lpm4: 46mpps

IPv6:

8 routes: radix6_lockless: ~20mpps, dpdk_lpm6: ~70mpps
100k routes: radix6_lockless: ~14mpps, dpdk_lpm6: ~57mpps

Forwarding performance:

+10-15% IPv4: small-fib, bsearch4
+25% IPv4: full-view, dpdk_lpm4
+20% IPv6: full-view, dpdk_lpm6

Status

Modular longest-prefix-match lookup algorithms (D27401) [ DONE ]
- Design control plane framework for attaching algorithms [ DONE ]
- Port DPDK IPv6 lockless lookup algorithm ( D27412) [ DONE ]
- Port DPDK IPv4 lockless lookup algorithm ( D27412) [ DONE ]

Scalable routing multipath support

Links
Implementation of scalable multipath	URL: https://reviews.freebsd.org/D24141#change-ZOjdMqgDgUr7
Introduce scalable route multipath	URL: https://reviews.freebsd.org/D26449

Contact: Alexander Chernikov <melifaro@FreeBSD.org>

This work targets implementing scalable routing multipath support and enabling it by default. It closes the long-standing feature gap with other modern networking OSes.

This work is a part of on-going efforts to modernize the routing subsystem.

Background

Initial FreeBSD multipath implementation, RADIX_MPATH, was added back in 2008. It was based on the radix changes and represented multipath routes as a linked-list of chained paths. It was not fully finished and tested, resulting in many crash reports.

Implementation overview

Multipath-related change changes are based on the introduction of the concept of next hops. Nexthops are separate data structures, containing the necessary information to perform packet forwarding. They are shared among the routes, providing more pre-computed cache-efficient data while requiring less memory. Interested readers can find a more detailed description in D24141. They can find another overview in Nexthop objects talk describing Linux kernel implementation.

Multipath implementation extends the nexthop concept further by introducing nexthop groups. Nexthop group is simply an array of nexthops, compiled according to each nexthop relative weight.

Status

Nexthop objects (D24232) [ DONE ]
- Introduction of nexthop objects [ DONE ]
- Conversion of old KPI users to the new one [ DONE ]
  - Conversion of route caching to nexthop caching [ DONE ]
- Conversion of struct rtentry field access to nhop field access [ DONE ]
- Eliminating old lookup KPI and hiding struct rtentry [ DONE ]
Multipath routing (D26449) [ DONE ]
- Switch control plane customers to use (rtentry, nexthop) pairs instead of rtentry to allow multipath changes happen transparently [ DONE ]
- Introduce nexthop group objects [ DONE ]
- Add multipath support for the rib (routing information base) manipulation functions [ DONE ]
- Add flowid generation for outbound traffic to enable load balancing [ DONE ]
Routing daemon support
- Add net/bird support for multipath routing [ NOT STARTED ]
- Add explicit nexthop/nexthop groups control via rtsock [ IN PROGRESS ]
- Work with FRR developers to add nexthop-based route control [ NOT STARTED ]

Thunderbolt3/USB4 stack

Contact: Scott Long <scottl@freebsd.org>

This project implements a driver stack for Thunderbolt3 and USB4. These technologies differ radically from USB3 and prior, and require completely new drivers for the host interface adapter and topology as well as configuration management layers. At their most fundamental level, a TBT3/USB4 topology appears as PCI bridges and buses, and attached devices appear as either PCI devices, USB3 devices, or DisplayPort devices. Early TBT3 controllers don't even appear in the system topology unless a TBT3 device is plugged in. These early TBT3 systems also implement a security policy meant to protect against unauthorised or malicious devices, though that scheme has been proven to not be effective and has been removed from later TBT3 and USB4 implementations. Besides security control, the TBT3/USB4 stack controls power management and topology hotplug.

The FreeBSD driver currently supports Alpine Ridge and Ice Lake TBT3 controllers, and can perform basic security validation and topology awareness. USB4 support as well as full connection manager and power management support is still being worked on. The current driver will be committed to FreeBSD in early January 2021.

Though this work is not sponsored, it has been done with the encouragement and support of the FreeBSD Foundation and Netgate.

Vectored AIO

Contact: Alan Somers <asomers@FreeBSD.org>

POSIX AIO is a facility for asynchronous I/O to files and devices. FreeBSD's implementation is efficient, especially when writing to disk files. But a long-standing defect in the standard API is a lack of vectored functions. That is, there is no asynchronous equivalent of pwritev(2) and preadv(2). A common workaround is to use lio_listio(2) instead. However, that has several drawbacks. It's more effort for the programmer, it might return early with only a subset of the operations completed, it requires more total syscalls, and there is no guarantee that the operations will complete in-order.

This quarter I added two new syscalls: aio_writev(2) and aio_readv(2). They work just like their non-vectored counterparts, but they take an array of iovec elements, just like pwritev and preadv. You can't use them in combination with lio_listio, but that could be added in the future.

ZSTD Compression in ZFS

Contact: Allan Jude <allanjude@freebsd.org>

This effort to complete the integration of ZSTD into ZFS is funded by the FreeBSD Foundation.

During the four quarter the final tasks in the project to integrate ZSTD into OpenZFS were completed.

Completed milestones in this project:

Integrated ZSTD in the FreeBSD boot loader (Warner Losh imp@freebsd.org)
Added a section to the FreeBSD Handbook ZFS chapter explaining ZSTD
Wrote a FreeBSD Journal Article explaining considerations when selecting a suitable compression level
Monitored for bug reports after the changes were integrated into -CURRENT

With all of these changes in place, it is now possible to boot from pools

compressed with zstd or zstd-fast. For comparison, a standard installation of FreeBSD 13 (without debug symbols) uncompressed is 1175 MB, and when compressed with LZ4, is only 570 MB (2.15x) but when compressed with ZSTD's default level of 3 is only 417 MB (3.00x), and with the maximum level, 19, only 374 MB (3.36x).

Sponsor: The FreeBSD Foundation

Architectures

Updating platform-specific features and bringing in support for new hardware platforms.

arm64 platform updates

Contact: Mitchell Horne <mhorne@FreeBSD.org>

In the interest of seeing the arm64 architecture promoted to Tier-1 status, an effort was undertaken to test building and serving of release and patch-level updates via freebsd-update(1). The conclusion of this investigation is that the process works with very few changes required; a small tweak is needed for the update build scripts, and a minor bugfix in the bsdiff(1) utility was committed. The hope is that the project can begin providing security updates for the platform with the release of FreeBSD 13.0, removing the requirement that users compile these updates from source.

Added this quarter was arm64 support for the new ossl(4) crypto driver. This driver provides acceleration of SHA-1 and SHA-2 cryptographic operations by leveraging OpenSSL's assembly routines. These routines will detect and use optimized instructions, as supported by the CPU. This support benefits userland applications via the cryptodev(4) device, and in-kernel consumers of the crypto(9) interface, such as the IPSec Authentication Header protocol and kernel TLS.

Finally, work was done to add the necessary machine-dependent bits for the kernel's gdb(4) interface. This enables remote debugging of the kernel with gdb(1) over a serial line.

Sponsor: The FreeBSD Foundation

FreeBSD/RISC-V Project

Links
Wiki	URL: https://wiki.freebsd.org/riscv

Contact: Mitchell Horne <mhorne@FreeBSD.org>

Contact: freebsd-riscv Mailing List
Contact: IRC #freebsd-riscv on freenode

The FreeBSD/RISC-V project is providing support for running FreeBSD on the RISC-V Instruction Set Architecture.

This quarter saw a number of improvements and bugfixes from committers and contributors alike. A few small items from this quarter:

Added riscv64 LINT kernel config plus CI job (FreeBSD-head-riscv64-LINT)
Switched emulators/riscv-isa-sim to official upstream and updated to 2020-11-02 snapshot
Created sysutils/u-boot-sifive-fu540, a u-boot port for the HiFive Unleashed
Improved SBI extension support

Further progress was made this quarter in building ports for RISC-V. Build and

runtime issues with large dependencies devel/python-setuptools and devel/glib20 were fixed, enabling several thousand skipped ports. There is some in-progress work to address failures in other significant ports, such as devel/nspr and databases/sqlite3. By addressing some of these small-effort issues, some 15000+ ports can now be built for the platform with qemu-user-static.

Finally, December saw the arrival of the first riscv64 weekly development snapshots. This includes the usual memstick installer, a virtual machine image, and a generic SD card image. There are still some minor tweaks to be made, but this marks a significant step forward for the platform, and lowers the barrier of entry for running a FreeBSD/RISC-V system. This also means that FreeBSD 13 will likely be the first downloadable release for the architecture. For those interested in trying out the VM image for themselves, see the Quick Start instructions on the wiki.

Userland Programs

Changes affecting the base system and programs in it.

Dual-stack ping command

Contact: Alan Somers <asomers@FreeBSD.org>

The venerable ping command has until now only supported IPv4. A separate utility, ping6, was originally written by WIDE as a research tool to develop IPv6. As a research tool, it didn't need IPv4 support, but since then, it's been put to practical use by countless developers and sysadmins everywhere.

The ping/ping6 split has been a perennial complaint. It's annoying that two separate commands are needed, even though they do almost exactly the same thing. This quarter, I merged Ján Sučan's GSoC work, which merged the two commands. Now ping can handle either protocol, based on the -4 and -6 switches, or based on the format of the target. A ping6 hard link is provided for backwards compatibility.

Sponsor: Google Summer of Code

Ports

Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves.

KDE on FreeBSD

Links
KDE FreeBSD	URL: https://freebsd.kde.org/
KDE Community FreeBSD	URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, graphics applications, instant-messengers, a video-editing suite, as well as a tea timer and hundreds of other applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine.

This quarter the kde@ team:

Landed the October, November and December updates to KDE Applications and to KDE Plasma
Landed all of the bi-weekly KDE Frameworks releases
Updated Qt to 5.12.2, including Qt5 WebEngine
Followed up with two cmake patch releases
Followed up one ninja patch release

There was lots of infrastructural work and individual application

updates and a new Matrix client from the KDE community as well, which we typically fail to administer and write about so this report is fairly short with mostly big-ticket items. We had fun, we chased the things that are most useful to us, and got through the year. Here's to next year with actually seeing FreeBSD people again.

I (adridg@) would like to especially thank Kai Knoblich (kai@) for chasing WebEngine: that's a huge and painful codebase to deal with, and here we are, all up-to-date.

FreeBSD Office team

Links
The FreeBSD Office project	URL: https://wiki.freebsd.org/Office

Contact: FreeBSD Office team ML <office@FreeBSD.org>
Contact: Dima Panov <fluffy@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The FreeBSD Office team works on a number of office-related software suites and tools such as OpenOffice and LibreOffice.

Work during this quarter focused on providing the latest stable release of LibreOffice suite and companion apps to all FreeBSD users.

Latest and quarterly ports branches got a series of updates of the LibreOffice suite from 7.0.1 thru 7.0.4 releases, compilation patches for all Tier 1 architectures, and updates of all companion libraries. Some of our local and critical to build patches were sent to and accepted by upstream.

Meanwhile, our WIP repository was moved to new home under official github.org/freebsd resources.

The WIP repository also has a major update with development versions of the LibreOffice suite, version 7.1.0.0.beta1 for now. Release will be planned in March, 2021.

We are looking for people to help the project. All unstable work with LibreOffice snapshots is staged in our WIP repository.
The open bugs list contains all filed issues which need some attention. Patches, comments and objections are always welcome in the mailing list and bugzilla.

Ports On Non-x86 Architectures

Contact: Mark Linimon <linimon@FreeBSD.org>

It has been some time since the last report on the status of FreeBSD ports on non-x86 architectures.

Traditionally, we have referred to these as "tier-2 architectures". However, aarch64 and powerpc64 have aspirations for tier-1. Also, although riscv64 is currently tier-3, it has aspirations for tier-2.

The big news is that, thanks to the FreeBSD Foundation (and the assistance of Philip Paeps), FreeBSD now has two new aarch64 boxes, which have replaced the previous, badly-aging, alternatives. For the first time since August, we once again have up-to-date aarch64 packages.
Thanks to the above, and the work of Emmanuel Vadot and others, some bitrot in aarch64 ports has been reversed.
Piotr Kubaj (pkubaj@) continues QA on powerpc64 (big-endian) ports. Almost everything that is buildable now does so. The Linux ports and some of the graphics drivers are excluded. Otherwise, powerpc64 is up to parity with amd64.
Piotr has also begun the task of bringing powerpc64le (little-endian) up to parity with powerpc64. Although several of the powerpc64 src committers (and your author) have a fondness for big-endian, the fact is that our most feasible path to getting graphics capability anywhere near parity with x86 is via the little-endian choice.
Mark Linimon (linimon@) has begun his own test-builds of ports on riscv64 just to ascertain overall buildability. Surprisingly, many ports do indeed build. Thanks to contributions from several people already working on riscv64, including John Baldwin (jhb@) with an LLVM fix, we are now able to build around 20,000 packages. NB: these packages are unofficial and not guaranteed.
The work of Kyle Evans (kevans@) on chasing bitrot in qemu has been key to work on both aarch64 and riscv64. All users are encouraged to update to the latest version.
Unfortunately mips/mips64 are badly in need of work. The fact that devel/libffi does not build on mips64 blocks nearly half the ports tree.

Tasklist:

We need users of riscv64 to actually test the packages that have been built (so far, they have only been tested for buildability). Contact linimon@ if you are interested.
If anyone is still using mips/mips64 for other than the most trivial tasks, we would welcome patches.

Python 2.7 removal from Ports

Links
About FreeBSD Ports	URL: https://www.FreeBSD.org/ports/
Ports Management Team	URL: https://www.freebsd.org/portmgr/index.html
[meta] Ports broken by Python 2.7 End-of-Life and removal	URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249337

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

As of January 2020, Python 2.7 reached its end-of-life after several years of extensions. Portmgr subsequently started the project of phasing Python 2.7 out of the Ports Tree by tagging lang/python27 for expiration on 2020-12-31. Last year, some 740 ports were removed from the Ports Tree as they were incompatible with Python 3, mostly because these ports were either unmaintained or abandoned upstream.

During this process, there were several instances of an upstream still being active but where the upstream have not had the resources yet to upgrade their software to Python 3. A noticeable example of this is www/chromium and derived software, such as devel/electron7 and www/qt5-webengine. Portmgr is currently looking into ideas on how to minimize the impact of Python 2.7 on the Ports Tree while keeping Chromium and KDE 5 functional. As various software packages on the FreeBSD cluster itself also use Python 2.7, portmgr started coordinating with affected parties on upgrade plans. Currently there are 40 ports left that directly depend on Python 2.7 to build or run, and an unknown number of indirect ports. All those ports should eventually be upgraded to Python 3 or be removed too, ideally some time this year.

Portmgr is currently cleaning up (unused) Python 2.7 code from ports which do not need Python 2.7. New ports should not be using Python 2.7 anymore, i.e. they should not have USES=python but instead something like USES=python:3.6+.

So while this all looks rather invasive, it is not feasible to keep Python 2.7 around for much longer. Over time security vulnerabilities might show up which will likely no longer be fixed, because the Python Software Foundation no longer supports Python 2.7. Other problems are that the software gets outdated over time and thereby loses its usefulness as part of a development kit.

Help needed:

Coordinate with postmaster on isolating or migrating away from mail/mailman
Coordinate with clusteradm (?) for upgrading svnweb and our wiki

Xfce on FreeBSD

Links
Xfce 4.16 Upstream Release Announcement	URL: https://xfce.org/about/news/?post=1608595200
Xfce meta-port on FreshPorts	URL: https://www.freshports.org/x11-wm/xfce4

Contact: Xfce team <xfce@FreeBSD.org>
Contact: Guido Falsi <madpilot@FreeBSD.org>

The FreeBSD Xfce team (xfce@) work to ensure the Xfce desktop environment is maintained and fully functional on FreeBSD.

This quarter the Xfce team are pleased to welcome Xfce 4.16 to the FreeBSD ports tree!

Some of the highlights of this Xfce 4.16 release:

The panel now supports dark mode (enabled by default) and an animated autohide transition
A new panel plugin dubbed "statustray" which combines both StatusNotifier and legacy Systray items
Fractional scaling support was added to the display dialog (helpful on HiDPI displays)
A new tab in the "About Xfce" dialog shows basic system information like CPU or GPU type
The settings manager has improved search and filter capabilities
All settings dialogs now use window decorations drawn by Gtk (client side decorations)
The "Mime Settings" and "Preferred Applications" dialogs were merged into the "Default Applications" dialog
The Thunar file manager now supports pause for copy/move operations, and queued file transfer
Generating thumbnails for .epub (e-book format) was added to tumbler
A new default wallpaper and icon theme
The application finder now allows for sorting applications by "frecency" - a combination of frequency and recency
Dropped GTK2 support from all components and plugins

For further details, refer to the Xfce 4.16 upstream release announcement.

Due to GTK2 and libxfce4gui support being removed, some panel plugins and libraries will be removed since they no longer work with Xfce 4.16:

deskutils/orage
deskutils/xfce4-volumed
print/xfce4-print
science/xfce4-equake-plugin
x11/xfce4-embed-plugin
x11/xfce4-quicklauncher-plugin
x11/xfce4-wmdock-plugin
x11-toolkits/libxfce4gui

WARNING: Unfortunately this update can reveal a bug in pkg which can cause files from the libexo package to be absent after upgrade. To avoid the issue, upgrade the libexo package by itself before the rest of the packages, as described in UPDATING entry 20210102.

Thanks also to riggs@, Olivier Duchateau duchateau.olivier@gmail.com, woodsb02@, Sergey Dyatko sergey.dyatko@gmail.com, and ehaupt@ for their help and contributions.

Documentation

Noteworthy changes in the documentation tree, in manpages, or in external books/documents.

FreeBSD Translations on Weblate

Links
Translate FreeBSD on Weblate wiki	URL: https://wiki.freebsd.org/DocTranslationOnWeblate
FreeBSD Weblate Instance	URL: https://translate-dev.freebsd.org/

Contact: Danilo G. Baio <dbaio@FreeBSD.org>
Contact: Edson Brandi <ebrandi@FreeBSD.org>

In search of new contributors an article was published in the September/October 2020 issue of the FreeBSD Journal about How to Become a FreeBSD Translator.

During the whole year we received new contributors to the effort; numbers are still growing and we are receiving translations almost daily on our Weblate platform.

Q4 2020 Status

11 languages (1 new language)
116 registered users (69 new users since 2020q1)

Languages

Chinese (Simplified) (zh_CN)
Chinese (Traditional) (zh_TW)
Dutch (nl_NL) - Added
French (fr_FR)
German (de_DE)
Italian (it_IT)
Norwegian (nb_NO)
Persian (fa_IR)
Portuguese (pt_BR)
Spanish (es_ES)
Turkish (tr-TR)

We want to thank everyone that contributed, translating or reviewing documents.

And please, help promote this effort on your local user group, we always need more volunteers.

DOCNG on FreeBSD

Links
DOCNG Website Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-website
DOCNG Documentation Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-documentation
DOCNG Share Repo	URL: https://gitlab.com/carlavilla/freebsd-hugo-data

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

The Doc New Generation project is finished. The switch-over date will be Saturday, January 23rd.

The objective of using Hugo and AsciiDoctor is to reduce the learning curve and let people to start quickly contributing to our documentation system. Other benefits of using Hugo is that we can use other technologies aside from AsciiDoctor, like MarkDown, RST, Pandoc, etc.

You can find a work in progress on updating the FreeBSD Documentation Project Primer to Hugo/AsciiDoctor.

Miscellaneous

Objects that defy categorization.

Prometheus NFS Exporter

Links
FreeBSD NFS statistics exporter for Prometheus	URL: https://github.com/Axcient/freebsd-nfs-exporter

Contact: Alan Somers <asomers@FreeBSD.org>

FreeBSD's nfsstat(8) utility provides a wealth of statistics, but I wanted to monitor them with Prometheus. Screen-scraping the --libxo output would've been possible, but some of the stats are preprocessed in a way that interferes with my Prometheus processing. So I wrote a separate utility that publishes the raw stats provided by the kernel. Along the way I found and fixed a few bugs in nfsstat, too. If anybody is interested, I can add a port for it.

Sponsor: Axcient

Third-Party Projects

FreeBSD Aarch64 under VMWare ESXi-ARM Fling

Links
ESXi-ARM Fling	URL: https://flings.vmware.com/esxi-arm-edition
FreeBSD Under VMWare ESXi-ARM Fling	URL: https://vincerants.com/freebsd-under-vmware-esxi-on-arm-fling/
FreeBSD on ESXi-ARM Fling: Fixing Virtual Hardware	URL: https://vincerants.com/freebsd-on-esxi-arm-fling-fixing-virtual-hardware/
open-vm-tools for FreeBSD VMWare ESXi-ARM Fling	URL: https://vincerants.com/open-vm-tools-on-freebsd-under-vmware-esxi-arm-fling/

Contact: Vincent Milum Jr <freebsd@darkain.com>

VMWare is a company that produces a commercial hypervisor known as vSphere ESXi for AMD64 and i386. In early October, they released a tech demo hypervisor for ARM Aarch64 which runs on ARM ServerReady hardware as well as single board computers such as the Raspberry Pi 4b (4GB and 8GB models). This new hypervisor is known as VMWare ESXi-ARM Fling.

Since the release of ESXi-ARM Fling, work has been done on both the hypervisor as well as FreeBSD, to make the two more compatible with one another. Even though the work was initially done to make these two work better together, the work overall has been more general purpose for FreeBSD in support of both bare-metal Aarch64 installations as well as running FreeBSD under other hypervisors such as QEMU.

An example of others building off of this work is Twitter user astr0baby getting FreeBSD working under QEMU on a new Apple M1 system.

When ESXi-ARM Fling first released, to get FreeBSD to work under it, the process required taking the Aarch64 premade VMDK file, uploading it to the hypervisor storage, and then running a series of CLI commands to convert the disk image to a supported file format. The initial work done was to get the FreeBSD Aarch64 ISO bootable and with the required drivers to complete the install process. With this, users can do fresh installs of FreeBSD Aarch64 using the same methods they would use for AMD64 or i386 under ESXi.

The CD-ROM driver's inclusion into FreeBSD 12 barely missed the cut-off date for 12.2-RELEASE. However, the very first 12.2-STABLE build published for Aarch64 includes the CD-ROM driver. FreeBSD 13-CURRENT also includes this driver. Due to this, only 12-STABLE and 13-CURRENT support fresh CD ISO installations.

The next step was getting the major pieces of virtual hardware working. This included adding more USB controllers, the vmxnet virtual network card, and pvscsi para-virtual SCSI drivers added to Aarch64 GENERIC.

There is a known bug in ESXi-ARM Fling's virtual UEFI that prevents booting from pvscsi, so for the time being the boot device must be on a virtual disk attached to the SATA controller inside the VM.

ESXi-ARM Fling uses a new virtual SVGA device which currently does not have working drivers on any platform, as the specifications are not finalized yet. Due to this, only efi-fb/scfb is available for console and Xorg for the time being.

The VMCI driver is currently not compiling at all. This driver has sections of x86 assembly code that will need to be converted over to ARM. This would be a great area for anyone to look into that is experienced with converting assembly language!

At ESXi-ARM Fling launch, there was a hypervisor bug preventing more than 1 vCPU from working inside FreeBSD. This has since been fixed, allowing up to 8 vCPUs. Going beyond this requires a a patch to FreeBSD, which was authored by VMWare developer Cypou.

Things that are currently fixed/working:

Booting from CD ISO image
Virtual USB 2.0 controller
Virtual USB 3.1 controller
Virtual USB Keyboard
Virtual USB Mouse
vmxnet3 Virtual Network Card
pvscsi Para-Virtual SCSI Storage Controller
open-vm-tools Guest Virtual Machine Tools
Xorg Enhanced Mouse Driver (untested)
Multi-Core CPU (up to 8 vCPUs inside guest)

Things that are still broken:

Booting from pvscsi
Xorg SVGA Driver
vmci Virtual Machine Communication Interface
Multi-Core CPU (more than 8 vCPUs)

With all of this done, it has made working on the Aarch64 ports

collection easier by having a high quality virtualization environment for development and testing. This environment has already been used to update the ZeroTier port and Facebook's RocksDB used in the MariaDB port.

FreeBSD now has a Discord chat! Discussion about FreeBSD on Aarch64 in general takes place in our #embedded channel. Despite the name, we discuss all levels of ARM development, from large servers, to virtualized environments, all the way down to single board computers.

Bastille

Links
Bastille GitHub	URL: https://github.com/bastillebsd/bastille
Bastille Templates	URL: https://gitlab.com/bastillebsd-templates
Bastille Website	URL: https://bastillebsd.org

Contact: Christer Edwards <christer.edwards@gmail.com>

Bastille is an open-source system for automating deployment and management of containerised applications on FreeBSD.

Bastille Templates automate container setup allowing you to easily reproduce containers as needed.

Bastille is available in ports as sysutils/bastille.

Q4 2020 Status

In Q4 2020 Bastille merged some exciting new features. Changes include:

full adoption of the previously experimental Bastillefile format
new config sub-command
default templates included and applied by default
support for -CURRENT jails on -CURRENT hosts
support for 32bit containers on 64bit hosts
support in templates for dynamic arguments & defining variables
over two dozen bug fixes and general improvements

More details about Bastille Releases.

upstream was updated to 0.8.202010101 (latest). ports (sysutils/bastille) was updated to 0.7.20200414. -

CheriBSD

Links

CHERI Project homepage - URL: http://www.cheri-cpu.org

CHERi github repo - URL: https://github.com/CTSRD-CHERI/cheribsd

Morello Platform landing page

CheriBSD

Links
CHERI Project Homepage	+ URL: http://www.cheri-cpu.org
CheriBSD GitHub Repo	+ URL: https://github.com/CTSRD-CHERI/cheribsd
Morello Platform Landing Page	URL: https://www.morello-project.org
CheriBSD Morello Developer Preview	- URL: https://morello-dist.cl.cam.ac.uk
ARM Morello Program	+ URL: https://morello-dist.cl.cam.ac.uk
Arm Morello Program	URL: https://developer.arm.com/architectures/cpu-architecture/a-profile/morello

Contact: Alex Richardson <arichardson@FreeBSD.org>
Contact: Andrew Turner <andrew@FreeBSD.org>
Contact: Brooks Davis <brooks@FreeBSD.org>
Contact: Edward Tomasz Napierala <trasz@FreeBSD.org>
Contact: George Neville-Neil <gnn@FreeBSD.org>
Contact: Jessica Clarke <jrtc27@FreeBSD.org>
Contact: John Baldwin <jhb@FreeBSD.org>
Contact: Robert Watson <rwatson@FreeBSD.org>
Contact: Ruslan Bukin <br@FreeBSD.org>

Arm Morello - In October, we released a developer preview of CheriBSD ported to Arm's Morello architecture. This release supports a dynamically linked runtime and is generally functional. It was cut from a development branch and work is in progress to merge the contents of this branch with the CheriBSD main line. We anticipate producing a new release from this branch in early 2021.
Kernel spatial memory safety (pure-capability kernel) - The current CheriBSD kernel is a hybrid C program where only pointers to userspace are CHERI capabilities. This ensures that the kernel follows the intent of the application runtime and cannot be used to defeat bounds on application pointers. We have developed and will soon merge a pure-capability kernel where all pointers in the kernel are appropriately bounded capabilities. This vastly reduces the opportunity for buffer overflows. This spatial memory safety lays the groundwork for future work such as device driver compartmentalization and kernel temporal safety.
Userspace heap temporal memory safety (Cornucopia) - CHERI capabilities provide the necessary features to enable robust and efficient revocation of freed pointers. With Cornucopia we have implemented a light-weight revocation framework providing protection from use-after-reallocation bugs with an average cost below 2%. We aim to bring these overheads down further over the next year and merge this functionality into the mainline CheriBSD.
Syncing with upstream FreeBSD - We spent considerable time this quarter catching up with FreeBSD-CURRENT. As of the beginning of December, we had caught up. Merges are currently paused while we work to land Morello and pure-capability kernel changes. In the interim, we have performed a test merge between our tree based on the legacy export of the FreeBSD tree to git and the new FreeBSD git repository. The process went smoothly and is expected to have few impacts.
We have been working on updating the arm64 bhyve from Politehnica University of Bucharest to have it committed to FreeBSD. We have been upstreaming initial changes to help support this.

Embedded Lab Project

Links
FreeBSD Embedded Lab Design	URL: https://www.funkthat.com/gitea/jmg/fbsdembdev
Lab API code	URL: https://www.funkthat.com/gitea/jmg/bitelab

Contact: John-Mark Gurney <jmg@FreeBSD.org>

The Embedded Lab Project's goal is to make SBCs and other devices more accessible to developers. Despite SBCs often being inexpensive, it is not inexpensive to maintain them, in terms of the cost of time to keep them up to date, infrastructure to support them, etc.

The goal of this project is to support and enhance the existing CI work but also make it easier for developers to test their code and changes on one, or many different boards.

Once the work is [mostly] complete, I will host a lab that will be freely available to everyone who has a FreeBSD.org account. Information about this will be sent once it is closer to launch.

The core part of the architecture is each time a board is reserved via the API, a new jail is created which contains the serial console tty, an interface for internet access, and an interface that is connected to the board's ethernet port (assuming it has one). This allows a clean system for each run, and allows complete control over the network interfaces to support netbooting and other development. The jail will have a basic set of FreeBSD packages installed that matches the board.

Part of the API will also allow power cycling the board to aid in debugging. This part is relatively extensible, so adding additional modules to provide additional support should not be difficult.

The API includes support for running interactive commands in the jail. This will make it easy to script control of the environment, such as directly running an expect script against the serial console, or even just running a script in the jail.

The work is progressing well, and currently a single board, a Pine64 A64-LTS, is integrated and working. Board reserves and releases are working, along with the ability to run commands in the jail via the API. Power control is functional, and is currently using a PoE smart switch to control power.

Work has stalled on being able to use the SDWire with an environment due to power issues. USB is not made for power isolation, which is causing issues w/ power control. The existing board, the A64-lTS, is using a USB serial console adapter that is opto-isolated, ensuring that there is no problems w/ power control. But there I have not found a solution for high speed USB. I believe that cutting the VBUS (power) line of a USB cable would allow fine grain power control, but tests have not been conducted yet.

Sponsor: The FreeBSD Foundation

FreshPorts

Links
FreshPorts	URL: http://freshports.org/
FreshPorts blog	URL: http://news.freshports.org/

Contact: Dan Langille <dan@langille.org>

FreshPorts, and its sister site, FreshSource, have reported upon FreeBSD commits for 20 years. They cover all commits, not just ports.

FreshPorts tracks the commits and extracts data from the port Makefiles to create a database of information useful to both port developers and port users.

For example, https://www.freshports.org/security/acme.sh/ shows the history of this port, back to its creation in May 2017.

git

The work to become git-ready is mostly complete. Both src and doc commits are flowing into devgit.freshports.org. Some work is required on various issues, but nothing that stops the flow of commits into the database.

Help wanted

Amazon have donated enough to try FreshPorts on AWS. I need help with the following:

getting IPv6 working
working with RDS

If you can help with this, please contact me. Thank you.

Thank you

helloSystem

Links
Documentation	URL: https://hellosystem.github.io/docs/

Contact: Simon Peter <probono@puredarwin.org>

Contact: #helloSystem on irc.freenode.net, mirrored to #helloSystem:matrix.org on Matrix

helloSystem is FreeBSD preconfigured as a desktop operating system with a focus on simplicity, elegance, and usability. Its design follows the “Less, but better” philosophy. It is intended as a system for “mere mortals”, welcoming to switchers from a world in which a global menu bar exists, the Command key is used rather than Control, and applications are contained in .app bundles.

helloSystem grew out of frustration with usability shortcomings of existing open source desktop environments. FreeBSD was chosen as the base because it offers one consistent base system rather than a fragmented landscape of distributions lacking a common platform.

helloSystem aims at providing a "it just works" out-of-the-box user experience in which a non-technical user can just use the system without ever opening the terminal, without having to configure anything, and without ever seeing white text on a black background scroll by during system boot. Technologies embraced include DNS-SD/Zeroconf (also known as Bonjour), IPP Everywhere (also known as AirPrint), eSCL (also known as AirScan), etc.

Prerelease installable Live ISO images are available.

Help is needed in a number of areas, especially:

FreeBSD/kernel: allowing to put the system into a read-only disk image with a writable overlay, e.g., using unionfs
Qt, Python: writing various easy-to use frontends for FreeBSD/OpenZFS functionality, e.g., Disk Utility.app
Testing and bugfixing

K8S-bhyve

Links
K8S-bhyve	URL: https://k8s-bhyve.convectix.com
K8S-bhyve	URL: https://github.com/k8s-bhyve
Kubernetes	URL: https://kubernetes.io/

Contact: Kirill Ponomarev <krion@FreeBSD.org>
Contact: Oleg Ginzburg <olevole@olevole.ru>

K8S-bhyve is opensource project concentrating primarily on deploying and use of kubernetes on FreeBSD/bhyve in a more agile and more comfortable manner. We are going to provide distributed multi-DC environment or just stand-alone clusters with native PV/PVC support.

For 2020Q4 we made and published a k8s-bhyve image which you might install with ISO/memstick, as well as with bsdinstall.

Puppet

Links
Puppet	URL: https://puppet.com/docs/puppet/latest/puppet_index.html
Puppet's FreeBSD slack channel	URL: https://puppetcommunity.slack.com/messages/C6CK0UGB1/
Bolt	URL: https://puppet.com/docs/bolt/latest/bolt.html
Choria	URL: https://choria.io/

Contact: Puppet Team <puppet@FreeBSD.org>

Since our last status report a few months ago, the FreeBSD ports tree has seen the addition of the Choria (sysutils/choria) orchestration tool, and the Puppet Platform 7 with the Puppet Agent (sysutils/puppet7), Puppet Server (sysutils/puppetserver7) and PuppetDB (databases/puppetdb7).

Older versions of Puppet (5 and 6) are still in the ports tree, allowing a smooth transition, but please note that Puppet 5 will reach EOL soon, and as it is not compatible with the recent ecosystem provided by FreeBSD (i.e. it is not compatible with the latest version of Ruby and depends on old FreeBSD primitives not available anymore), it is recommended to update at least to Puppet 6 as soon as possible.

Ports depending on Puppet (e.g. sysutils/rubygem-bolt) have been updated to add options allowing to choose which version of Puppet to depend on. For now, the default is Puppet 6, but we plan to switch the default to Puppet 7 in a few weeks, probably when Puppet 5 will have reached EOL.

News Home | Status Home

Header And Logo

Peripheral Links

Search

Site Navigation

Section Navigation

Introduction

COVID-19 Impact on the Foundation

Partnerships and Commercial User Support

Fundraising Efforts

OS Improvements

University of Waterloo Co-op

Continuous Integration and Quality Assurance

Supporting FreeBSD Infrastructure

FreeBSD Advocacy and Education

Foundation Board Meeting

Legal/FreeBSD IP

Background

Implementation overview

Status

TODO

Q2 2020 Status

Q2 2020 Status

Languages

git

Help wanted

Packages

Dependency lines

Watch ports I maintain

Repology links

Further reading

Javascript help wanted

Header And Logo

Peripheral Links

Search

Site Navigation

Section Navigation

Introduction

COVID-19 Impact to the Foundation

Partnerships and Commercial User Support

Fundraising Efforts

OS Improvements

University of Waterloo Co-op

Continuous Integration and Quality Assurance

Supporting FreeBSD Infrastructure

FreeBSD Advocacy and Education

Legal/FreeBSD IP

Other

Atheros 11ac driver athp

Newer Intel Wireless device support