Index: head/en_US.ISO8859-1/htdocs/news/status/Makefile =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/Makefile +++ head/en_US.ISO8859-1/htdocs/news/status/Makefile @@ -83,6 +83,7 @@ XMLDOCS+= report-2018-01-2018-09 XMLDOCS+= report-2018-09-2018-12 XMLDOCS+= report-2019-01-2019-03 +XMLDOCS+= report-2019-04-2019-06 XSLT.DEFAULT= report.xsl Index: head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml +++ head/en_US.ISO8859-1/htdocs/news/status/report-2019-04-2019-06.xml @@ -0,0 +1,2458 @@ + + + + + + + + + + April-June + + 2019 + + +
+ Introduction + +

This quarter our report includes + some interesting topics easily accessible to anyone, even if + you are not a programmer: we report the link to a presentation + of the 2019 FreeBSD survey results at BSDCan 2019 and describe + an interesting experience of a 3-person hackaton, which might + encourage you to host one yourself, possibly with more participants. + We also provide some up to date information about the status + of our IRC channels.

+ +

For those who have some more technical skills, we give some + news about the role of git in the FreeBSD project, describe + the status of some tools to hunt bugs or enhance security and + announce a clone of sysctl.

+ +

Finally, those who are more experienced with programming will + probably be interested in the great work that has been done + with drivers: in particular, an aknowledgement is due to Alan + Somers for having started to bring up to date our FUSE + implementation, which was about 11 years behind. Other important + improvements include a more user-friendly experience with + trackpoints and touchpads enabled by default, much low level + work on graphics, many new bhyve features, updates to the + linux compatibility layer, various kernel improvements.

+ +

Have a nice read!
+ + -- Lorenzo Salvadore

+
+ + + team + + &os; Team Reports + +

Entries from the various official and semi-official teams, + as found in the Administration + Page.

+ + + + proj + + Projects + +

Projects that span multiple categories, from the kernel and userspace + to the Ports Collection or external projects.

+ + + + kern + + Kernel + +

Updates to kernel subsystems/features, driver support, + filesystems, and more.

+ + + + arch + + Architectures + +

Updating platform-specific features and bringing in support + for new hardware platforms.

. + + + + bin + + Userland Programs + +

Changes affecting the base system and programs in it.

+ + + + ports + + Ports + +

Changes affecting the Ports Collection, whether sweeping + changes that touch most of the tree, or individual ports + themselves.

+ + + + doc + + Documentation + +

Noteworthy changes in the documentation tree or new external + books/documents.

+ + + + misc + + Miscellaneous + +

Objects that defy categorization.

+ + + + third + + Third-Party Projects + +

Many projects build upon &os; or incorporate components of + &os; into their project. As these projects may be of interest + to the broader &os; community, we sometimes include brief + updates submitted by these projects in our quarterly report. + The &os; project makes no representation as to the accuracy or + veracity of any claims in these submissions.

+ + + + Release Engineering Team + + + + FreeBSD Release Engineering Team + re@FreeBSD.org + + + + + FreeBSD 11.3-RELEASE schedule + FreeBSD 11.3-RELEASE announcement + FreeBSD 12.1-RELEASE schedule + FreeBSD development snapshots + + + +

The FreeBSD Release Engineering Team is responsible for + setting + and publishing release schedules for official project + releases + of FreeBSD, announcing code freezes and maintaining the + respective branches, among other things.

+ +

During the second quarter of 2019, the FreeBSD Release + Engineering team + started the 11.3-RELEASE cycle, with the code slush + starting May 3rd. + Throughout the cycle, there were three BETA builds and + three RC builds, + all of which in line with the originally-published + schedule. The final RC + build started June 28th, with the final release build + targeted for July 5th.

+ +

FreeBSD 11.3-RELEASE will be the fourth release from the + stable/11 + branch, building on the stability and reliability of + 11.2-RELEASE.

+ +

The FreeBSD Release Engineering Team also published the + schedule for the + 12.1-RELEASE, targeted to start September 6th. One + important thing to note + regarding the published schedule is it excludes a hard + freeze on the + stable/12 branch, as a test run for eliminating + code freezes entirely during + a release cycle. Commits to what will be the + releng/12.1 branch will still + require explicit approval from the Release Engineering + Team, however.

+ +

Additionally throughout the quarter, several development + snapshots builds + were released for the head, stable/12, + and stable/11 branches.

+ +

Much of this work was sponsored by the FreeBSD Foundation + and Rubicon + Communications, LLC (Netgate).

+ + + + + + + Ports Collection + + + + René Ladan + portmgr-secretary@FreeBSD.org + + + FreeBSD Ports Management Team + portmgr@FreeBSD.org + + + + + About FreeBSD Ports + Contributing to Ports + FreeBSD Ports Monitoring + Ports Management Team">Ports Management Team + + + +

The following was done during the last quarter by portmgr + to keep things in + the Ports Tree going:

+ +

During the last quarter the number of ports rose to just + under 37,000. At the + end of the quarter, there were 2146 open PRs and 7837 + commits (excluding 499 on + the quarterly branch) from 172 committers. This shows a + slight decrease in + activity compared to previous quarter.

+ +

People come and go, last quarter we welcomed Pedro Giffuni + (pfg@), Piotr Kubaj + (pkubaj@) and Hans Petter Selasky (hselasky@). Pedro and + Hans Petter were + already active as src committers. We said goodbye to + gordon@, kan@, tobez@, + and wosch@.

+ +

On the infrastructure side, a new USES=cabal was + introduced and various default + versions were updated: MySQL to 5.7, Python to 3.6, Ruby + to 2.5, Samba to 4.8 + and Julia gained a default version of 1.0. The web + browsers were also updated: + Firefox to 68.0 and Chromium to 75.0.3770.100

+ +

During the last quarter, antoine@ ran a total of 41 + exp-runs to test various + package updates, bump the stack protector level to + "strong", switch the default + Python version to 3.6 as opposed to 2.7, remove sys/dir.h + from base which has + been deprecated for over 20 years, and convert all Go + ports to USES=go.

+ + + + + + + FreeBSD Core Team + + + + FreeBSD Core Team + core@FreeBSD.org + + + + +

The FreeBSD Core Team is the governing body of FreeBSD.

+ + + + + + + +

+ There is a variety of viewpoints within core regarding + where and how + to host a Git repository, however core feels that Git is + the prudent + path forward.

+ + + + + + + + + Continuous Integration + + + + Jenkins Admin + jenkins-admin@FreeBSD.org + + + Li-Wen Hsu + lwhsu@FreeBSD.org + + + + + FreeBSD Jenkins Instance + FreeBSD CI artifact archive + FreeBSD Jenkins wiki + freebsd-testing Mailing List + freebsd-ci Repository + Tickets related to freebsd-testing@ + Hosted CI wiki + FreeBSD CI weekly report + + + +

The FreeBSD CI team maintains continuous integration + system and related tasks + for the FreeBSD project. The CI system regularly checks + the committed changes + can be successfully built, then performs various tests and + analysis of the + results. The results from build jobs are archived in an + artifact server, for + the further testing and debugging needs. The CI team + members examine the + failing builds and unstable tests, and work with the + experts in that area to + fix the code or adjust test infrastructure. The details + are of these efforts + are available in the weekly CI reports.

+ +

The + FCP + for CI policy + is in "feedback" state, please provide any comments to + freebsd-testing@ or + other suitable lists.

+ +

We had a testing working group in 201905 + DevSummit

+ +

Please see freebsd-testing@ related tickets for more + information.

+ +

Work in progress:

+ + + + + + + + + FreeBSD Graphics Team status report + + + + FreeBSD Graphics Team + x11@freebsd.org + + + Niclas Zeising + zeising@freebsd.org + + + + + Project GitHub page + + + +

The FreeBSD X11/Graphics team maintains the lower levels + of the FreeBSD graphics + stack. + This includes graphics drivers, graphics libraries such as + the + MESA OpenGL implementation, the X.org xserver with related + libraries and + applications, and Wayland with related libraries and + applications.

+ +

In the last report, half a year ago, several updates and + changes had been made + to the FreeBSD graphics stack.

+ +

To further improve the user experience, and to improve + input device handling, + evdev was enabled in the default configuration in late + 2018. Building on that, + we have enabled IBM/Lenovo trackpoints and elantech and + synaptics touchpads by + default as well.

+ +

The input device library libinput has been updated as the + last in a series of + updates bringing the userland input stack up to date. + This is work that was started in 2018.

+ +

We have made several improvements to the drm kernel + drivers. + A long-standing memory leak in the Intel (i915) driver has + been fixed, and + several other updates and improvements have been made to + the various drm + kernel driver components.

+ +

A port of the drm kernel drivers using the 5.0 Linux + kernel sources has been + created and committed to FreeBSD ports as + graphics/drm-devel-kmod. + This driver requires a recent Linux KPI and is only + available on recent + versions of FreeBSD CURRENT.

+ +

This version of the driver contains several development + improvements. + The generic drm (drm.ko) driver as well as the i915 + (i915kms.ko) driver + can now be unloaded and reloaded to ease in development + and testing. + This causes issues with the virtual consoles, however, so + an SSH connection is + recommended. + To aid debugging i915kms.ko use of debugfs has + been improved, but there are + still limitations preventing it from being fully + functional. + Since debugfs is based on pseudofs it is possible that + this will prevent a fully + functional debugfs in its current state, so we might have + to look into adding + the required functionality to pseudofs or use another + framework.

+ +

The new in-kernel drm driver for VirtualBox, + vboxvideo.ko has been ported from + Linux. + Support is currently an experimental work in progress. + For example the virtual console won't update after loading + the driver, but X- + and Wayland-based compositors are working.

+ +

Mesa has been updated to 18.3.2 and switched from using + devel/llvm60 to use + the Ports default version of llvm, currently + devel/llvm80.

+ +

Several userland Xorg drivers, applications, and libraries + have been updated, + and other improvements to the various userland components + that make up the + Graphics Stack have been made.

+ +

We have also continued our regularly scheduled bi-weekly + meetings, although work + remains in sending out timely meeting minutes afterwards.

+ +

People who are interested in helping out can find us on + the x11@FreeBSD.org + mailing list, or on our gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. + We are also available in #freebsd-xorg on EFNet.

+ +

We also have a team area on GitHub where our work + repositories can be found: + https://github.com/FreeBSDDesktop

+ + + + + + + IRC Admin + + + + IRC Admin + irc@FreeBSD.org + + + + +

The FreeBSD IRC Admin team manages the FreeBSD Project's + presence + and activity on the freenode IRC network, looking after:

+ + + +

+ While the FreeBSD Project does not _currently_ endorse IRC + as an + official support channel [1][2], as it has not been able + to guarantee + a consistent or positive user experience, IRC Admin has + been working + toward creating a high quality experience, by + standardising channel + administration and moderation expectations, and ensuring + the projects + ability to manage all channels within its namespace.

+ +

In the last quarter, IRC Admin:

+ + + +

+ Upcoming changes:

+ + + +

+ Lastly, and to repeat a previous call, while the vast + majority of + the broader user community interacts on the freenode IRC + network, + the FreeBSD developer presence still needs to be + significantly + improved on freenode.

+ +

There are many opportunities to be had by increasing the + amount and + quality of interaction between FreeBSD users and + developers, both + in terms of developers keeping their finger on the pulse + of the + community and in encouraging and cultivating greater + contributions + to the Project over the long term.

+ +

It is critical to have a strong developer presence amongst + users, + and IRC Admin would like again to call on all developers + to join + the FreeBSD freenode channels to increase that presence.

+ +

Users are invited to /join #freebsd-irc on the + freenode IRC network + if they have questions, ideas, constructive criticism, and + feedback + on how the FreeBSD Project can improve the service and + experience + it provides to the community on IRC.

+ +

[1] https://www.freebsd.org/community/irc.html + [2] + https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/support.html#irc

+ + + + + + + bhyve - Live Migration + + + + Elena Mihailescu + elenamihailescu22@gmail.com + + + Darius Mihai + dariusmihaim@gmail.com + + + Mihai Carabas + mihai@freebsd.org + + + + + Github wiki - How to Live and Warm Migrate a bhyve guest + Github - Warm Migration branch + Github - Live Migration branch + + + +

The Migration feature uses the Save/Restore feature to + migrate a bhyve guest + from a FreeBSD host to another FreeBSD host. To migrate a + bhyve guest, + one needs to start an empty guest on the destination host + from a shared guest + image using the bhyve tool with the -R option + followed by the source host + IP and the port to listen to migration request. On the + source host, the + migration is started by executing the bhyvectl command + with the --migrate + or --migrate-live option, followed by the + destination host IP and the + port to send to the messages.

+ +

New features added:

+ + + +

+ Future tasks:

+ + + + + + + Matthew Grooms + + + + + + bhyve - Save/Restore + + + + Elena Mihailescu + elenamihailescu22@gmail.com + + + Darius Mihai + dariusmihaim@gmail.com + + + Mihai Carabas + mihai@freebsd.org + + + + + Github repository for the snapshot feature for bhyve + Github wiki - How to Save and Restore a bhyve guest + Github wiki - Suspend/resume test matrix + Phabricator review - bhyve Snapshot Save and Restore + + + +

The Save/Restore for bhyve feature is a suspend and resume + facility added to the + FreeBSD/amd64's hypervisor, bhyve. The bhyvectl tool is + used to save the guest + state in three files (a file for the guest memory, a file + for the states of + various devices and the state of the CPU, and another one + for some metadata that + is used in the restore process). + To suspend a bhyve guest, the bhyvectl tool must be run + with the --suspend + <state_file_name> + option followed by the guest name.

+ +

To restore a bhyve guest from a checkpoint, one simply has + to add the -r option + followed by the main state file (the same file that was + given to the --suspend + option for bhyvectl) when starting the VM.

+ +

New features added:

+ + + +

+ Future tasks:

+ + + + + + + Matthew Grooms + + + + + + ENA FreeBSD Driver Update + + + + Michal Krawczyk + mk@semihalf.com + + + Maciej Bielski + mba@semihalf.com + + + Marcin Wojtas + mw@semihalf.com + + + + + ENA README + + + +

ENA (Elastic Network Adapter) is the smart NIC available + in the + virtualized environment of Amazon Web Services (AWS). The + ENA + driver supports multiple transmit and receive queues and + can handle + up to 100 Gb/s of network traffic, depending on the + instance type + on which it is used.

+ +

ENAv2 has been under development for FreeBSD, similar to + Linux + and DPDK. Since the last update internal review and + improvements + of the patches were done, followed by validation on + various AWS + instances.

+ +

Completed since the last update:

+ + + +

+ Todo:

+ + + + + + + Amazon.com Inc + + + + + + FUSE + + + + Alan Somers + asomers@FreeBSD.org + + + + +

FUSE (File system in USErspace) allows a userspace program + to + implement a file system. It is widely used to support + out-of-tree file + systems like NTFS, as well as for exotic pseudo file + systems like + sshfs. FreeBSD's fuse driver was added as a GSoC project + in 2012. + Since that time, it has been largely neglected. The FUSE + software is + buggy + and out-of-date. Our implementation is about 11 years + behind.

+ +

During Q2 I nearly finished the FUSE overhaul that I + begain in Q1. I raised + the protocol level from 7.8 to 7.23, fixed many bugs (see + 199934, + 216391, + 233783, + 234581, + 235773, + 235774, + 235775, + 236226, + 236231, + 236236, + 239291, + 236329, + 236379, + 236381, + 236405, + 236327, + 236466, + 236472, + 236473, + 236474, + 236530, + 236557, + 236560, + 236647, + 236844, + 237052, + 237181, + 237588, + and + 238565), + and added + the following features:

+ + + +

+ I also added the following performance enhancements:

+ + + +

+ All that remains is to finish merging the branch, and deal + with any newly + introduced bugs.

+ + + + + The FreeBSD Foundation + + + + + + Kernel ZLIB Update + + + + Yoshihiro Ota + ota@j.email.ne.jp + + + + + Review D19706 + + + +

Kernel zlib upgrade is in progress.

+ +

Xin (delphij@) and I have been working closely for zlib + upgrade. + We relocated contrib/zlib to sys/contrib/zlib in order for + kernel + code to access zlib in the tree. We also deleted dead code + that + depended on zlib and inflate - inflate is a fork of unzip + to + uncompress gzip files. We also renamed crc.h to avoid + conflicts + with zlib/crc.h.

+ +

Next goal is to compile both old zlib and new zlib into + the kernel + allowing to switch each zlib user independently.

+ + + + + + + Linux compatibility layer update + + + + Edward Tomasz Napierala + trasz@FreeBSD.org + + + + +

The project aims to improve the Linux compatibility layer, + to make + it more compatible with recent Linux releases, and also to + lower + the bar for potential developers who want to start + contributing to it.

+ +

The initial effort focused on tooling, to make it easier + to debug + problems and to prevent future regressions. The first part + involved + making it possible to use Linux strace(1) utility and + providing it + as linux-c7-strace package. The reason is that + while FreeBSD + truss(1) and ktrace(1) can trace Linux + binaries, they cannot + decode Linux-specific flags and structures.

+ +

The second part involved providing Linux Test Project + binaries as + linux-ltp package. There is ongoing work to hook + it up to the + FreeBSD CI infrastructure http://ci.FreeBSD.org.

+ +

There was also a number of improvements and fixes to bugs + discovered + in the process. One of them (not yet committed) fixes + binaries + linked against newer version of libc, effectively + unbreaking binaries + from recent Ubuntu releases.

+ + + + + FreeBSD Foundation + + + + + + Lock-less delayed invalidation for amd64 pmap + + + + Konstantin Belousov + kib@freebsd.org + + + + +

The Virtual Memory machine-dependent layer (pmap) on amd64 + needs to + track all mappings for the managed physical memory pages, + to be able + to either destroy all of them (for page-out), or change + them from + writeable to read-only (e.g. to sync the page content to + file, without + racing with modifications through user writes). The + mappings are + accounted by creating pv_entry which records the address + space + (implicitly, by linking the pv entry to pmap) and the + virtual address + of the mapping.

+ +

Previous work split the lock protecting the pv entries + lists from + other VM locks into the pvh_global_lock lock, which was + global for all + address spaces. You can see it in i386 pmap.c still. + Later, hashed + per-page pv lists locks were introduced, which would + reduce contention + on pv lists maninulations for different pages, but + unfortunately the + pvh_global_lock was still needed to guarantee the safety + of some + operations.

+ +

Problem arises because amd64 pmap uses pmap lock to + protect page + tables and TLB consistency, which is per-pmap locks + different from pv + lists locks. When updating page table entry, we never drop + pmap lock + until the necessary TLB invalidation is done globally, + including + signalling other CPUs with IPI. But pv list locks can be + unlocked + before the necessary invalidation is done. So for instance + when pmap + is asked to remove all mappings of the specific page + (pmap_remove_all(9)), it checks pv list of the page to + find the + mappings. The list might appear empty despite other CPUs + TLB were not + yet invalidated. If such page is reused, other CPUs might + change its + content using cached TLB entries. Allowing that means + allowing both + silent data corruption and opening security hole.

+ +

So the global pvh lock was held until all pmaps + invalidated their + TLBs. This mechanism has obvious scalability issues, and + instead a + generation-count based scheme for handling delayed + invalidation (DI) + was developed, where each thread that might remove entry + from pv list + acquired a generation number and marked the page with it, + see + pmap_delayed_invl_page(9). Then, on e.g. + pmap_remove_all(9) or + pmap_remove_write(9), pmap code waits for the maximum + current thread's + invalidation generation number to pass the page's + generation, which + guarantees that all required TLB invalidations are done.

+ +

Original implementation of DI allowed to get rid of + pvh_global_lock, + and only used a private mutex to handle sequential + queueing of the + coming and leaving threads, protecting a bounded region. A + problem + with that appeared e.g. in scalability benchmarks which + did massive + parallel unmaps, causing most of the threads to contend on + DI + queueing.

+ +

Current implementation of DI switched to lock-less queue + algorithm + using the approach proposed by T.L. Harris and relying on + double-CAS + to coalesce generation count and queueing. It uses ifuncs + to select + either previous locked DI or current lock-less + implementation, only + old AMD Athlons which did not implemented the CMPXCHG16B + instruction + falls to the locked implementation by default. Lock-less + implementation still blocks the waiting thread on + turnstile to avoid + priority-inversion issues, but practically the wait occur + very rare, + typical parallel buildworld generates single-digit number + of the + events.

+ +

The patch got a lot of testing from Peter Holm, continuous + reviews by + Mark Johnston while I worked out bugs and live-lock + problems in the + implementation, and additional testing by Mateusz Guzik + who helped to + identify a priority inversion bug with the wait.

+ + + + + The FreeBSD Foundation + + + + + + Mellanox Drivers Update + + + + Slava Shwartsman Hans Petter Selasky Konstantin Belousov + freebsd-drivers@mellanox.com + + + + + Mellanox OFED for FreeBSD Documentation + + + +

The mlx5 driver provides support for ConnectX-4 [Lx], + ConnectX-5 [Ex] and + ConnectX-6 [Dx] adapter cards. The mlx5en driver provides + support for Ethernet + adapter cards, whereas mlx5ib driver provides support for + InfiniBand adapters + and RDMA over Converged Ethernet (RoCE).

+ +

Following updates done in mlx5 drivers:

+ + + +

+ General updates:

+ + + + + + + Mellanox Technologies + + + + + + NFSv4.2 client/server implementation for FreeBSD + + + + Rick Macklem + rmacklem@freebsd.org + + + + + current sources + + + +

NFSv4.2 is a newer minor version of NFSv4, made up of a + set of optional + operations/features. A majority of these operations are + related to + the POSIX operations posix_fadvise(2), posix_fallocate(2) + and lseek(2)'s + support for SEEKHOLE/SEEKDATA. There is also a Copy + operation that allows + a byte range of a file to be copied to another file + locally on the NFS + server, avoiding data transfer over the wire in both + directions. + FreeBSD-current now has a Linux compatible + copy_file_range(2) syscall + that will invoke this Copy operation on NFSv4.2 mounts. + There is also support for MAC labelling, but it requires + changes to the + RPCSEC_GSS implementation to add V3 support and, as such, + may not happen + soon.

+ +

The implementation of NFSv4.2 (RFC-7862) is progressing + nicely. + At this time, the LayoutError, IOAdvise, Allocate and Copy + operations + have been implemented. There is still work to be done on + Copy, to add + asynchronous support, so that large copies do not result + in a long delay + for the RPC's reply.

+ +

The major operation that will be implemented next is Seek, + so that + lseek(SEEKHOLE/SEEKDATA) will work for the NFSv4.2 mounts.

+ +

It is hoped that this implementation will be ready for + FreeBSD-current/head + in time for the FreeBSD-13 release.

+ +

Testing is always appreciated and can be done by + downloading the modified + kernel from the svn repository in base/rojects/nfsv42 and + then building + and testing it on a couple of recent FreeBSD-current + systems.

+ +

If anyone is conversant with Kerberos and wants to take on + the challenge + of adding RPCSEC_GSS_V3 support to the kernel RPC, a patch + that does + that would also be greatly appreciated.

+ + + + + + + NUMA awareness in the FreeBSD kernel + + + + Jeff Roberson + jeff@FreeBSD.org + + + Andrew Gallatin + gallatin@FreeBSD.org + + + Mark Johnston + markj@FreeBSD.org + + + + +

A set of patches to improve the state of NUMA awareness in + the FreeBSD + kernel are being developed and refined. This work also + aims to + generally improve the performance of FreeBSD's memory + management + subsystem on systems with many CPUs.

+ +

FreeBSD 12.0 featured a number of large changes which + improve its + performance on systems with a non-uniform memory + architecture. That is, + systems in which memory access latency for a given address + varies + depending on the CPU. Another round of improvements is + being developed + and will soon be available in FreeBSD-CURRENT. Short + descriptions of + some of these patches follow; a few have already been + committed to + FreeBSD-CURRENT.

+ +

In FreeBSD terminology, a memory page whose contents may + not be evicted + is referred to as "wired." Pages may be wired under + different + circumstances: for instance, all kernel memory is wired, + and userland + applications may request that ranges of memory be wired + using the + mlock(2) and mlockall(2) system calls. FreeBSD has + historically defined + a system-wide limit on the number of wired pages so as to + avoid + deadlocks that may arise when too much of a system's + memory cannot be + reclaimed to satisfy new memory allocations. This limit + was applied + only to userland wiring requests, but kernel wirings were + counted + against the limit, so a large source of kernel wirings + could cause + mlock(2) failures. This occurs frequently with a large ZFS + ARC, for + example. In FreeBSD-CURRENT this limit has been changed + such that only + userland wirings are counted against the limit; the kernel + contains a + number of mechanisms to apply back-pressure to kernel + memory usage, so + the use of a global limit on all wirings did not provide + much benefit. + This fixes a common problem on large ZFS systems, and + helps enable some + other architectural improvements to the code which manages + page wirings.

+ +

FreeBSD has historically maintained two separate reference + counters in + the structure which describes a single physical page of + memory. These + counters initially had quite different properties, but + have over time + become more and more similar. Some work to merge the two + counters has + landed in FreeBSD-CURRENT. This does not have any + user-visible effects, + but it simplifies the page management code and removes a + large amount of + code which existed solely to transform references of one + type to the + other. Such code also made use of heavily contended locks, + so the + simplification improved kernel scalability for some + workloads and has + enabled further scalability improvements.

+ +

UMA is the slab allocator used in FreeBSD's kernel. It is + the backend + which services virtually all dynamic memory allocations + performed in the + kernel. The first round of NUMA improvements added NUMA + awareness to + the "keg" layer of UMA, which allocates and manages slabs. + However, the + frontend of UMA, which provides several layers of caching + for objects, + did not provide domain-aware caching, so over time the + caches would + become "polluted" with objects from different memory + domains. However, + this caching layer is being modified to ensure that + objects from + different memory domains are partitioned, helping ensure + that consumers + can perform domain-local allocations and frees + efficiently. This will + enable a global "first-touch" allocation policy for + UMA-managed objects.

+ +

During boot, the FreeBSD kernel allocates a number of + static data + structures to track physical memory. These structures have + historically + lived in the lowest available range of physical memory, so + they many not + inhabit the same NUMA domain as the memory that they + track. This is + suboptimal when one tries to affinitize a workload to a + particular NUMA + domain: if while executing the workload the kernel + frequently accesses + page structures for local memory, and the page structures + themselves + are not placed in local memory, the kernel will perform + many remote + memory accesses. Some in-progress work for the amd64 + platform creates + multiple arrays of page tracking structures, one per NUMA + domain, and + ensures that each array is local to its domain. This + complicates the + task of initializing kernel data structures during boot, + but can + substantially reduce the amount of cross-domain + communication that + occurs while the kernel is performing useful work. + Similarly, some + patches to affinitize per-CPU structures are being + developed; while + most per-CPU memory allocations already return CPU-local + memory, some + structures allocated during boot are not yet properly + placed with + respect to the accessing CPU's memory domain.

+ + + + + Netflix + + + + + + FreeBSD SDIO and Broadcom FullMAC WiFi Support + + + + Bjoern Zeeb + bz@FreeBSD.ORG + + + + + FreeBSD Wiki SDIO page + + + +

SDIO is an interface designed as an extension to SD Cards + to allow attachments of various other peripherals, e.g., + WiFi or Bluetooth.

+ +

Work has been ongoing by Ilya Bakulin on the MMCCAM stack + to provide the infrastructure to be able to have SD cards + and SDIO devices attached side-by-side facilitating + FreeBSD's CAM framework. + Based on this excellent work over the last years, + SDIO support was finished earlier this year and committed + to FreeBSD HEAD with the intention to merge to 12 at a + later time.

+ +

Facilitating the newly available SDIO bus, work started to + port Broadcom's FullMAC WiFi driver. This work is still + in progress and expected to complete later this year. + With this WiFi support for the Raspberry Pi and other + embedded boards will become available. + Likewise drivers for other SDIO devices can be developed + now.

+ + + + + The FreeBSD Foundation + + + + + + BIO_DELETE support for the swap pager + + + + Doug Moore + dougm@FreeBSD.org + + + Alan Cox + alc@FreeBSD.org + + + Mark Johnston + markj@FreeBSD.org + + + + +

An ongoing project aims to teach the swap pager to send + SCSI UNMAP or + ATA TRIM commands to the swap device when a block of swap + space has been + freed, for example when the application owning that block + is exiting.

+ +

SSDs have become commonplace and feature low latency for + random I/O + requests. This makes them appealing for use as swap + devices, since + lower latencies mean that applications spend less time + blocked while + waiting for a page-in from the swap device. To maximize + write + performance, some SSDs require the operating system to + send a + notification to the disk when a sector is no longer in + use; this helps + the disk optimize their usage of NAND flash cells. In + FreeBSD such a + notification is called a BIO_DELETE.

+ +

FreeBSD's UFS and ZFS filesystems have for a long time + been able to + transmit BIO_DELETE requests to the devices backing the + filesystem. For + example, for UFS this support is enabled by specifying -t + in newfs(8) or + tunefs(8)'s parameters. However, FreeBSD has historically + not had a + corresponding implementation for swap devices.

+ +

Thanks to Doug Moore, as of r349286 in -CURRENT and + r349930 in stable/12 + swapon(8) can send BIO_DELETE to all blocks on the + specified device + immediately prior to configuring it as a swap device. This + is enabled + by specifying -E in the swapon(8) parameters, or by adding + the + "trimonce" option to the swap device's /etc/fstab entry. + Some + in-progress work on the swap pager implements online block + deletion, in + which BIO_DELETE is transmitted for blocks as they are + freed by + applications; this will hopefully be implemented in + FreeBSD 13.0.

+ + + + + + + Fuzzing FreeBSD with syzkaller + + + + Mark Johnston + markj@FreeBSD.org + + + Andrew Turner + andrew@FreeBSD.org + + + Michael Tuexen + tuexen@FreeBSD.org + + + Ed Maste + emaste@FreeBSD.org + + + + + syzkaller + + + +

See the syzkaller entry in the 2019q1 quarterly report for + an + introduction to syzkaller.

+ +

syzkaller continues to find FreeBSD kernel bugs. A number + of + such bugs have been fixed in the past quarter, and we + continue + to investigate and fix bug reports from syzkaller. Work to + extend syzkaller's capabilites has progressed: Andrew + Turner + has implemented support for fuzzing the 32-bit + compatibility + layer in amd64 kernels, helping illuminate some of the + darker + corners of the kernel, and it is now possible to use bhyve + as + a VM backend to syzkaller, so it is now efficient and + convenient + to fuzz FreeBSD on FreeBSD.

+ +

Some planned work includes: enabling the use of ZFS as the + base filesystem for fuzzer VMs; extending the range of + system + calls and ioctls covered by syzkaller; enabling LLVM + sanitizers + in the kernel so as to catch more issues; and making use + of + netdump(4) to capture kernel dumps for panics found by + syzkaller, + making it much easier to diagnose bugs for which syzkaller + was + unable to find a reproducible test case.

+ + + + + The FreeBSD Foundation + + + + + + Locking changes for vnodes during execve(2) + + + + Konstantin Belousov + kib@freebsd.org + + + + +

The execve(2) family of syscalls replaces the executing + image in the + current process. The file containing the program text, + data, and + arbitrary other pre-initialized segments for the newly + activated image + is usually called the text file. FreeBSD marks the text + file as such, + the mark is mutually exclusive with any opening of the + file for write. + In other words, file opened for write cannot be executed, + and text + file cannot be opened for write.

+ +

During the execve(2) syscall processing, kernel needs to + lock the text + file' vnode. This is done both to satisfy the VFS calls + protocol, and + to ensure that there is no incompatible parallel changes + occuring to + the text vnode. A vnode can be locked either in exclusive + mode, which + is mutually incompatible with any other lock acquisition, + or in shared + mode, which is only incompatible with exclusive requests, + but allows + other shared owners.

+ +

In principle, there is no reason why would execve(2) need + an exclusive + vnode lock, since it does not modify neither content nor + metadata for + the text vnode. The only exception is the marking of the + vnode as + text, which was done using VV_TEXT flag in v_vflag and + protected by + the vnode lock. Since we modify v_vflag, the vnode lock + protecting + the modification should be taken exclusive.

+ +

The end result is that execve(2)'s of the same file are + serialized. For + instance, if user runs parallel build, which executes more + than one + job for compiling, all invocation of the compiler are + serialized + during execve(2).

+ +

The count of opens for write is contained in other struct + vnode member + named v_writecount, which was protected by the vnode lock + as well. + Since text is mutually exclusive with an open for write, I + reused + v_writecount to indicate text references. Now, negative + v_writecount + counts the number of text references. The v_writecount + content is + literally protected by the vnode interlock, but normally + all mutators + also own vnode lock at least in the shared mode.

+ +

This way, we no longer need to acquire exclusive text + vnode lock + during execve(2), removing the serializing point. + Additional positive + effect is that we started to account the precise number of + text + references on the vnode. Before, we cleared VV_TEXT on the + last unmap + of the text vnode, potentially allowing obscure DoS where + mapping the + text file while it is executed prevented writes until the + mapping is + destroyed. Now we mark the mappings for text explicitly in + the + vm_map_entry and dereference v_writecount by +1 when such + entry is + unmapped.

+ + + + + The FreeBSD Foundation + + + + + + Broadcom ARM64 SoC support + + + + Michal Stanek + mst@semihalf.com + + + Kornel Duleba + mindal@semihalf.com + + + Marcin Wojtas + mw@semihalf.com + + + + +

The Semihalf team continued working on FreeBSD support for + the + Broadcom + BCM5871X SoC series

+ +

BCM5871X are quad-core 64-bit ARMv8 Cortex-A57 + communication + processors targeted for networking applications such as + 10G routers, + gateways, control plane processing and NAS.

+ +

Completed since the last update:

+ + + +

+ In progress:

+ + + +

+ Todo:

+ + + + + + + Juniper Networks, Inc + + + + + + NXP ARM64 SoC support + + + + Marcin Wojtas + mw@semihalf.com + + + Artur Rojek + ar@semihalf.com + + + + +

The Semihalf team initiated working on FreeBSD support for + the + NXP + LS1046A SoC

+ +

LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors + with + integrated packet processing acceleration and high speed + peripherals + including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 + for a wide + range of networking, storage, security and industrial + applications.

+ +

Already completed:

+ + + +

+ In progress:

+ + + +

+ Todo:

+ + + + + + + Alstom Group + + + + + + Aberdeen Hackathon + + +

At BSDCam in Cambridge last year we had a discussion to + create a template + Hackathon in the same way we have a template for + Devsummits. To test out the + idea I was convinced (I swear tricked is the correct word) + to host a Hackathon + in Aberdeen.

+ +

As a project I think we benefit a lot from hackathons, but + they do take a + little organisation. The worst part of this is dealing + with getting money from + attendees so you can pay for events. I spoke with Deb + Goodkin from the + foundation at BSDCam and we arranged to use their new + EventBrite based system + to handle ticketing.

+ +

Overall this system made it straight forward for attendees + to register and get + me their details and requirements. After the event the + expenses were then + recouped from the foundation. This was much easier than me + putting together a + custom system or even setting up and using EventBrite + myself.

+ +

The hackathon went well, you can read in Benedict and + Kristof's reports that + follow, but it was less well attended than I originally + expected.

+ +

For hackers planning future hackathons remember to take + heed of common national + holidays (we could have planned the event to not land at + Easter) and expect + major geopolitical events to make things unpredictable (we + knew Brexit would do + something, but not when).

+ +

I need to thank the University of Aberdeen for providing + the location for the + Hackathon and to encourage you to run a hackathon where + you are. The next one + should be in your home town.

+ +

Benedict Reuschling

+ +

The hackathon in Aberdeen was happening in the week of + Easter at the University + of Aberdeen. Although only Kristof Provost (kp@) and + myself joined our host Tom + Jones, I still consider it a productive week for us. The + overall theme of the + hackathon was networking and each of us provided something + towards that goal + (be it PRs, submitting unfinished work, or other bits and + pieces). We got + together the night of Tuesday, April 16 over dinner and + talked about what our + plans were for the week.

+ +

Kristof and I had talked at AsiaBSDcon when I took his + tutorial about Testing + in FreeBSD that we should add a chapter about it in the + developers handbook. We + also used our first meeting to synchronize each other + about the latest news in + FreeBSD from our developers viewpoint.

+ +

The next day, we met up at the Frazer Noble building where + the hackathon was + taking place. It was one of the newer buildings on campus, + nicely integrated + into the older houses of the city. Since we were only a + handful, we sat in + Tom's office for the hackathon, which had plenty of room. + He also showed us the + room where we are supposed to be having the hackathon if + we were more people + and Tom gave us a little tour. Working in a university + myself, I'm always + interested in how other education organizations are + structured and the rooms + and equipment they provide for learning. Overall, my + impression was that there + is a good amount of space and equipment available, which + we could have used in + the hackathon.

+ +

After returning, we decided to use a special tag in the + commits we would be + doing to identify them as coming from this hackathon. We + chose "Event:" for it + as it is a general enough term to be used at other events + like conferences, + too. The "Sponsored by:" line we used in the past is more + for companies or + individuals sponsoring certain features, so I created a + review to add this line + to the committers guide.

+ +

Kristof had a couple of changes to the pf chapter in the + FreeBSD handbook for + me, so I started going through those. I created a review + for him and the commit + was made there and then, making use of the short feedback + cycle. Originally, we + thought about bringing in people via hangouts, but then + resolved to contact + people via our usual IRC channel if we needed their input.

+ +

Kristof and Tom worked on some network specific stuff, + whereas I started work + on creating an initial draft for the testing chapter. We + would occasionally + start talking about something and then return to our work + in silence. If we + needed to coordinate or had questions, we simply asked and + could continue once + we got our answer. This provided a nice atmosphere to work + in. I tackled some + doc PRs while Kristof found a bug in pf and fixed it.

+ +

The afternoons were spent at different locations within + walking distance. Tom + made sure we got a good impression on how it is to be a + student and that there + is both taste and variety of food available. In the + evenings, Tom drove us into + town to have dinner at various restaurants over the week.

+ +

Aberdeen has a lot to offer as a city. Starting from the + second day, Kristof + and I would meet up at my hotel, which was close to the + Aberdeen beach and walk + along it to the University. According to Tom, it is + possible to see Dolphins + when the weather is right and the gulf stream provides the + city with enough + warmth that the winters aren't as bad as you'd think this + far up north.

+ +

Tom also gave us a tour of the zoological department of + the university, which + offered a beautiful garden with various plants and trees, + as well as a museum + with zoological specimen. This offered a great spot for + photographs and to + unwind a bit from the technical discussions we've had. Tom + also had t-shirts + made for the event, which are already rare collectors + items.

+ +

I had to return on Sunday, so Tom took us on a tour of the + Scottish highlands + in his car the day before. We stopped at a couple of + places to take pictures + and Tom would explain at lot to us having lived there all + his life. We came to + Stonehaven and had fish and chips there from a take-out + restaurant that had a + lot of awards for sustainable fishing. This was certainly + a highlight for the + week and even then, we couldn't stop talking about FreeBSD + and networking.

+ +

Although more people would maybe have produced more + output, the three of us + were certainly productive as a small group. It also made + planning and + coordination easier and more flexible. Tom Jones had done + a lot of preparation + and was an excellent guide. I would encourage him to host + another such + hackathon in the future and hope that next time, more + people will take a trip + to Aberdeen to spend some time hacking on FreeBSD

+ +

Kristof Provost

+ +

While I’d been to Scotland before I’d never seen Aberdeen. + It’s a charming + city, and I enthusiastically recommend visiting.

+ +

I arrived a little while after Benedict, but made it to my + hotel easily, and + turned up in time to join Benedict and Tom for dinner.

+ +

Despite being small (or perhaps because of it), the + hackathon was remarkably + productive. Benedict and I went through the pf + documentation in the handbook, + so that Benedict could rework and improve it. (Benedict’s + doing the work, but + I’m going to take credit anyway.)

+ +

Tom and I looked at the GSoC proposals and tried to find + potential mentors for + two promising proposals. Both of us are candidate mentors + as well. We should + know soon if our students are awarded slots.

+ +

Tom also proposed a patch to eliminate RFC 2675 IPv6 + Jumbograms. It has my + enthusiastic support.

+ +

I managed to look at a couple of open pf issues:

+ + + +

+ On Saturday Tom took us out to discover some of the pretty + bits of Scotland. It + turns out there are a lot of them. I can’t really do it + justice, but Tom has a + promising career at the Scottish tourism board when this + computers fad blows + over.

+ +

On my way home I passed through Oslo, and took the + opportunity to meet with + (have lunch with) two of the EuroBSDCon local organisers. + EuroBSDCon is filling + up fast, make sure to register now to secure your place!

+ + + + + + + Bring more Security Intelligence to FreeBSD + + + + Michael Muenz + m.muenz@gmail.com + + + + + Maltrail - distributed Malware detection + Wazuh - thread detection and incident response + + + +

To bring more Security Intelligence we maintain the + FreeBSD port + of zmaltail. This open source project based on + Python can act + as a sensor and/or as a central server. It listens in + defined ports + or protocols and compares IP addresses and domains against + static + and dynamic feeds, contributed by the community.

+ +

As you can install this piece of software on multiple + firewalls + and let them send to a central server, you are able to + detect attacks + and compromises very fast. Within Q2 we updated the port + to the latest + version and are constantly in contact with the core + developer + (also co-author of SQLmap) to bring out new + features.

+ +

The second project we are currently trying to add as a + port is Wazuh. + Wazuh is a fork of Ossec which is already in the + ports tree. + Compared to Ossec, Wazuh has some intelligent addition + like full + ELK-Stack integration with own apps and + dashboards.

+ +

With Wazuh installed on your webserver, or even on your + windows desktop + you can monitor file integrity or log files for most kind + of attacks. + Active response features let you e.g. send API calls to + your firewalls + to dynamically block this offender.

+ +

As Wazuh offers a complete ELK-Stack you can use it also + as a central + logging solution for better security insights into your + network.

+ + + + + m.a.x. Informationstechnologie AG + + + + + + nsysctl 1.0 + + + + Alfonso Sabato Siciliano + alfonso.siciliano@email.com + + + + + gitlab.com/alfix/nsysctl + sysutils/nsysctl port + Tutorial + + + +

The nsysctl utility is a /sbin/sysctl + clone, to get or set the kernel + state, supporting libxo and extra options.

+ +

+ nsysctl [--libxo=opts [-r tagname]]
+ [-DdFGgIilmNpqTt[V|v[h[b|o|x]]]Wy]
+ [-e sep] [-B <bufsize>] [-f filename]
+ name[=value[,value]] ...
+ nsysctl [--libxo=opts [-r tagname]]
+ [-DdFGgIlmNpqTt[V|v[h[b|o|x]]]Wy]
+ [-e sep] [-B <bufsize>] -A|a|X
+

+ +

You could use nsysctl to explore the sysctl MIB + showing the value and the + info of an object. The output is explicitly indicated by + the options and is + printed via libxo in human and machine readable + formats, moreover some value + is parsed to display it in a structured mode (e.g., + vm.phys_free). + The support for efi_map_header was added but it + is untested, someone could + help by trying it via machdep.efi_map.

+ +

Please refer to the tutorial for a more thorough + description.

+ + + + + + + libvdsk - QCOW2 implementation + + + + Sergiu Weisz + sergiu121@gmail.com + + + Marcel Molenaar + marcel@freebsd.org + + + Marcelo Araujo + araujo@freebsd.org + + + Mihai Carabas + mihai@freebsd.org + + + + + Github - libvdsk repo + + + +

Add support for using QCOW in bhyve using the libvdsk + library. Libvdsk was used + to substitute the regular disk operations from bhyve with + a call to libvdsk + which will in turn call the disk-specific handler for the + operation.

+ +

To use this feature one has to install the libvdsk-enabled + bhyve version along + with libvdsk from the libvdsk repo linked above.

+ +

New features added:

+ + + +

+ Future tasks:

+ + + + + Matthew Grooms + + + + + + +