Introduction

Index: head/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml =================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml (revision 42672) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml (revision 42673) @@ -1,247 +1,324 @@ July-September 2013

Introduction

This report covers &os;-related projects between July and September 2013. This is the third of four reports planned for 2013.

Thanks to all the reporters for the excellent work! This report - contains 5 entries and we hope you enjoy reading it.

+ contains 6 entries and we hope you enjoy reading it.

The deadline for submissions covering between October and December 2013 is not yet decided.

proj Projects kern Kernel + + ports + + Ports + + AES-NI Improvements for GELI John-Mark Gurney jmg@FreeBSD.org

An enhancement to the AES-NI implementation for OpenCrypto has been committed that significantly improves AES-XTS and AES-CBC decryption performance. This gives geli(8) around a three times performance boost on gnop(8) using AES-XTS compared to the old code.

These improvements are available to users of the OpenCrypto framework and crypto(4).

Static Code Analysis Ulrich Spoerlein uqs@FreeBSD.org Coverity Scan Clang Static Analyzer Scan for &os; Clang Static Analyzer Home Page

With our own (old and unstable) instance of Coverity Prevent gone, we have now fully transitioned to the Scan project run by Coverity (see links), which Open Source projects can use to learn about possible defects in their source code.

We also continue to run our code base through the Static Analyzer that is shipped with Clang/LLVM. It cannot track the state of the code over time, but has the benefit that everyone can use it without any special setup. See the home page at the links section for more information on the Clang Static Analyzer project in general, and head over to the &os; Clang Static Analyzer Scan page (see links) to see those possible defects (no signup required).

We are looking for a co-admin for both of these projects to increase the bus-factor and the chance of survival for these services. Fame and fortune await!

Maybe turn on email reports for new defects to the internal list of &os; developers. Find co-admin. Fix the defects reported by Coverity and Clang. GEOM Direct Dispatch and Fine-Grained CAM Locking Alexander Motin mav@FreeBSD.org Project SVN branch Project patches

Last year's high-performance storage vendors reported a performance bottleneck in the &os; block storage subsystem, limiting peak performance around 300-500K IOPS. While that is still more than enough for average systems, detailed investigation has shown a number of places that require radical improvement. Unmapped I/O support implemented early this year already improved I/O performance by about 30% and moved more accents toward GEOM and CAM subsystems scalability. Fixing these issues was the goal of this project.

The existing GEOM design assumed most I/O handling to be done by only two kernel threads (g_up() and g_down()). That simplified locking in some cases, but limited potential SMP scalability and created additional scheduler overhead. This project introduces the concept of direct I/O dispatch into GEOM for cases where it is known to be safe and efficient. That implies marking some GEOM consumers and providers with one or two new flags, declaring situations when a direct function call can be used instead of normal request queuing. That allows avoiding any context switches inside GEOM for the most widely used topologies, simultaneously processing multiple I/Os from multiple calling threads.

Having GEOM passing through multiple concurrent calls down to the underlying layers exposed major lock congestion in CAM. In the existing CAM design all devices connected to the same ATA/SCSI controller are sharing a single lock, which can be quite busy due to multiple controller hardware accesses and/or code logic. Experiments have shown that applying only the above GEOM direct dispatch changes burns up to 60% of system CPU time or even more in attempts to obtain these locks by multiple callers, killing any benefits of GEOM direct dispatch. To overcome that, new fine-grained CAM locking design was implemented. It implies splitting big per-SIM locks into several smaller ones: per-LUN locks, per-bus locks, queue locks, etc. After these changes, the remaining per-SIM lock protects only the controller driver internals, reducing lock congestion down to an acceptable level and keeping keep compatibility with existing drivers.

Together, GEOM and CAM changes double the peak I/O rate, reaching up to 1,000,000 IOPS on contemporary hardware.

The changes were tested by a number of people and will be committed into &os; head and merged to stable/10 after the end of the &os; 10.0 release cycle.

The project is sponsored by iXsystems, Inc.

More reviews, more stability and performance tests. VMware VMXNET3 Driver Bryan Venteicher bryanv@freebsd.org

A port of the OpenBSD vmx(4) ethernet driver has been committed. The driver can be used in place of the VMware Tools vmxnet3 driver, which currently does not support 10.0-RELEASE (or anything past 9.0-RELEASE).

Performance improvements, multiqueue support. Merge to stable/9. VirtIO Network Multiqueue Bryan Venteicher bryanv@freebsd.org

The VirtIO network driver, vtnet(4), recently gained support for multiple queues, along with a significant cleanup and support for a few additional features.

+ + + + &os; Python Ports + + + + + &os; + Python Team + + python@FreeBSD.org + + + + + The &os; Python Team Page + IRC channel + + + +

We are currently working on cleaning up the + lang/python* ports to improve the compatibility with + the original upstream build behaviour and to lower the need for + &os;-specific build patches. A first step was made in September + by reducing the flags injected into the different Python + interpreter versions.

+ +

The first tasks for supporting the installation of packages for + different Python ports have been done. A new metaport structure + has replaced the original Python port behaviour and will be + enhanced over the next months to enable improved installation + support of packages for different Python versions at the same + time.

+ +

Plans are made to enhance the Python ports framework by + automating the insertion of byte-compiled files into + pkg-plist files. Besides the reduction of packaging + list sizes, this is also required to offer packages for multiple + Python versions.

+ +

PyPy was heavily enhanced over the last couple of months. + Major updates of the port solved integration issues and a new + pypy-devel port for snapshots and previews was added. + Since the PyPy 3 release, there is a new + pypy3-devel port available to provide not only + compatibility for Python 2.x specific scripts, but also for + those using the 3.x language specification.

+ +

IronPython found its way into the &os; ports tree, which + provides an implementation of the Python language based on .NET + and Mono.

+ + + + Develop a high-level and lightweight Python Ports + Policy. + + Chase the unification of Distribute + (devel/py-distribute) and Setuptools + (devel/py-setuptools*). + + Add support for granular dependencies (for example + >=1.0 or < 2.0). + + Look at what adding pip (Python Package Index) + support looks like. + + More tasks can be found on the Team's wiki page (see + links). +