The netperf cluster provides a multi-node, SMP-capable, network functionality and performance test capability for the FreeBSD Project, supporting a variety of on-going sub-projects including the netperf project, Dingo project, and on-going work on high performance threading. The cluster is available on a check out basis for developers, who must request accounts be created by contacting one of the netperf cluster admins.
The netperf cluster was made possible through the generous donation of a number of organizations, including:
Sentex Data Communications, who not only host the complete cluster, provide front-end build system, and the management infrastructure (remote power, serial console, network switch, etc), but also appear to be endlessly willing to help configure, reconfigure, and troubleshoot at almost any time of day or night.
FreeBSD Systems, who through a generous matching grant with the FreeBSD Foundation, provide the majority of testing hardware used in the cluster, including three dual-Xeon test systems.
The FreeBSD Foundation, who provided a matching grant for the purposes of purchasing testing hardware, as well as taking ownership of hardware, offering tax receipts to donors in its role as a non-profit, and participating in cluster planning.
IronPort Systems, who have generously donated additional test hardware for use in the netperf cluster.
Donations to support the netperf cluster have an immediate and substantial impact on the success of a number of on-going performance projects, providing access to high-end hardware to a large number of developers. If you or your company are interested in helping to support continued development of the netperf cluster as a resource for FreeBSD development, please contact the netperf cluster admins.
The FreeBSD netperf cluster is managed by a small team of developer/administrators to support SMP development and performance testing on high-end hardware. If you have any questions, including questions about access to the cluster as a developer, or about possible future donations of testing hardware, please feel free to contact the following:
&a.rwatson;
&a.bmilekic;
The Netperf cluster consists of several systems interconnected using a management network, as well as individual back-to-back gigabit ethernet links for a test network. The following systems are available as testing resources on a check-out basis:
zoo.FreeBSD.org is the front-end build and management system. All netperf cluster users are provided with accounts on this box, typically using the same account name and SSH keys as used to access the FreeBSD.org central cluster. Connected to zoo's second network interface is a gigabit link to the internal management network. Zoo provides DHCP, tftp, and NFS services to boxes in the cluster, as well as having serial access to the remote power system and serial console access to the test boxes. Test kernels and software will typically be built and configured on zoo, then exported to the cluster via NFS. Zoo exports its /zoo file system to the cluster, and cluster users will have a directory, /zoo/username, in which they can place any files to export. Each machine has a /zoo/hostname directory, which consists of the root of an NFS root file system, as well as the tree from which tftp exports pxeboot loaders. By substituting kernels and configuration files in these trees, most aspects of the test systems may be directly managed.
elephant is a dual-PIII 800MHz system with ATA disk subsystem. Currently, the ATA disk holds two partitions, one with FreeBSD 4.x, and one with FreeBSD 5.x user space configuration on.
orangutan is a dual-Xeon 2GHz system equipped with an Adaptec SCSI RAID array. Currently, the RAID array is configured to expose a single volume holding FreeBSD 6.x.
tiger-1, tiger-2, and tiger-3 are a set of interconnected, matching dual-Xeon 3GHz systems with ATA disk subsystems. Each has four if_em network interfaces, and these are interconnected so that various topologies can be created. Two ATA disks are connected, one with a FreeBSD 4.x and one with a FreeBSD 5.x user space configuration on.
cheetah is a quad-processor amd64 system.
cheetah Is a dual core Opteron system with two + 2GHz CPUs each with two cores. The machine identifies as a quad + processor machine in dmesg. The system has ATA disk, 2GB of + RAM, 1GB for each processor, and 5 ethernet ports. fxp0 is the + management port and em0, em1, bge0 and bge1 are gigE interfaces + which will eventually connect cheetah to elephant and orangutan. + We are currently waiting on a switch to be installed
apc2 is the remote power console for the test network.
The current serial port and network configuration of test systems, as well as password information, can be found in /etc/motd on zoo. We are currently interested in adding amd64 and em64t hardware to the cluster.
As the netperf cluster is a centrally managed and shared resource, understanding and consistent following of its procedures is important. In particular, following of the procedures makes it easier for developers to have reasonable expectations about the configuration of systems in the cluster, as well as to avoid treading on each others toes.
Reserving a system is currently done on an ad hoc basis. A wiki used to system reservation will shortly be introduced. In general, it is recommended that reservations of systems occur for periods of 8 hours or shorter, in order to facilitate shared use, although longer tests or experiments are certainly possible.
Power cycling a system is currently performed by connecting to apc2, the remote power system, using a serial port on zoo. Because there is a single serial connection to the power system, it is requested that developers minimize the amount of time spent connected to it, so that parallel use of the system can occur more readily. In particular, please do not leave a cu/tip session to the power system open for extended periods of time.
Booting a specific configuration is currently performed by interrupting the boot of a system using the serial console, or modifying the /zoo/hostname/boot/loader.conf file to point at a new kernel. Please do not replace the kernel in the NFS boot tree for a host; instead, install kernels in your /zoo/username tree, and explicitly configure hosts to use that kernel. Please restore the defaults for a system, including removing any non-standard entries from loader.conf, when you are done with a test or experiment.
Reconfiguring on-disk installs should be done only in coordination with the cluster admins. If you change configuration settings for hosts during an experiment, please restore them. If you install packages on the file systems of tests, please remove them when done. A useful technique is to store required packages for experiments in NFS, and to add the packages for the duration of a development session. While data can be left on systems between test runs, developers who are using data sets for common applications, such as MySQL, may wish to store that data in a personal directory as other testing on the host could interfere with it. Developers are advised to check the configuration of systems before using them to make sure all is as expected.
Reconfiguring system BIOSes should be avoided if at all possible. Please restore any settings as soon as a test or experiment is complete. Please do not turn off BIOS serial console redirection, or modify any settings associated with console redirection!
Check software configurations before testing -- specifically, use uname to confirm that the right kernel is running, that it's not configured for WITNESS or INVARIANTS if that's not needed, and that /etc/malloc.conf is set to "aj" if no debugging is desired. Use 'ps' and 'pkg_info' to make sure only the software you expect is running, and that the version of the software you expect is installed.
A few hopefully up-to-date configuration notes that may be relevant to users of the netperf cluster:
20050624 - cheetah is now online!
20050204 - orangutan is now configured to use PXEboot, thanks to help from Sentex.
20050203 - system upgrades to tiger-1, tiger-2, and tiger-3 have been completed -- the latest versions of 4.x (ar0s1) and 6.x (ar0s2) are now installed.
20050203 - zoo.FreeBSD.org has been updated to the most recent version of 5-STABLE.
Currently, no automated reservation system is in place for the netperf cluster, but we hope to introduce some system for reservation shortly. In the meantime, the following standing reservations are in place for the netperf cluster:
| System | Developer(s) | Project |
|---|---|---|
| elephant | &a.gnn; | KAME IPv6/IPSEC locking |
| orangutan | &a.davidxu; | libthread development, testing, and performance measurement |