diff --git a/en_US.ISO8859-1/articles/releng-packages/Makefile b/en_US.ISO8859-1/articles/releng-packages/Makefile new file mode 100644 index 0000000000..00cedcd8e7 --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/Makefile @@ -0,0 +1,20 @@ +# +# $FreeBSD$ +# +# Article: FreeBSD Release Engineering of Third Party Software Packages + +DOC?= article + +FORMATS?= html + +INSTALL_COMPRESSED?= gz +INSTALL_ONLY_COMPRESSED?= + +# SGML content +SRCS= article.sgml + +CSS_SHEET_ADDITIONS= extra.css + +DOC_PREFIX?= ${.CURDIR}/../../.. + +.include "${DOC_PREFIX}/share/mk/doc.project.mk" diff --git a/en_US.ISO8859-1/articles/releng-packages/article.sgml b/en_US.ISO8859-1/articles/releng-packages/article.sgml new file mode 100644 index 0000000000..ef4636c263 --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/article.sgml @@ -0,0 +1,368 @@ + + + +%man; + +%freebsd; + +%authors; +]> + +
+ FreeBSD Release Engineering for Third Party Software + Packages + + + + Steve + Price + +
steve@FreeBSD.org
+ + + + + $FreeBSD$ + + + This paper describes the approach used by the FreeBSD + release engineering team to produce a high quality package set + suitable for official FreeBSD release media. This document is + a work in progress, but eventually it will cover the process + used to build a clean package set on the FreeBSD.org "Ports + Cluster", how to configure any other set of machines as a + ports cluster, how to split up the packages for the release + media, and how to verify that a package set is + consistent. + + + + + + + Building packages from the Ports Collection + + The FreeBSD Ports + collection is a collection of over &os.numports; + third-party software packages available for FreeBSD. The ports + team (portmgr@FreeBSD.org) is responsible for maintaining a + consistent ports tree that can be used to create the binary + packages that accompany a given FreeBSD release. + + + The Ports Cluster + + In order to provide a consistent set of third-party + packages for FreeBSD releases, every port is built in a + separate chroot environment, starting with an empty + /usr/local and + /usr/X11R6. The requisite dependencies + are installed as packages before the build proceeds. This + enforces consistency in the package build + process. By starting the package build in a pristine + environment, we can assure that the package metadata (such as + required dependencies) is accurate, and so we will never + generate packages that might work on some systems and not on + others depending on what software was previously + installed. + + The Ports Cluster for the x86 architecture + currently consists of a master node (Dual Pentium III 733Mhz) + and 8 slave nodes (Pentium III 800Mhz) to do the actual + package builds. With this configuration, a complete package + build takes over 24 hours. These machines are co-located with + the other FreeBSD Project equipment at Yahoo's corner of + Exodus in Santa Clara, CA. + + The Ports Cluster for the Alpha + architecture consists of 7 PWS 500A machines donated by Compaq + and also co-located with Yahoo's facilities. + + + + + The Package Split + + For FreeBSD 4.4 over 4.1 gigabytes of packages were created. + This causes a problem for CDROM distributions because we would + like to ship as many packages as possible without making the + user insert another disc to satisfy dependencies. The solution + is to create clusters of like packages with + similar dependencies and group these onto specific discs. This + section describes the software and methodology used to create + those package sets for the official FreeBSD release + discs. + + First off you'll need to get a copy of the tarball from the + following URL : + + + + Copy thisc archive to a machine that has enough free HD + space to hold 2 to 3 times the size of the package set that you + wish to split. The tarball will extract into the current + working directory so make sure you've created one suitably named + directory for the release you are working on. + + After you've extracted the files, you will notice the + following files : + + + + config + + This file contains the free space on each disc + and whether packages, distfiles, or both are allowed on any + given disc. The first column is the disc name. It must be + of the form disc[0-9a-z]. Currently it is setup to allow + for 10 discs (4 for the release set and 6 for the toolkit). + There's an implied extra disc called scratch where all of + the remaining distfiles/packages land if they don't fit + elsewhere. The second column can be either a 1 or 0 where 1 + says that it is okay to place packages on this disc. The + third column works the same way except that it controls + whether distfiles are placed on this disc. The last column + denotes the number of bytes of free space on a + disc. + + + + doit.sh + + This is the workhorse. Once you have all the + files in place and things properly configured this script + directs the process of splitting packages. Beware it is + interactive so you need to keep an eye on it as it runs. + More details on what happens in this script will + follow. + + + + scripts/checkdeps.pl + + Makes sure all packages dependencies are + satisfied given an INDEX file and a directory of + packages. + + + + scripts/oneshot.pl + + This is where all the magic (and I use that + term loosely as it is mostly just a brute force approach) + happens. Given a list of required packages for each disc + and a set of packages/distfiles this is the script that + places a package or distfile on a disc along with all of its + dependencies. + + + + scripts/print-cdrom-packages.sh + + This file is a copy of + src/release/scripts/print-cdrom-packages.sh + from the release you are working on. + + + + scripts/scrubindex.pl + + This script removes lines from an INDEX file + for packages that aren't present. It also removes the + XFree86 dependencies. NOTE: you'll need to tweak the value + of the xdep variable to make sure the version number is + correct. + + + + scripts/setup.sh + + This is a helper script that I use on the + bento cluster to grab a copy of the ports tree and the + matching set of the packages/distfiles. + + + + Here's a checklist of things you'll need to check or + configure before going any further. + + + Edit config to denote the + number of discs you have, their sizes, and whether you want + them want to contain packages, distfiles, both, or + neither. + + Make sure you remove the gen directory if + there's an old one laying around. This directory contains + working files that will only be valid for the current + split. + + On your first pass through a split it is best to + fake the copying of packages and distfiles. This will save + both time and diskspace while you do a couple of trial runs to + make sure things fit, etc. In the + scripts/oneshot.pl set the fake variable + to 1 and instead of actually copying the files it will + &man.touch.1; them. Be sure you turn this off or set fake to + 0 before you give the resultant discs to the person that will + be mastering the discs otherwise they'll get a directory full + of zero-sized files. + + Make sure you have a recent copy of the + print-cdrom-packages.sh and that it is + from the correct release. + + Check to make sure the XFree86 dependency in + scripts/scrubindex.pl has the correct + version number. You'll also need to make sure this value is + correct in doit.sh as + well. + + + Next you'll need to get a copy of the ports tree, packages, + and distfiles from a recent build on the package cluster. See + the scripts/setup.sh for a working example + but essentially here's what needs to be done. + + + + Grab a copy of ports.tar.gz + and extract it into the ports directory alongside + doit.sh and the + scripts directory. + + Remove the packages/distfiles directories or + symlinks. Bento has these as symlinks and you'll have mixed + results if you don't get rid of them before + proceeding. + + Create a new ports/packages directory and copy + the package set from the package building + cluster. + + Create a new ports/distfiles directory and copy + the distfiles from the package building cluster. NOTE: if you + don't want any distfiles simply create the directory and leave + it empty. This directory must be present even if it doesn't + contain anything. + + + Now we're finally ready for the fun task of actually + splitting the packages. You start the processing by running + ./doit.sh. The first time you run it here's + what it does. + + + Create a list of the restricted (can't be on the + master FTP site) ports. + + Asks you if you'd like to remove the restricted + ports. Most of the time you'll want to answer (y)es + here. + + Create a list of the packages/distfiles that + can't be put on the discs. + + Asks you if you'd like to remove the + non-cdromable packages/distfiles. Most of the time you'll + want to answer (y)es here. + + Copies the INDEX from the ports directory to the + gen directory. In doing so it removes the lines for ports + where the packages don't exist. It also checks to make sure + that all of the required dependency packages are + present. + + Create a list of packages that are required on + each disc. + + Asks you if you'd like to populate the discs. + After populating each disc it will check for missing + dependencies, scrub the INDEX file, and create the + CHECKSUM.MD5 file. + + Check to make sure the required packages made it + on each disc and gives you a summary of the sizes of each + disc. + + + After going through this the first time if you are lucky + enough that all of the required packages built and fit on each + disc. All you need to do is set fake to 0 in + scripts/oneshot.pl and re-run + ./doit.sh. The second and subsequent times + around it will skip steps 1-5 above. If you want to re-run any + of those steps refer to doit.sh for which files need to be + removed to not short-circuit those steps. If you want to repeat + all of these steps then the easiest way is to rm -rf + gen. + + Upon successful completion the packages/distfiles will be in + the disc* directories and the leftover will + be in the scratch directory. + + What to do if things go wrong? Here's some common gotchas + and workarounds. + + + + Missing required packages + + This is a pretty common occurrence. You'll + either need to wait for a new set of packages where the + missing packages were built or get someone to re-start the + package build for you. DO NOT attempt to build the missing + packages on your own machine and add them into the fray. + While you might be able to get away with this if you are + extremely careful the vast majority of the time you'll miss + some little detail and the simple process of adding a + package could make hundreds of others come up mysteriously + broken. + + + + Required packages won't fit + + This happens on occasion too and is relatively + easy to fix. Simply edit print-cdrom-packages.sh to move + packages around until they fit. Yes this is an iterative + process and one of the reasons why you should enable fake in + scripts/oneshot.pl until you've gotten + things the way you want them. Re-run + ./doit.sh after you made your + adjustments. + + + + Required packages not on the right (or any) disc + + This usually means you didn't add them to + print-cdrom-packages.sh or you put them + on the wrong disc. This script is the gospel by which this + whole process determines where a package must be. If you + want to force a package to land on a particular disc this is + the only way to ensure that it will + happen. + + + + If you get completely stuck and can't figure out why things + are borked or how to fix them then email &a.steve; for + assistance. + + + +
diff --git a/en_US.ISO8859-1/articles/releng-packages/extra.css b/en_US.ISO8859-1/articles/releng-packages/extra.css new file mode 100644 index 0000000000..7cf2be1d7d --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/extra.css @@ -0,0 +1,16 @@ +/* + * Netscape 4 does not recognice the @import directive of CSS, so we + * can't add an additional stylesheet layer on top of the default one. + * Instead, we use this hack to copy this file to the end of + * docbook.css. + * + * $FreeBSD$ + */ + +/* @import "docbook.css"; */ + +/* Customization that looks good for this particular article. */ + +DIV.TITLEPAGE { + text-align: center; +}