diff --git a/documentation/content/en/articles/building-products/_index.adoc b/documentation/content/en/articles/building-products/_index.adoc index 8998e863f0..cf50b55302 100644 --- a/documentation/content/en/articles/building-products/_index.adoc +++ b/documentation/content/en/articles/building-products/_index.adoc @@ -1,376 +1,382 @@ --- title: Building Products with FreeBSD authors: - author: Joseph Koshy email: jkoshy@FreeBSD.org organizations: - organization: The FreeBSD Project description: How FreeBSD can help you build a better product trademarks: ["freebsd", "general"] tags: ["FreeBSD", "FreeBSD as base for your product"] --- = Building Products with FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: + +ifeval::["{backend}" == "html5"] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] - -ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/building-products/ endif::[] ifeval::["{backend}" == "pdf"] +include::../../../../shared/releases.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/articles/building-products/ endif::[] ifeval::["{backend}" == "epub3"] +include::../../../../shared/releases.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/articles/building-products/ endif::[] [.abstract-title] Abstract The FreeBSD project is a worldwide, volunteer based, and collaborative project, which develops a portable and high-quality operating system. The FreeBSD project distributes the source code for its product under a liberal license, with the intention of encouraging the use of its code. Collaborating with the FreeBSD project can help organizations reduce their time to market, reduce engineering costs and improve their product quality. This article examines the issues in using FreeBSD code in appliances and software products. It highlights the characteristics of FreeBSD that make it an excellent substrate for product development. The article concludes by suggesting a few "best practices" for organizations collaborating with the FreeBSD project. ''' toc::[] [[introduction]] == Introduction FreeBSD today is well-known as a high-performance server operating system. It is deployed on millions of web servers and internet-facing hosts worldwide. FreeBSD code also forms an integral part of many products, ranging from appliances such as network routers, firewalls, and storage devices, to personal computers. Portions of FreeBSD have also been used in commercial shrink-wrapped software (see <>). In this article we look at the link:https://www.FreeBSD.org/[FreeBSD project] as a software engineering resource-as a collection of building blocks and processes which you can use to build products. While FreeBSD's source is distributed freely to the public, to fully enjoy the benefits of the project's work, organizations need to _collaborate_ with the project. In subsequent sections of this article we discuss effective means of collaboration with the project and the pitfalls that need to be avoided while doing so. *Caveat Reader.* The author believes that the characteristics of the FreeBSD Project listed in this article were substantially true at the time the article was conceived and written (2005). However, the reader should keep in mind that the practices and processes used by open-source communities can change over time, and that the information in this article should therefore be taken as indicative rather than normative. === Target Audience This document would be of interest to the following broad groups of people: * Decision makers in product companies looking at ways to improve their product quality, reduce their time to market and lower engineering costs in the long term. * Technology consultants looking for best-practices in leveraging "open-source". * Industry observers interested in understanding the dynamics of open-source projects. * Software developers seeking to use FreeBSD and looking for ways to contribute back. === Article Goals After reading this article you should have: * An understanding of the goals of the FreeBSD Project and its organizational structure. * An understanding of its development model and release engineering processes. * An understanding of how conventional corporate software development processes differ from that used in the FreeBSD project. * Awareness of the communication channels used by the project and the level of transparency you can expect. * Awareness of optimal ways of working with the project-how best to reduce engineering costs, improve time to market, manage security vulnerabilities, and preserve future compatibility with your product as the FreeBSD project evolves. === Article Structure The rest of the article is structured as follows: * <> introduces the FreeBSD project, explores its organizational structure, key technologies and release engineering processes. * <> describes ways to collaborate with the FreeBSD project. It examines common pitfalls encountered by corporates working with voluntary projects like FreeBSD. * <> concludes. [[freebsd-intro]] == FreeBSD as a set of building blocks FreeBSD makes an excellent foundation on which to build products: * FreeBSD source code is distributed under a liberal BSD license facilitating its adoption in commercial products <> with minimum hassle. * The FreeBSD project has excellent engineering practices that can be leveraged. * The project offers exceptional transparency into its workings, allowing organizations using its code to plan effectively for the future. * The culture of the FreeBSD project, carried over from the Computer Science Research Group at The University of California, Berkeley <>, fosters high-quality work. Some features in FreeBSD define the state of the art. <> examines the business reasons for using open-source in greater detail. For organizations, the benefits of using FreeBSD components in their products include a shorter time to market, lower development costs and lower development risks. === Building with FreeBSD Here are a few ways organizations have used FreeBSD: * As an upstream source for tested code for libraries and utilities. + By being "downstream" of the project, organizations leverage the new features, bug fixes and testing that the upstream code receives. * As an embedded OS (for example, for an OEM router and firewall device). In this model, organizations use a customized FreeBSD kernel and application program set along with a proprietary management layer for their device. OEMs benefit from new hardware support being added by the FreeBSD project upstream, and from the testing that the base system receives. + FreeBSD ships with a self-hosting development environment that allows easy creation of such configurations. * As a Unix compatible environment for the management functions of high-end storage and networking devices, running on a separate processor "blade". + FreeBSD provides the tools for creating dedicated OS and application program images. Its implementation of a BSD unix API is mature and tested. FreeBSD can also provide a stable cross-development environment for the other components of the high-end device. * As a vehicle to get widespread testing and support from a worldwide team of developers for non-critical "intellectual property". + In this model, organizations contribute useful infrastructural frameworks to the FreeBSD project (for example, see man:netgraph[3]). The widespread exposure that the code gets helps to quickly identify performance issues and bugs. The involvement of top-notch developers also leads to useful extensions to the infrastructure that the contributing organization also benefits from. * As a development environment supporting cross-development for embedded OSes like http://www.rtems.com/[RTEMS] and http://ecos.sourceware.org/[eCOS]. + There are many full-fledged development environments in the {numports}-strong collection of applications ported and packaged with FreeBSD. * As a way to support a Unix-like API in an otherwise proprietary OS, increasing its palatability for application developers. + Here parts of FreeBSD's kernel and application programs are "ported" to run alongside other tasks in the proprietary OS. The availability of a stable and well tested Unix(TM) API implementation can reduce the effort needed to port popular applications to the proprietary OS. As FreeBSD ships with high-quality documentation for its internals and has effective vulnerability management and release engineering processes, the costs of keeping up-to-date are kept low. [[freebsd-technologies]] === Technologies There are a large number of technologies supported by the FreeBSD project. A selection of these are listed below: * A complete system that can cross-host itself for link:https://www.FreeBSD.org/platforms/[many architectures:] * A modular symmetric multiprocessing capable kernel, with loadable kernel modules and a flexible and easy to use configuration system. * Support for emulation of Linux(TM) and SVR4 binaries at near machine speeds. Support for binary Windows(TM) (NDIS) network drivers. * Libraries for many programming tasks: archivers, FTP and HTTP support, thread support, in addition to a full POSIX(TM) like programming environment. * Security features: Mandatory Access Control (man:mac[9]), jails (man:jail[2]), ACLs, and in-kernel cryptographic device support. * Networking features: firewall-ing, QoS management, high-performance TCP/IP networking with support for many extensions. + FreeBSD's in-kernel Netgraph (man:netgraph[4]) framework allows kernel networking modules to be connected together in flexible ways. * Support for storage technologies: Fibre Channel, SCSI, software and hardware RAID, ATA and SATA. + FreeBSD supports a number of filesystems, and its native UFS2 filesystem supports soft updates, snapshots and very large filesystem sizes (16TB per filesystem) <>. + FreeBSD's in-kernel GEOM (man:geom[4]) framework allows kernel storage modules to be composed in flexible ways. * Over {numports} ported applications, both commercial and open-source, managed via the FreeBSD ports collection. === Organizational Structure FreeBSD's organizational structure is non-hierarchical. There are essentially two kinds of contributors to FreeBSD, general users of FreeBSD, and developers with write access (known as _committers_ in the jargon) to the source base. There are many thousands of contributors in the first group; the vast majority of contributions to FreeBSD come from individuals in this group. Commit rights (write access) to the repository are granted to individuals who contribute consistently to the project. Commit rights come with additional responsibilities, and new committers are assigned mentors to help them learn the ropes. .FreeBSD Organization image::freebsd-organization.png[] Conflict resolution is performed by a nine member "Core Team" that is elected from the group of committers. FreeBSD does not have "corporate" committers. Individual committers are required to take responsibility for the changes they introduce to the code. The link:{committers-guide}[FreeBSD Committer's guide] <> documents the rules and responsibilities for committers. FreeBSD's project model is examined in detail in <>. === FreeBSD Release Engineering Processes FreeBSD's release engineering processes play a major role in ensuring that its released versions are of a high quality. At any point of time, FreeBSD's volunteers support multiple code lines (<>): * New features and disruptive code enters on the development branch, also known as the _-CURRENT_ branch. * _-STABLE_ branches are code lines that are branched from HEAD at regular intervals. Only tested code is allowed onto a -STABLE branch. New features are allowed once they have been tested and stabilized in the -CURRENT branch. * _-RELEASE_ branches are maintained by the FreeBSD security team. Only bug fixes for critical issues are permitted onto -RELEASE branches. [[fig-freebsd-branches]] .FreeBSD Release Branches image::freebsd-branches.png[] Code lines are kept alive for as long as there is user and developer interest in them. Machine architectures are grouped into "tiers"; _Tier 1_ architectures are fully supported by the project's release engineering and security teams, _Tier 2_ architectures are supported on a best effort basis, and experimental architectures comprise _Tier 3_. The list of link:{committers-guide}#archs[supported architectures] is part of the FreeBSD documentation collection. The release engineering team publishes a link:https://www.FreeBSD.org/releng/[road map] for future releases of FreeBSD on the project's web site. The dates laid down in the road map are not deadlines; FreeBSD is released when its code and documentation are ready. FreeBSD's release engineering processes are described in <>. [[freebsd-collaboration]] == Collaborating with FreeBSD Open-source projects like FreeBSD offer finished code of a very high quality. While access to quality source code can reduce the cost of initial development, in the long-term the costs of managing change begin to dominate. As computing environments change over the years and new security vulnerabilities are discovered, your product too needs to change and adapt. Using open-source code is best viewed not as a one-off activity, but as an __ongoing process__. The best projects to collaborate with are the ones that are __live__; i.e., with an active community, clear goals and a transparent working style. * FreeBSD has an active developer community around it. At the time of writing there are many thousands of contributors from every populated continent in the world and over 300 individuals with write access to the project's source repositories. * The goals of the FreeBSD project are <>: ** To develop a high-quality operating system for popular computer hardware, and, ** To make our work available to all under a liberal license. * FreeBSD enjoys an open and transparent working culture. Nearly all discussion in the project happens by email, on https://lists.freebsd.org/mailman/listinfo[public mailing lists] that are also archived for posterity. The project's policies are link:https://www.FreeBSD.org/internal/policies/[documented] and maintained under revision control. Participation in the project is open to all. [[freebsd-org]] === Understanding FreeBSD culture To be able to work effectively with the FreeBSD project, you need to understand the project's culture. Volunteer driven projects operate under different rules than for-profit corporates. A common mistake that companies make when venturing into the open-source world is that of underplaying these differences. *Motivation.* Most contributions to FreeBSD are done voluntarily without monetary rewards entering the picture. The factors that motivate individuals are complex, ranging from altruism, to an interest in solving the kinds of problems that FreeBSD attempts to solve. In this environment, "elegance is never optional"<>. *The Long Term View.* FreeBSD traces its roots back nearly twenty years to the work of the Computer Science Research Group at the University of California Berkeley.footnote:[FreeBSD's source repository contains a history of the project since its inception, and there are CDROMs available that contain earlier code from the CSRG.] A number of the original CSRG developers remain associated with the project. The project values long-term perspectives <>. A frequent acronym encountered in the project is DTRT, which stands for "Do The Right Thing". *Development Processes.* Computer programs are tools for communication: at one level programmers communicate their intentions using a precise notation to a tool (a compiler) that translates their instructions to executable code. At another level, the same notation is used for communication of intent between two programmers. Formal specifications and design documents are seldom used in the project. Clear and well-written code and well-written change logs (<>) are used in their place. FreeBSD development happens by "rough consensus and running code"<>. [.programlisting] .... r151864 | bde | 2005-10-29 09:34:50 -0700 (Sat, 29 Oct 2005) | 13 lines Changed paths: M /head/lib/msun/src/e_rem_pio2f.c Use double precision to simplify and optimize arg reduction for small and medium size args too: instead of conditionally subtracting a float 17+24, 17+17+24 or 17+17+17+24 bit approximation to pi/2, always subtract a double 33+53 bit one. The float version is now closer to the double version than to old versions of itself -- it uses the same 33+53 bit approximation as the simplest cases in the double version, and where the float version had to switch to the slow general case at |x| == 2^7*pi/2, it now switches at |x| == 2^19*pi/2 the same as the double version. This speeds up arg reduction by a factor of 2 for |x| between 3*pi/4 and 2^7*pi/4, and by a factor of 7 for |x| between 2^7*pi/4 and 2^19*pi/4. .... .A sample change log entry [[fig-change-log]] Communication between programmers is enhanced by the use of a common coding standard man:style[9]. *Communication Channels.* FreeBSD's contributors are spread across the world. Email (and to a lesser extent, IRC) is the preferred means of communication in the project. === Best Practices for collaborating with the FreeBSD project We now look at a few best practices for making the best use of FreeBSD in product development. Plan for the long term:: Setup processes that help in tracking the development of FreeBSD. For example: + *Track FreeBSD source code.* The project makes it easy to mirror its SVN repository using link:{committers-guide}#svn-advanced-use-setting-up-svnsync[svnsync]. Having the complete history of the source is useful when debugging complex problems and offers valuable insight into the intentions of the original developers. Use a capable source control system that allows you to easily merge changes between the upstream FreeBSD code base and your own in-house code. + <> shows a portion of an annotated listing of the file referenced by the change log in <>. The ancestry of each line of the source is clearly visible. Annotated listings showing the history of every file that is part of FreeBSD are https://svnweb.freebsd.org/[available on the web]. + [.programlisting] .... #REV #WHO #DATE #TEXT 176410 bde 2008-02-19 07:42:46 -0800 (Tue, 19 Feb 2008) #include 176410 bde 2008-02-19 07:42:46 -0800 (Tue, 19 Feb 2008) __FBSDID("$FreeBSD$"); 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) /* __ieee754_rem_pio2f(x,y) 8870 rgrimes 1995-05-29 22:51:47 -0700 (Mon, 29 May 1995) * 176552 bde 2008-02-25 05:33:20 -0800 (Mon, 25 Feb 2008) * return the remainder of x rem pi/2 in *y 176552 bde 2008-02-25 05:33:20 -0800 (Mon, 25 Feb 2008) * use double precision for everything except passing x 152535 bde 2005-11-16 18:20:04 -0800 (Wed, 16 Nov 2005) * use __kernel_rem_pio2() for large x 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) */ 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) 176465 bde 2008-02-22 07:55:14 -0800 (Fri, 22 Feb 2008) #include 176465 bde 2008-02-22 07:55:14 -0800 (Fri, 22 Feb 2008) 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) #include "math.h" .... .An annotated source listing generated using `svn blame` [[fig-svn-blame]] + *Use a gatekeeper.* Appoint a _gatekeeper_ to monitor FreeBSD development, to keep an eye out for changes that could potentially impact your products. + *Report bugs upstream.* If you notice bug in the FreeBSD code that you are using, file a https://www.FreeBSD.org/support/bugreports/[bug report]. This step helps ensure that you do not have to fix the bug the next time you take a code drop from upstream. Leverage FreeBSD's release engineering efforts:: Use code from a -STABLE development branch of FreeBSD. These development branches are formally supported by FreeBSD's release engineering and security teams and comprise of tested code. Donate code to reduce costs:: A major proportion of the costs associated with developing products is that of doing maintenance. By donating non-critical code to the project, you benefit by having your code see much wider exposure than it would otherwise get. This in turn leads to more bugs and security vulnerabilities being flushed out and performance anomalies being identified and fixed. Get support effectively:: For products with tight deadlines, it is recommended that you hire or enter into a consulting agreement with a developer or firm with FreeBSD experience. The {freebsd-jobs} is a useful communication channel to find talent. The FreeBSD project maintains a link:https://www.FreeBSD.org/commercial/consult_bycat/[gallery of consultants and consulting firms] undertaking FreeBSD work. The http://www.bsdcertification.org/[BSD Certification Group] offers certification for all the major BSD derived OSes. + For less critical needs, you can ask for help on the http://lists.FreeBSD.org/mailman/listinfo[project mailing lists]. A useful guide to follow when asking for help is given in <>. Publicize your involvement:: You are not required to publicize your use of FreeBSD, but doing so helps both your effort as well as that of the project. + Letting the FreeBSD community know that your company uses FreeBSD helps improve your chances of attracting high quality talent. A large roster of support for FreeBSD also means more mind share for it among developers. This in turn yields a healthier foundation for your future. Support FreeBSD developers:: Sometimes the most direct way to get a desired feature into FreeBSD is to support a developer who is already looking at a related problem. Help can range from hardware donations to direct financial assistance. In some countries, donations to the FreeBSD project enjoy tax benefits. The project has a dedicated link:https://www.FreeBSD.org/donations/[donations liaison] to assist donors. The project also maintains a web page where developers link:https://www.FreeBSD.org/donations/wantlist/[list their needs]. + As a policy the FreeBSD project link:{contributors}[acknowledges] all contributions received on its web site. [[conclusion]] == Conclusion The FreeBSD project's goals are to create and give away the source code for a high-quality operating system. By working with the FreeBSD project you can reduce development costs and improve your time to market in a number of product development scenarios. We examined the characteristics of the FreeBSD project that make it an excellent choice for being part of an organization's product strategy. We then looked at the prevailing culture of the project and examined effective ways of interacting with its developers. The article concluded with a list of best-practices that could help organizations collaborating with the project. :sectnums!: [bibliography] == Bibliography [[Carp1996]] [Carp1996] http://www.ietf.org/rfc/rfc1958.txt[The Architectural Principles of the Internet] B. Carpenter. The Internet Architecture Board.The Internet Architecture Board. Copyright(R) 1996. [[ComGuide]] [ComGuide] link:{committers-guide}[Committer's Guide] The FreeBSD Project. Copyright(R) 2005. [[GoldGab2005]] [GoldGab2005] http://dreamsongs.com/IHE/IHE.html[Innovation Happens Elsewhere: Open Source as Business Strategy] Ron Goldman. Richard Gabriel. Copyright(R) 2005. Morgan-Kaufmann. [[Hub1994]] [Hub1994] link:{contributing}[Contributing to the FreeBSD Project] Jordan Hubbard. Copyright(R) 1994-2005. The FreeBSD Project. [[McKu1999]] [McKu1999] http://www.usenix.org/publications/library/proceedings/usenix99/mckusick.html[Soft Updates: A Technique for Eliminating Most Synchronous Writes in the Fast Filesystem] Kirk McKusick. Gregory Ganger. Copyright(R) 1999. [[McKu1999-1]] [McKu1999-1] http://www.oreilly.com/catalog/opensources/book/kirkmck.html[Twenty Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable] Marshall Kirk McKusick. http://www.oreilly.com/catalog/opensources/book/toc.html[Open Sources: Voices from the Open Source Revolution] O'Reilly Inc.. Copyright(R) 1993. [[Mon2005]] [Mon2005] link:{bsdl-gpl}[Why you should use a BSD style license for your Open Source Project] Bruce Montague. The FreeBSD Project. Copyright(R) 2005. [[Nik2005]] [Nik2005] link:{dev-model}[A project model for the FreeBSD Project] Niklas Saers. Copyright(R) 2005. The FreeBSD Project. [[Nor1993]] [Nor1993] http://www.norvig.com/luv-slides.ps[Tutorial on Good Lisp Programming Style] Peter Norvig. Kent Pitman. Copyright(R) 1993. [[Nor2001]] [Nor2001] http://www.norvig.com/21-days.html[Teach Yourself Programming in Ten Years] Peter Norvig. Copyright(R) 2001. [[Ray2004]] [Ray2004] http://www.catb.org/~esr/faqs/smart-questions.html[How to ask questions the smart way] Eric Steven Raymond. Copyright(R) 2004. [[RelEngDoc]] [RelEngDoc] link:{releng}[FreeBSD Release Engineering] Murray Stokely. Copyright(R) 2001. The FreeBSD Project. diff --git a/documentation/content/en/articles/committers-guide/_index.adoc b/documentation/content/en/articles/committers-guide/_index.adoc index 54cf8bc241..e1ea6a26d6 100644 --- a/documentation/content/en/articles/committers-guide/_index.adoc +++ b/documentation/content/en/articles/committers-guide/_index.adoc @@ -1,3793 +1,3809 @@ --- title: Committer's Guide authors: - author: The FreeBSD Documentation Project copyright: 1999-2021 The FreeBSD Documentation Project description: Introductory information for FreeBSD committers trademarks: ["freebsd", "coverity", "ibm", "intel", "general"] weight: 25 tags: ["FreeBSD Committer's Guide", "Guide", "Community"] --- = Committer's Guide :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/teams.adoc[lines=16..-1] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/teams.adoc[lines=16..-1] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/teams.adoc[lines=16..-1] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This document provides information for the FreeBSD committer community. All new committers should read this document before they start, and existing committers are strongly encouraged to review it from time to time. Almost all FreeBSD developers have commit rights to one or more repositories. However, a few developers do not, and some of the information here applies to them as well. (For instance, some people only have rights to work with the Problem Report database). Please see <> for more information. This document may also be of interest to members of the FreeBSD community who want to learn more about how the project works. ''' toc::[] [[admin]] == Administrative Details [.informaltable] [cols="1,1", frame="none"] |=== |_Login Methods_ |man:ssh[1], protocol 2 only |_Main Shell Host_ |`freefall.FreeBSD.org` |_SMTP Host_ |`smtp.FreeBSD.org:587` (see also <>). |`_src/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/src.git` (see also <>). |`_doc/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/doc.git` (see also <>). |`_ports/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/ports.git` (see also <>). |_Internal Mailing Lists_ |developers (technically called all-developers), doc-developers, doc-committers, ports-developers, ports-committers, src-developers, src-committers. (Each project repository has its own -developers and -committers mailing lists. Archives for these lists can be found in the files [.filename]#/local/mail/repository-name-developers-archive# and [.filename]#/local/mail/repository-name-committers-archive# on the `FreeBSD.org` cluster.) |_Core Team monthly reports_ |[.filename]#/home/core/public/monthly-reports# on the `FreeBSD.org` cluster. |_Ports Management Team monthly reports_ |[.filename]#/home/portmgr/public/monthly-reports# on the `FreeBSD.org` cluster. |_Noteworthy `src/` Git Branches:_ |`stable/n` (`n`-STABLE), `main` (-CURRENT) |=== man:ssh[1] is required to connect to the project hosts. For more information, see <>. Useful links: * link:https://www.FreeBSD.org/internal/[FreeBSD Project Internal Pages] * link:https://www.FreeBSD.org/internal/machines/[FreeBSD Project Hosts] * link:https://www.FreeBSD.org/administration/[FreeBSD Project Administrative Groups] [[pgpkeys]] == OpenPGP Keys for FreeBSD Cryptographic keys conforming to the OpenPGP (__Pretty Good Privacy__) standard are used by the FreeBSD project to authenticate committers. Messages carrying important information like public SSH keys can be signed with the OpenPGP key to prove that they are really from the committer. See http://www.nostarch.com/pgp_ml.htm[PGP & GPG: Email for the Practical Paranoid by Michael Lucas] and http://en.wikipedia.org/wiki/Pretty_Good_Privacy[] for more information. [[pgpkeys-creating]] === Creating a Key Existing keys can be used, but should be checked with [.filename]#documentation/tools/checkkey.sh# first. In this case, make sure the key has a FreeBSD user ID. For those who do not yet have an OpenPGP key, or need a new key to meet FreeBSD security requirements, here we show how to generate one. [[pgpkeys-create-steps]] [.procedure] ==== . Install [.filename]#security/gnupg#. Enter these lines in [.filename]#~/.gnupg/gpg.conf# to set minimum acceptable defaults: + [.programlisting] .... fixed-list-mode keyid-format 0xlong personal-digest-preferences SHA512 SHA384 SHA256 SHA224 default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP Uncompressed use-agent verify-options show-uid-validity list-options show-uid-validity sig-notation issuer-fpr@notations.openpgp.fifthhorseman.net=%g cert-digest-algo SHA512 .... . Generate a key: + [source,shell] .... % gpg --full-gen-key gpg (GnuPG) 2.1.8; Copyright (C) 2015 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Warning: using insecure memory! Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 2048 <.> Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 3y <.> Key expires at Wed Nov 4 17:20:20 2015 MST Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Chucky Daemon <.> Email address: notreal@example.com Comment: You selected this USER-ID: "Chucky Daemon " Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o You need a Passphrase to protect your secret key. .... <.> 2048-bit keys with a three-year expiration provide adequate protection at present (2013-12). http://danielpocock.com/rsa-key-sizes-2048-or-4096-bits[] describes the situation in more detail. <.> A three year key lifespan is short enough to obsolete keys weakened by advancing computer power, but long enough to reduce key management problems. <.> Use your real name here, preferably matching that shown on government-issued ID to make it easier for others to verify your identity. Text that may help others identify you can be entered in the `Comment` section. + After the email address is entered, a passphrase is requested. Methods of creating a secure passphrase are contentious. Rather than suggest a single way, here are some links to sites that describe various methods: http://world.std.com/~reinhold/diceware.html[], http://www.iusmentis.com/security/passphrasefaq/[], http://xkcd.com/936/[], http://en.wikipedia.org/wiki/Passphrase[]. ==== Protect the private key and passphrase. If either the private key or passphrase may have been compromised or disclosed, immediately notify mailto:accounts@FreeBSD.org[accounts@FreeBSD.org] and revoke the key. Committing the new key is shown in <>. [[kerberos-ldap]] == Kerberos and LDAP web Password for FreeBSD Cluster The FreeBSD cluster requires a Kerberos password to access certain services. The Kerberos password also serves as the LDAP web password, since LDAP is proxying to Kerberos in the cluster. Some of the services which require this include: * https://bugs.freebsd.org/bugzilla[Bugzilla] * https://ci.freebsd.org[Jenkins] To create a new Kerberos account in the FreeBSD cluster, or to reset a Kerberos password for an existing account using a random password generator: [source,shell] .... % ssh kpasswd.freebsd.org .... [NOTE] ==== This must be done from a machine outside of the FreeBSD.org cluster. ==== A Kerberos password can also be set manually by logging into `freefall.FreeBSD.org` and running: [source,shell] .... % kpasswd .... [NOTE] ==== Unless the Kerberos-authenticated services of the FreeBSD.org cluster have been used previously, `Client unknown` will be shown. This error means that the `ssh kpasswd.freebsd.org` method shown above must be used first to initialize the Kerberos account. ==== [[committer.types]] == Commit Bit Types The FreeBSD repository has a number of components which, when combined, support the basic operating system source, documentation, third party application ports infrastructure, and various maintained utilities. When FreeBSD commit bits are allocated, the areas of the tree where the bit may be used are specified. Generally, the areas associated with a bit reflect who authorized the allocation of the commit bit. Additional areas of authority may be added at a later date: when this occurs, the committer should follow normal commit bit allocation procedures for that area of the tree, seeking approval from the appropriate entity and possibly getting a mentor for that area for some period of time. [.informaltable] [cols="1,1,1", frame="none"] |=== |__Committer Type__ |__Responsible__ |__Tree Components__ |src |core@ |src/ |doc |doceng@ |doc/, ports/, src/ documentation |ports |portmgr@ |ports/ |=== Commit bits allocated prior to the development of the notion of areas of authority may be appropriate for use in many parts of the tree. However, common sense dictates that a committer who has not previously worked in an area of the tree seek review prior to committing, seek approval from the appropriate responsible party, and/or work with a mentor. Since the rules regarding code maintenance differ by area of the tree, this is as much for the benefit of the committer working in an area of less familiarity as it is for others working on the tree. Committers are encouraged to seek review for their work as part of the normal development process, regardless of the area of the tree where the work is occurring. === Policy for Committer Activity in Other Trees * All committers may modify [.filename]#src/share/misc/committers-*.dot#, [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd#, and [.filename]#ports/astro/xearth/files#. * doc committers may commit documentation changes to [.filename]#src# files, such as man pages, READMEs, fortune databases, calendar files, and comment fixes without approval from a src committer, subject to the normal care and tending of commits. * Any committer may make changes to any other tree with an "Approved by" from a non-mentored committer with the appropriate bit. Mentored committers can provide a "Reviewed by" but not an "Approved by". * Committers can acquire an additional bit by the usual process of finding a mentor who will propose them to core, doceng, or portmgr, as appropriate. When approved, they will be added to 'access' and the normal mentoring period will ensue, which will involve a continuing of "Approved by" for some period. [[git-primer]] == Git Primer [NOTE] ==== this section is a work in progress... ==== [[git-basics]] === Git basics There are many primers on how to use Git on the web. There's a lot of them (search for "Git primer"). https://danielmiessler.com/study/git/ and https://gist.github.com/williewillus/068e9a8543de3a7ef80adb2938657b6b are good overviews. The Git book is also complete, but much longer https://git-scm.com/book/en/v2. There is also this website https://ohshitgit.com/ for common traps and pitfalls of Git, in case you need guidance to fix things up. This document will assume that you've read through it and will try not to belabor the basics (though it will cover them briefly). [[git-mini-primer]] === Git Mini Primer This primer is less ambitiously scoped than the old Subversion Primer, but should cover the basics. ==== Scope If you want to download FreeBSD, compile it from sources, and generally keep up to date that way, this primer is for you. It covers getting the sources, updating the sources, bisecting and touches briefly on how to cope with a few local changes. It covers the basics, and tries to give good pointers to more in-depth treatment for when the reader finds the basics insufficient. Other sections of this guide cover more advanced topics related to contributing to the project. The goal of this section is to highlight those bits of Git needed to track sources. They assume a basic understanding of Git. There are many primers for Git on the web, but the https://git-scm.com/book/en/v2[Git Book] provides one of the better treatments. ==== Keeping Current With The FreeBSD src Tree [[keeping_current]] First step: cloning a tree. This downloads the entire tree. There are two ways to download. Most people will want to do a deep clone of the repository. However, there are times when you may wish to do a shallow clone. ===== Branch names The branch names in the new Git repository are similar to the old names. For the stable branches, they are stable/X where X is the major release (like 11 or 12). The main branch in the new repository is 'main'. The main branch in the old GitHub mirror was 'master', but is now 'main'. Both reflect the defaults of Git at the time they were created. The 'main' branch is the default branch if you omit the '-b branch' or '--branch branch' options below. ===== Repositories Please see the <> for the latest information on where to get FreeBSD sources. $URL below can be obtained from that page. Note: The project doesn't use submodules as they are a poor fit for our workflows and development model. How we track changes in third-party applications is discussed elsewhere and generally of little concern to the casual user. ===== Deep Clone A deep clone pulls in the entire tree, as well as all the history and branches. It is the easiest to do. It also allows you to use Git's worktree feature to have all your active branches checked out into separate directories but with only one copy of the repository. [source,shell] .... % git clone -o freebsd $URL -b branch [dir] .... is how you make a deep clone. 'branch' should be one of the branches listed in the previous section. It is optional if it is the main branch. 'dir' is an optional directory to place it in (the default will be the name of the repo you are cloning (src, doc, etc)). You will want a deep clone if you are interested in the history, plan on making local changes, or plan on working on more than one branch. It is the easiest to keep up to date as well. If you are interested in the history, but are working with only one branch and are short on space, you can also use --single-branch to only download the one branch (though some merge commits will not reference the merged-from branch which may be important for some users who are interested in detailed versions of history). ===== Shallow Clone A shallow clone copies just the most current code, but none or little of the history. This can be useful when you need to build a specific revision of FreeBSD, or when you are just starting out and plan to track the tree more fully. You can also use it to limit history to only so many revisions. However, see below for a significant limitation of this approach. [source,shell] .... % git clone -o freebsd -b branch --depth 1 $URL [dir] .... This clones the repository, but only has the most recent version in the repository. The rest of the history is not downloaded. Should you change your mind later, you can do 'git fetch --unshallow' to get the old history. [WARNING] ==== When you make a shallow clone, you will lose the commit count in your uname output. This can make it more difficult to determine if your system needs to be updated when a security advisory is issued. ==== ===== Building Once you've downloaded, building is done as described in the handbook, eg: [source,shell] .... % cd src % make buildworld % make buildkernel % make installkernel % make installworld .... so that won't be covered in depth here. If you want to build a custom kernel, link:{handbook}#kernelconfig[the kernel config section] of the FreeBSD Handbook recommends creating a file MYKERNEL under sys/${ARCH}/conf with your changes against GENERIC. To have MYKERNEL disregarded by Git, it can be added to .git/info/exclude. ===== Updating To update both types of trees uses the same commands. This pulls in all the revisions since your last update. [source,shell] .... % git pull --ff-only .... will update the tree. In Git, a 'fast forward' merge is one that only needs to set a new branch pointer and doesn't need to re-create the commits. By always doing a 'fast forward' merge/pull, you'll ensure that you have an exact copy of the FreeBSD tree. This will be important if you want to maintain local patches. See below for how to manage local changes. The simplest is to use --autostash on the 'git pull' command, but more sophisticated options are available. ==== Selecting a Specific Version In Git, the 'git checkout' checks out both branches and specific versions. Git's versions are the long hashes rather than a sequential number. When you checkout a specific version, just specify the hash you want on the command line (the git log command can help you decide which hash you might want): [source,shell] .... % git checkout 08b8197a74 .... and you have that checked out. You will be greeted with a message similar to the following: [source,shell] .... Note: checking out '08b8197a742a96964d2924391bf9fdfeb788865d'. You are in a 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 08b8197a742a hook gpiokeys.4 to the build .... where the last line is generated from the hash you are checking out and the first line of the commit message from that revision. The hash can be abbreviated to the shortest unique length. Git itself is inconsistent about how many digits it displays. ==== Bisecting Sometimes, things go wrong. The last version worked, but the one you just updated to does not. A developer may ask you to bisect the problem to track down which commit caused the regression. Git makes bisecting changes easy with a powerful 'git bisect' command. Here's a brief outline of how to use it. For more information, you can view https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination or https://git-scm.com/docs/git-bisect for more details. The man git-bisect page is good at describing what can go wrong, what to do when versions won't build, when you want to use terms other than 'good' and 'bad', etc, none of which will be covered here. `git bisect start` will start the bisection process. Next, you need to tell a range to go through. 'git bisect good XXXXXX' will tell it the working version and 'git bisect bad XXXXX' will tell it the bad version. The bad version will almost always be HEAD (a special tag for what you have checked out). The good version will be the last one you checked out. [TIP] ==== If you want to know the last version you checked out, you should use 'git reflog': [source,shell] .... 5ef0bd68b515 (HEAD -> main, freebsd/main, freebsd/HEAD) HEAD@{0}: pull --ff-only: Fast-forward a8163e165c5b (upstream/main) HEAD@{1}: checkout: moving from b6fb97efb682994f59b21fe4efb3fcfc0e5b9eeb to main ... .... shows me moving the working tree to the main branch (a816...) and then updating from upstream (to 5ef0...). In this case, bad would be HEAD (or 5rf0bd68) and good would be a8163e165. As you can see from the output, HEAD@{1} also often works, but isn't foolproof if you have done other things to your Git tree after updating, but before you discover the need to bisect. ==== Set the 'good' version first, then set the bad (though the order doesn't matter). When you set the bad version, it will give you some statistics on the process: [source,shell] .... % git bisect start % git bisect good a8163e165c5b % git bisect bad HEAD Bisecting: 1722 revisions left to test after this (roughly 11 steps) [c427b3158fd8225f6afc09e7e6f62326f9e4de7e] Fixup r361997 by balancing parens. Duh. .... You would then build/install that version. If it's good you'd type 'git bisect good' otherwise 'git bisect bad'. If the version doesn't compile, type 'git bisect skip'. You will get a similar message to the above after each step. When you are done, report the bad version to the developer (or fix the bug yourself and send a patch). 'git bisect reset' will end the process and return you back to where you started (usually tip of main). Again, the git-bisect manual (linked above) is a good resource for when things go wrong or for unusual cases. [[git-gpg-signing]] ==== Signing the commits, tags, and pushes, with GnuPG Git knows how to sign commits, tags, and pushes. When you sign a Git commit or a tag, you can prove that the code you submitted came from you and wasn't altered while you were transferring it. You also can prove that you submitted the code and not someone else. A more in-depth documentation on signing commits and tags can be found in the https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work[Git Tools - Signing Your Work] chapter of the Git's book. The rationale behind signing pushes can be found in the https://github.com/git/git/commit/a85b377d0419a9dfaca8af2320cc33b051cbed04[commit that introduced the feature]. The best way is to simply tell Git you always want to sign commits, tags, and pushes. You can do this by setting a few configuration variables: [source,shell] .... % git config --add user.signingKey=LONG-KEY-ID % git config --add commit.gpgSign=true % git config --add tag.gpgSign=true % git config --add push.gpgSign=if-asked .... // push.gpgSign should probably be set to `yes` once we enable it, or be set with --global, so that it is enabled for all repositories. [NOTE] ====== To avoid possible collisions, make sure you give a long key id to Git. You can get the long id with: `gpg --list-secret-keys --keyid-format LONG`. ====== [TIP] ====== To use specific subkeys, and not have GnuPG to resolve the subkey to a primary key, attach `!` to the key. For example, to encrypt for the subkey `DEADBEEF`, use `DEADBEEF!`. ====== ===== Verifying signatures Commit signatures can be verified by running either `git verify-commit `, or `git log --show-signature`. Tag signatures can be verifed with `git verity-tag `, or `git tag -v `. //// Commented out for now until we decide what to do. Git pushes are a bit different, they live in a special ref in the repository. TODO: write how to verify them //// ==== Ports Considerations The ports tree operates the same way. The branch names are different and the repositories are in different locations. The cgit repository web interface for use with web browsers is at https://cgit.FreeBSD.org/ports/ . The production Git repository is at https://git.FreeBSD.org/ports.git and at ssh://anongit@git.FreeBSD.org/ports.git (or anongit@git.FreeBSD.org:ports.git). There is also a mirror on GitHub, see link:{handbook}mirrors/#mirrors[External mirrors] for an overview. The 'current' branch is 'main' . The quarterly branches are named 'yyyyQn' for year 'yyyy' and quarter 'n'. ===== Commit message formats A hook is available in the ports repository to help you write up your commit messages in https://cgit.freebsd.org/ports/tree/.hooks/prepare-commit-msg[.hooks/prepare-commit-message]. It can be enabled by running ``git config --add core.hooksPath .hooks``. The main point being that a commit message should be formatted in the following way: .... category/port: Summary. Description of why the changes where made. PR: 12345 .... [IMPORTANT] ==== The first line is the subject of the commit, it contains what port was changed, and a summary of the commit. It should contain 50 characters or less. A blank line should separate it from the rest of the commit message. The rest of the commit message should be wrapped at the 72 characters boundary. Another blank line should be added if there are any metadata fields, so that they are easily distinguishable from the commit message. ==== ==== Managing Local Changes This section addresses tracking local changes. If you have no local changes, you can stop reading now (it is the last section and OK to skip). One item that is important for all of them: all changes are local until pushed. Unlike Subversion, Git uses a distributed model. For users, for most things, there is very little difference. However, if you have local changes, you can use the same tool to manage them as you use to pull in changes from FreeBSD. All changes that you have not pushed are local and can easily be modified (git rebase, discussed below does this). ===== Keeping local changes The simplest way to keep local changes (especially trivial ones) is to use 'git stash'. In its simplest form, you use 'git stash' to record the changes (which pushes them onto the stash stack). Most people use this to save changes before updating the tree as described above. They then use 'git stash apply' to re-apply them to the tree. The stash is a stack of changes that can be examined with 'git stash list'. The git-stash man page (https://git-scm.com/docs/git-stash) has all the details. This method is suitable when you have tiny tweaks to the tree. When you have anything non trivial, you'll likely be better off keeping a local branch and rebasing. Stashing is also integrated with the 'git pull' command: just add '--autostash' to the command line. ===== Keeping a local branch [[keeping_a_local_branch]] It is much easier to keep a local branch with Git than Subversion. In Subversion you need to merge the commit, and resolve the conflicts. This is manageable, but can lead to a convoluted history that's hard to upstream should that ever be necessary, or hard to replicate if you need to do so. Git also allows one to merge, along with the same problems. That's one way to manage the branch, but it's the least flexible. In addition to merging, Git supports the concept of 'rebasing' which avoids these issues. The 'git rebase' command replays all the commits of a branch at a newer location on the parent branch. We will cover the most common scenarios that arise using it. ====== Create a branch Let's say you want to make a change to FreeBSD's ls command to never, ever do color. There are many reasons to do this, but this example will use that as a baseline. The FreeBSD ls command changes from time to time, and you'll need to cope with those changes. Fortunately, with Git rebase it usually is automatic. [source,shell] .... % cd src % git checkout main % git checkout -b no-color-ls % cd bin/ls % vi ls.c # hack the changes in % git diff # check the changes diff --git a/bin/ls/ls.c b/bin/ls/ls.c index 7378268867ef..cfc3f4342531 100644 --- a/bin/ls/ls.c +++ b/bin/ls/ls.c @@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#undef COLORLS #ifdef COLORLS #include #include % # these look good, make the commit... % git commit ls.c .... The commit will pop you into an editor to describe what you've done. Once you enter that, you have your own **local** branch in the Git repo. Build and install it like you normally would, following the directions in the handbook. Git differs from other version control systems in that you have to tell it explicitly which files to commit. I have opted to do it on the commit command line, but you can also do it with 'git add' which many of the more in depth tutorials cover. ====== Time to update When it is time to bring in a new version, it is almost the same as w/o the branches. You would update like you would above, but there is one extra command before you update, and one after. The following assumes you are starting with an unmodified tree. It is important to start rebasing operations with a clean tree (Git usually requires this). [source,shell] .... % git checkout main % git pull --no-ff % git rebase -i main no-color-ls .... This will bring up an editor that lists all the commits in it. For this example, do not change it at all. This is typically what you are doing while updating the baseline (though you also use the Git rebase command to curate the commits you have in the branch). Once you are done with the above, you have to move the commits to ls.c forward from the old version of FreeBSD to the newer one. Sometimes there are merge conflicts. That is OK. Do not panic. Instead, handle them the same as any other merge conflicts. To keep it simple, I will just describe a common issue that may arise. A pointer to a more complete treatment can be found at the end of this section. Let's say the includes changes upstream in a radical shift to terminfo as well as a name change for the option. When you updated, you might see something like this: [source,shell] .... Auto-merging bin/ls/ls.c CONFLICT (content): Merge conflict in bin/ls/ls.c error: could not apply 646e0f9cda11... no color ls Resolve all conflicts manually, mark them as resolved with "git add/rm ", then run "git rebase --continue". You can instead skip this commit: run "git rebase --skip". To abort and get back to the state before "git rebase", run "git rebase --abort". Could not apply 646e0f9cda11... no color ls .... which looks scary. If you bring up an editor, you will see it is a typical 3-way merge conflict resolution that you may be familiar with from other source code systems (the rest of ls.c has been omitted): [source,shell] .... <<<<<<< HEAD #ifdef COLORLS_NEW #include ======= #undef COLORLS #ifdef COLORLS #include >>>>>>> 646e0f9cda11... no color ls .... The new code is first, and your code is second. The right fix here is to just add a #undef COLORLS_NEW before #ifdef and then delete the old changes: [source,shell] .... #undef COLORLS_NEW #ifdef COLORLS_NEW #include .... save the file. The rebase was interrupted, so you have to complete it: [source,shell] .... % git add ls.c % git rebase --continue .... which tells Git that ls.c has been fixed and to continue the rebase operation. Since there was a conflict, you will get kicked into the editor to update the commit message if necessary. If the commit message is still accurate, just exit the editor. If you get stuck during the rebase, do not panic. git rebase --abort will take you back to a clean slate. It is important, though, to start with an unmodified tree. An aside: The above mentioned 'git reflog' comes in handy here, as it will have a list of all the (intermediate) commits that you can view or inspect or cherry-pick. For more on this topic, https://www.freecodecamp.org/news/the-ultimate-guide-to-git-merge-and-git-rebase/ provides a rather extensive treatment. It is a good resource for issues that arise occasionally but are too obscure for this guide. ===== Switching to a Different FreeBSD Branch If you wish to shift from stable/12 to the current branch. If you have a deep clone, the following will suffice: [source,shell] .... % git checkout main % # build and install here... .... If you have a local branch, though, there are one or two caveats. First, rebase will rewrite history, so you will likely want to do something to save it. Second, jumping branches tends to cause more conflicts. If we pretend the example above was relative to stable/12, then to move to main, I'd suggest the following: [source,shell] .... % git checkout no-color-ls % git checkout -b no-color-ls-stable-12 # create another name for this branch % git rebase -i stable/12 no-color-ls --onto main .... What the above does is checkout no-color-ls. Then create a new name for it (no-color-ls-stable-12) in case you need to get back to it. Then you rebase onto the main branch. This will find all the commits to the current no-color-ls branch (back to where it meets up with the stable/12 branch) and then it will replay them onto the main branch creating a new no-color-ls branch there (which is why I had you create a place holder name). ===== Migrating from an existing Git clone If you have work based on a previous Git conversion or a locally running git-svn conversion, migrating to new repository can encounter problems because Git has no knowledge about the connection between the two. When you have only a few local changes, the easiest way would be to cherry-pick those changes to the new base: [source,shell] .... % git checkout main % git cherry-pick old_branch..your_branch .... Or alternatively, do the same thing with rebase: [source,shell] .... % git rebase --onto main master your_branch .... If you do have a lot of changes, you would probably want to perform a merge instead. The idea is to create a merge point that consolidates the history of the old_branch, and the new FreeBSD repository (main). You can find out by looking up the same commit that are found on both parents: [source,shell] .... % git show old_branch .... You will see a commit message, now search for that in the new branch: [source,shell] .... % git log --grep="commit message on old_branch" freebsd/main .... You would help locate the commit hash on the new main branch, create a helper branch (in the example we call it 'stage') from that hash: [source,shell] .... % git checkout -b stage _hash_found_from_git_log_ .... Then perform a merge of the old branch: [source,shell] .... % git merge -s ours -m "Mark old branch as merged" old_branch .... With that, it's possible to merge your work branch or the main branch in any order without problem. Eventually, when you are ready to commit your work back to main, you can perform a rebase to main, or do a squash commit by combining everything into one commit. [[mfc-with-git]] === MFC (Merge From Current) Procedures ==== Summary MFC workflow can be summarized as `git cherry-pick -x` plus `git commit --amend` to adjust the commit message. For multiple commits, use `git rebase -i` to squash them together and edit the commit message. ==== Single commit MFC [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH --edit .... For MFC commits, for example a vendor import, you would need to specify one parent for cherry-pick purposes. Normally, that would be the "first parent" of the branch you are cherry-picking from, so: [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH -m 1 --edit .... If things go wrong, you'll either need to abort the cherry-pick with `git cherry-pick --abort` or fix it up and do a `git cherry-pick --continue`. Once the cherry-pick is finished, push with `git push`. If you get an error due to losing the commit race, use `git pull --rebase` and try to push again. ==== MFC to RELENG branch MFCs to branches that require approval require a bit more care. The process is the same for either a typical merge or an exceptional direct commit. * Merge or direct commit to the appropriate `stable/X` branch first before merging to the `releng/X.Y` branch. * Use the hash that's in the `stable/X` branch for the MFC to `releng/X.Y` branch. * Leave both "cherry picked from" lines in the commit message. * Be sure to add the `Approved by:` line when you are in the editor. [source,shell] .... % git checkout releng/13.0 % git cherry-pick -x $HASH --edit .... If you forget to to add the `Approved by:` line, you can do a `git commit --amend` to edit the commit message before you push the change. ==== Multiple commit MFC [source,shell] .... % git checkout -b tmp-branch stable/X % for h in $HASH_LIST; do git cherry-pick -x $h; done % git rebase -i stable/X # mark each of the commits after the first as 'squash' # Update the commit message to reflect all elements of commit, if necessary. # Be sure to retain the "cherry picked from" lines. % git push freebsd HEAD:stable/X .... If the push fails due to losing the commit race, rebase and try again: [source,shell] .... % git checkout stable/X % git pull % git checkout tmp-branch % git rebase stable/X % git push freebsd HEAD:stable/X .... Once the MFC is complete, you can delete the temporary branch: [source,shell] .... % git checkout stable/X % git branch -d tmp-branch .... ==== MFC a vendor import Vendor imports are the only thing in the tree that creates a merge commit in the main line. Cherry picking merge commits into stable/XX presents an additional difficulty because there are two parents for a merge commit. Generally, you'll want the first parent's diff since that's the diff to mainline (though there may be some exceptions). [source,shell] .... % git cherry-pick -x -m 1 $HASH .... is typically what you want. This will tell cherry-pick to apply the correct diff. There are some, hopefully, rare cases where it's possible that the mainline was merged backwards by the conversion script. Should that be the case (and we've not found any yet), you'd change the above to '-m 2' to pickup the proper parent. Just do [source,shell] .... % git cherry-pick --abort % git cherry-pick -x -m 2 $HASH .... to do that. The `--aboort` will cleanup the failed first attempt. ==== Redoing a MFC If you do a MFC, and it goes horribly wrong and you want to start over, then the easiest way is to use `git reset --hard` like so: [source,shell] .... % git reset --hard freebsd/stable/12 .... though if you have some revs you want to keep, and others you don't, using 'git rebase -i' is better. ==== Considerations when MFCing When committing source commits to stable and releng branches, we have the following goals: * Clearly mark direct commits distinct from commits that land a change from another branch. * Avoid introducing known breakage into stable and releng branches. * Allow developers to determine which changes have or have not been landed from one branch to another. With Subversion, we used the following practices to achieve these goals: * Using 'MFC' and 'MFS' tags to mark commits that merged changes from another branch. * Squashing fixup commits into the main commit when merging a change. * Recording mergeinfo so that `svn mergeinfo --show-revs` worked. With Git, we will need to use different strategies to achieve the same goals. This document aims to define best practices when merging source commits using Git that achieve these goals. In general, we aim to use Git's native support to achieve these goals rather than enforcing practices built on Subversion's model. One general note: due to technical differences with Git, we will not be using Git "merge commits" (created via `git merge`) in stable or releng branches. Instead, when this document refers to "merge commits", it means a commit originally made to `main` that is replicated or "landed" to a stable branch, or a commit from a stable branch that is replicated to a releng branch with some variation of `git cherry-pick`. ==== Finding Eligible Hashes to MFC Git provides some built-in support for this via the `git cherry` and `git log --cherry` commands. These commands compare the raw diffs of commits (but not other metadata such as log messages) to determine if two commits are identical. This works well when each commit from head is landed as a single commit to a stable branch, but it falls over if multiple commits from main are squashed together as a single commit to a stable branch. There are a few options for resolving this: 1. We could ban squashing of commits and instead require that committers stage all of the fixup / follow-up commits to stable into a single push. This would still achieve the goal of stability in stable and releng branches since pushes are atomic and users doing a simple pull will never end up with a tree that has the main commit without the fixup(s). `git bisect` is also able to cope with this model via `git bisect skip`. 2. We could adopt a consistent style for describing MFCs and write our own tooling to wrap around `git cherry` to determine the list of eligible commits. A simple approach here might be to use the syntax from `git cherry-pick -x`, but require that a squashed commit list all of the hashes (one line per hash) at the end of the commit message. Developers could do this by using `git cherry-pick -x` of each individual commit into a branch and then use `git rebase` to squash the commits down into a single commit, but collecting the `-x` annotations at the end of the landed commit log. ==== Commit message standards ===== Marking MFCs The project has adopted the following practice for marking MFCs: * Use the `-x` flag with `git cherry-pick`. This adds a line to the commit message that includes the hash of the original commit when merging. Since it is added by Git directly, committers do not have to manually edit the commit log when merging. When merging multiple commits, keep all the "cherry picked from" lines. ===== Trim Metadata? One area that was not clearly documented with Subversion (or even CVS) is how to format metadata in log messages for MFC commits. Should it include the metadata from the original commit unchanged, or should it be altered to reflect information about the MFC commit itself? Historical practice has varied, though some of the variance is by field. For example, MFCs that are relevant to a PR generally include the PR field in the MFC so that MFC commits are included in the bug tracker's audit trail. Other fields are less clear. For example, Phabricator shows the diff of the last commit tagged to a review, so including Phabricator URLs replaces the `main` commit with the landed commits. The list of reviewers is also not clear. If a reviewer has approved a change to `main`, does that mean they have approved the MFC commit? Is that true if it's identical code only, or with merely trivial reworkes? It's clearly not true for more extensive reworks. Even for identical code what if the commit doesn't conflict but introduces an ABI change? A reviewer may have ok'd a commit for `main` due to the ABI breakage but may not approve of merging the same commit as-is. One will have to use one's best judgement until clear guidelines can be agreed upon. For MFCs regulated by re@, new metadata fields are added, such as the Approved by tag for approved commits. This new metadata will have to be added via `git commit --amend` or similar after the original commit has been reviewed and approved. We may also want to reserve some metadata fields in MFC commits such as Phabricator URLs for use by re@ in the future. Preserving existing metadata provides a very simple workflow. Developers can just use `git cherry-pick -x` without having to edit the log message. If instead we choose to adjust metadata in MFCs, developers will have to edit log messages explicitly via the use of `git cherry-pick --edit` or `git commit --amend`. However, as compared to svn, at least the existing commit message can be pre-populated and metadata fields can be added or removed without having to re-enter the entire commit message. The bottom line is that developers will likely need to curate their commit message for MFCs that are non-trivial. ==== Examples ===== Merging a Single Subversion Commit This walks through the process of merging a commit to stable/12 that was originally committed to head in Subversion. In this case, the original commit is r368685. The first step is to map the Subversion commit to a Git hash. Once you have fetched refs/notes/commits, you can pass the revision number to `git log --grep`: [source,shell] .... % git log main --grep 368685 commit ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81 Author: John Baldwin Date: Wed Dec 16 00:11:30 2020 +0000 Use the 't' modifier to print a ptrdiff_t. Reviewed by: imp Obtained from: CheriBSD Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D27576 Notes: svn path=/head/; revision=368685 .... Next, MFC the commit to a `stable/12` checkout: [source,shell] .... git checkout stable/12 git cherry-pick -x ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81 --edit .... Git will invoke the editor. Use this to remove the metadata that only applied to the original commit (Phabricator URL and Reviewed by). After the editor saves the updated log message, Git completes the commit: [source,shell] .... [stable/12 3e3a548c4874] Use the 't' modifier to print a ptrdiff_t. Date: Wed Dec 16 00:11:30 2020 +0000 1 file changed, 1 insertion(+), 1 deletion(-) .... The contents of the MFCd commit can be examined via `git show`: [source,shell] .... % git show commit 3e3a548c487450825679e4bd63d8d1a67fd8bd2d (HEAD -> stable/12) Author: John Baldwin Date: Wed Dec 16 00:11:30 2020 +0000 Use the 't' modifier to print a ptrdiff_t. Obtained from: CheriBSD Sponsored by: DARPA (cherry picked from commit ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81) diff --git a/sys/compat/linuxkpi/common/include/linux/printk.h b/sys/compat/linuxkpi/common/include/linux/printk.h index 31802bdd2c99..e6510e9e9834 100644 --- a/sys/compat/linuxkpi/common/include/linux/printk.h +++ b/sys/compat/linuxkpi/common/include/linux/printk.h @@ -68,7 +68,7 @@ print_hex_dump(const char *level, const char *prefix_str, printf("[%p] ", buf); break; case DUMP_PREFIX_OFFSET: - printf("[%p] ", (const char *)((const char *)buf - + printf("[%#tx] ", ((const char *)buf - (const char *)buf_old)); break; default: .... The MFC commit can now be published via `git push` [source,shell] .... % git push freebsd Enumerating objects: 17, done. Counting objects: 100% (17/17), done. Delta compression using up to 4 threads Compressing objects: 100% (7/7), done. Writing objects: 100% (9/9), 817 bytes | 204.00 KiB/s, done. Total 9 (delta 5), reused 1 (delta 1), pack-reused 0 To gitrepo-dev.FreeBSD.org:src.git 525bd9c9dda7..3e3a548c4874 stable/12 -> stable/12 .... ===== Merging a Single Subversion Commit with a Conflict This example is similar to the previous example except that the commit in question encounters a merge conflict. In this case, the original commit is r368314. As above, the first step is to map the Subversion commit to a Git hash: [source,shell] .... % git log main --grep 368314 commit 99963f5343a017e934e4d8ea2371a86789a46ff9 Author: John Baldwin Date: Thu Dec 3 22:01:13 2020 +0000 Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. Reviewed by: np MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27469 Notes: svn path=/head/; revision=368314 .... Next, MFC the commit to a `stable/12` checkout: [source,shell] .... % git checkout stable/12 % git cherry-pick -x 99963f5343a017e934e4d8ea2371a86789a46ff9 --edit Auto-merging sys/dev/cxgbe/tom/t4_cpl_io.c CONFLICT (content): Merge conflict in sys/dev/cxgbe/tom/t4_cpl_io.c warning: inexact rename detection was skipped due to too many files. warning: you may want to set your merge.renamelimit variable to at least 7123 and retry the command. error: could not apply 99963f5343a0... Don't transmit mbufs that aren't yet ready on TOE sockets. hint: after resolving the conflicts, mark the corrected paths hint: with 'git add ' or 'git rm ' hint: and commit the result with 'git commit' .... In this case, the commit encountered a merge conflict in sys/dev/cxge/tom/t4_cpl_io.c as kernel TLS is not present in stable/12. Note that Git does not invoke an editor to adjust the commit message due to the conflict. `git status` confirms that this file has merge conflicts: [source,shell] .... % git status On branch stable/12 Your branch is up to date with 'upstream/stable/12'. You are currently cherry-picking commit 99963f5343a0. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Unmerged paths: (use "git add ..." to mark resolution) both modified: sys/dev/cxgbe/tom/t4_cpl_io.c no changes added to commit (use "git add" and/or "git commit -a") .... After editing the file to resolve the conflict, `git status` shows the conflict as resolved: [source,shell] .... % git status On branch stable/12 Your branch is up to date with 'upstream/stable/12'. You are currently cherry-picking commit 99963f5343a0. (all conflicts fixed: run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: modified: sys/dev/cxgbe/tom/t4_cpl_io.c .... The cherry-pick can now be completed: [source,shell] .... % git cherry-pick --continue .... Since there was a merge conflict, Git invokes the editor to adjust the commit message. Trim the metadata fields from the commit log from the original commit to head and save the updated log message. The contents of the MFC commit can be examined via `git show`: [source,shell] .... % git show commit 525bd9c9dda7e7c7efad2d4570c7fd8e1a8ffabc (HEAD -> stable/12) Author: John Baldwin Date: Thu Dec 3 22:01:13 2020 +0000 Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. Sponsored by: Chelsio Communications (cherry picked from commit 99963f5343a017e934e4d8ea2371a86789a46ff9) diff --git a/sys/dev/cxgbe/tom/t4_cpl_io.c b/sys/dev/cxgbe/tom/t4_cpl_io.c index 8e8c2b8639e6..43861f10b689 100644 --- a/sys/dev/cxgbe/tom/t4_cpl_io.c +++ b/sys/dev/cxgbe/tom/t4_cpl_io.c @@ -746,6 +746,8 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) for (m = sndptr; m != NULL; m = m->m_next) { int n; + if ((m->m_flags & M_NOTAVAIL) != 0) + break; if (IS_AIOTX_MBUF(m)) n = sglist_count_vmpages(aiotx_mbuf_pages(m), aiotx_mbuf_pgoff(m), m->m_len); @@ -821,8 +823,9 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) /* nothing to send */ if (plen == 0) { - KASSERT(m == NULL, - ("%s: nothing to send, but m != NULL", __func__)); + KASSERT(m == NULL || (m->m_flags & M_NOTAVAIL) != 0, + ("%s: nothing to send, but m != NULL is ready", + __func__)); break; } @@ -910,7 +913,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) toep->txsd_avail--; t4_l2t_send(sc, wr, toep->l2te); - } while (m != NULL); + } while (m != NULL && (m->m_flags & M_NOTAVAIL) == 0); /* Send a FIN if requested, but only if there's no more data to send */ if (m == NULL && toep->flags & TPF_SEND_FIN) .... The MFC commit can now be published via `git push` [source,shell] .... git push freebsd Enumerating objects: 13, done. Counting objects: 100% (13/13), done. Delta compression using up to 4 threads Compressing objects: 100% (7/7), done. Writing objects: 100% (7/7), 819 bytes | 117.00 KiB/s, done. Total 7 (delta 6), reused 0 (delta 0), pack-reused 0 To gitrepo.FreeBSD.org:src.git f4d0bc6aa6b9..525bd9c9dda7 stable/12 -> stable/12 .... [[vendor-import-git]] === Vendor Imports with Git This section describes the vendor import procedure with Git in detail. ==== Branch naming convention All vendor branches and tags start with `vendor/`. These branches and tags are visible by default. [NOTE] ==== This chapter follows the convention that the `freebsd` origin is the origin name for the official FreeBSD Git repository. If you use a different convention, replace `freebsd` with the name you use instead in the examples below. ==== We will explore an example for updating NetBSD's mtree that is in our tree. The vendor branch for this is `vendor/NetBSD/mtree`. ==== Updating an old vendor import The vendor trees usually have only the subset of the third-party software that is appropriate to FreeBSD. These trees are usually tiny in comparison to the FreeBSD tree. Git worktrees are thus quite small and fast and the preferred method to use. Make sure that whatever directory you choose below (the `../mtree`) does not currently exist. [source,shell] .... % git worktree add ../mtree vendor/NetBSD/mtree .... ==== Update the Sources in the Vendor Branch Prepare a full, clean tree of the vendor sources. Import everything but merge only what is needed. This example assumes the NetBSD checked out from their GitHub mirror in `~/git/NetBSD`. Note that "upstream" might have added or removed files, so we want to make sure deletions are propagated as well. rsync(1) is commonly installed, so I'll use that. [source,shell] .... % cd ../mtree % rsync -va --del --exclude=".git" ~/git/NetBSD/usr.sbin/mtree/ . % git add -A % git status ... % git diff --staged ... % git commit -m"Vendor import of NetBSD's mtree at 2020-12-11" [vendor/NetBSD/mtree 8e7aa25fcf1] Vendor import of NetBSD's mtree at 2020-12-11 7 files changed, 114 insertions(+), 82 deletions(-) % git tag -a vendor/NetBSD/mtree/20201211 .... Note: I run the `git diff` and `git status` commands to make sure nothing weird was present. Also I used `-m` to illustrate, but you should compose a proper message in an editor (using a commit message template). It is also important to create an annotated tag, otherwise the push will be rejected. Only annotated tags are allowed to be pushed. The annotated tag gives you a chance to enter a commit message. Enter the version you are importing, along with any salient new features or fixes in that version. ==== Updating the FreeBSD Copy At this point you can push the import to vendor into our repo. [source,shell] .... % git push --follow-tags freebsd vendor/NetBSD/mtree .... `--follow-tags` tells `git push` to also push tags associated with the locally committed revision. ==== Updating the FreeBSD source tree Now you need to update the mtree in FreeBSD. The sources live in `contrib/mtree` since it is upstream software. [source,shell] .... % cd ../src % git subtree merge -P contrib/mtree vendor/NetBSD/mtree .... This would generate a subtree merge commit of `contrib/mtree` against the local `vendor/NetBSD/mtree` branch. If there were conflicts, you would need to fix them before committing. Include details about the changes being merged in the merge commit message. ==== Rebasing your change against latest FreeBSD source tree Because the current policy recommends against using merges, if the upstream FreeBSD `main` moved forward before you get a chance to push, you would have to redo the merge. Regular `git rebase` or `git pull --rebase` doesn't know how to rebase a merge commit **as a merge commit**, so instead of that you would have to recreate the commit. The easiest way to do this would be to create a side branch with the **contents** of the merged tree: [source,shell] .... % cd ../src % git fetch freebsd % git checkout -b merge_result % git merge freebsd/main .... Typically, there would be no merge conflicts here (because developers tend to work on different components). In the worst case scenario, you would still have to resolve merge conflicts, if there was any, but this should be really rare. Now, checkout `freebsd/main` again as `new_merge`, and redo the merge: [source,shell] .... % git checkout -b new_merge freebsd/main % git subtree merge -P contrib/mtree vendor/NetBSD/mtree .... Instead of resolving the conflicts, perform this instead: [source,shell] .... % git checkout merge_result . .... Which will overwrite the files with conflicts with the version found in `merge_result`. Examine the tree against `merge_result` to make sure that you haven't missed deleted files: [source,shell] .... % git diff merge_result .... ==== Pushing the changes Once you are sure that you have a set of deltas you think is good, you can push it to a fork off GitHub or GitLab for others to review. One nice thing about Git is that it allows you to publish rough drafts of your work for others to review. While phabricator is good for content review, publishing the updated vendor branch and merge commits lets others check the details as they will eventually appear in the repository. After review, when you are sure it is a good change, you can push it to the FreeBSD repo: [source,shell] .... % git push freebsd main .... === Creating a new vendor branch There are a number of ways to create a new vendor branch. The recommended way is to create a new repository and then merge that with FreeBSD. If one is importing `glorbnitz` into the FreeBSD tree, release 3.1415. For the sake of simplicity, we will not trim this release. It is a simple user command that puts the nitz device into different magical glorb states and is small enough trimming will not save much. ==== Create the repo [source,shell] .... % cd /some/where % mkdir glorbnitz % cd glorbnitz % git init % git checkout -b vendor/glorbnitz .... At this point, you have a new repo, where all new commits will go on the `vendor/glorbnitz` branch. Git experts can also do this right in their FreeBSD clone, using `git checkout --orphan vendor/glorbnitz` if they are more comfortable with that. ==== Copy the sources in Since this is a new import, you can just cp the sources in, or use tar or even rsync as shown above. And we will add everything, assuming no dot files. [source,shell] .... % cp -r ~/glorbnitz/* . % git add * .... At this point, you should have a pristine copy of glorbnitz ready to commit. [source,shell] .... % git commit -m"Import GlorbNitz frobnosticator revision 3.1415" .... As above, I used `-m` for simplicity, but you should likely create a commit message that explains what a Glorb is and why you'd use a Nitz to get it. Not everybody will know. But for your actual commit, you should follow the <> section instead of emulating the brief style used here. ==== Now import it into our repository Now you need to import the branch into our repository. [source,shell] .... % cd /path/to/freebsd/repo/src % git remote add glorbnitz /some/where/glorbnitz % git fetch glorbnitz vendor/glorbnitz .... Note the vendor/glorbnitz branch is in the repo. At this point the `/some/where/glorbnitz` can be deleted, if you like. It was only a means to an end. ==== Tag and push Steps from here on out are much the same as they are in the case of updating a vendor branch, though without the updating the vendor branch step. [source,shell] .... % git worktree add ../glorbnitz vendor/glorbnitz % cd ../glorbnitz % git tag --annotate vendor/glorbnitz/3.1415 # Make sure the commit is good with "git show" % git push --follow-tags freebsd vendor/glorbnitz .... By 'good' we mean: . All the right files are present . None of the wrong files are present . The vendor branch points at something sensible . The tag looks good, and is annotated . The commit message for the tag has a quick summary of what's new since the last tag ==== Time to finally merge it into the base tree [source,shell] .... % cd ../src % git subtree add -P contrib/glorbnitz vendor/glorbnitz # Make sure the commit is good with "git show" % git commit --amend # one last sanity check on commit message % git push freebsd .... Here 'good' means: . All the right files, and none of the wrong ones, were merged into contrib/glorbnitz. . No other changes are in the tree. . The commit messages look <>. It should contain a summary of what's changed since the last merge to the FreeBSD main line and any caveats. . UPDATING should be updated if there is anything of note, such as user visible changes, important upgrade concerns, etc. [NOTE] ==== This hasn't connected `glorbnitz` to the build yet. How so do that is specific to the software being imported and is beyond the scope of this tutorial. ==== === FreeBSD Src Committer Transition Guide This section is designed to walk people through the conversion process from Subversion to Git, written from the source committer's point of view. ==== Migrating from a Subversion tree This section will cover a couple of common scenarios for migrating from using the FreeBSD Subversion repo to the FreeBSD source Git repo. The FreeBSD Git conversion is still in beta status, so some minor things may change between this and going into production. The first thing to do is install Git. Any version of Git will do, though the latest one in ports / packages generally will be good. Either build it from ports, or install it using pkg (though some folks might use `su` or `doas` instead of `sudo`): [source,shell] .... % sudo pkg install git .... ===== No staged changes migration If you have no changes pending, the migration is straightforward. In this, you abandon the Subversion tree and clone the Git repository. It's likely best to retain your Subversion tree, in case there's something you've forgotten about there. First, let's clone the repository: [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src .... will create a clone of the FreeBSD src repository into a subdirectory called `freebsd-src` and include the 'notes' about the revisions. We are currently mirroring the source repository to https://github.com/freebsd/freebsd-src.git as well. https://github.com/freebsd/freebsd-legacy.git has the old GitHub mirror with the old hashes should you need that for your migration. The GitHub `master` branch has been frozen. As the default in Git has changed, we've shifted from `master` to `main`; the new repository uses `main`. We also mirror the repository to GitLab at https://gitlab.com/FreeBSD/src.git . It's useful to have the old Subversion revisions available. This data is stored using Git notes, but Git doesn't fetch those by default. The --config and the argument above changed the default to fetch the notes. If you've cloned the repository without this, or wish to add notes to a previously cloned repository, use the following commands: [source,shell] .... % git config --add remote.freebsd.fetch "+refs/notes/*:refs/notes/*" % git fetch .... At this point you have the src checked out into a Git tree, ready to do other things. ===== But I have changes that I've not committed If you are migrating from a tree that has changes you've not yet committed to FreeBSD, you'll need to follow the steps from the previous section first, and then follow these. [source,shell] .... % cd path-to-svn-checkout-tree % svn diff > /tmp/src.diff % cd _mumble_/freebsd-src % git checkout -b working .... This will create a diff of your current changes. The last command creates a branch called `working` though you can call it whatever you want. [source,shell] .... % git apply /tmp/src.diff .... this will apply all your pending changes to the working tree. This doesn't commit the change, so you'll need to make this permanent: [source,shell] .... % git add _files_ % git commit .... The last command will commit these changes to the branch. The editor will prompt you for a commit message. Enter one as if you were committing to FreeBSD. At this point, your work is preserved, and in the Git repository. ===== Keeping current So, time passes. It's time now to update the tree for the latest changes upstream. When you checkout `main` make sure that you have no diffs. It's a lot easier to commit those to a branch (or use `git stash`) before doing the following. If you are used to `git pull`, we strongly recommend using the `--ff-only` option, and further setting it as the default option. Alternatively, `git pull --rebase` is useful if you have changes staged in the main branch. [source,shell] .... % git config --global pull.ff only .... You may need to omit the --global if you want this setting to apply to only this repository. [source,shell] .... % cd freebsd-src % git checkout main % git pull (--ff-only|--rebase) .... There is a common trap, that the combination command `git pull` will try to perform a merge, which would sometimes creates a merge commit that didn't exist before. This can be harder to recover from. The longer form is also recommended. [source,shell] .... % cd freebsd-src % git checkout main % git fetch freebsd % git merge --ff-only freebsd/main .... These commands reset your tree to the main branch, and then update it from where you pulled the tree from originally. It's important to switch to `main` before doing this so it moves forward. Now, it's time to move the changes forward: [source,shell] .... % git rebase -i main working .... This will bring up an interactive screen to change the defaults. For now, just exit the editor. Everything should just apply. If not, then you'll need to resolve the diffs. https://docs.github.com/en/free-pro-team@latest/github/using-git/resolving-merge-conflicts-after-a-git-rebase[This github document] can help you navigate this process. ===== Time to push changes upstream First, ensure that the push URL is properly configured for the upstream repository. [source,shell] .... % git remote set-url --push freebsd ssh://git@gitrepo.freebsd.org/src.git .... Then, verify that user name and email are configured right. We require that they exactly match the passwd entry in FreeBSD cluster. Use [source,shell] .... freefall% gen-gitconfig.sh .... on freefall.freebsd.org to get a recipe that you can use directly, assuming /usr/local/bin is in the PATH. The below command merges the 'working' branch into the upstream main line. It's important that you curate your changes to be just like you want them in the FreeBSD source repo before doing this. [source,shell] .... % git push freebsd working:main .... If your push is rejected due to losing a commit race, rebase your branch before trying again: [source,shell] .... % git checkout working % git fetch freebsd % git rebase freebsd/main % git push freebsd working:main .... ===== Finding the Subversion Revision You'll need to make sure that you've fetched the notes (see the `No staged changes migration` section above for details. Once you have these, notes will show up in the git log command like so: [source,shell] .... % git log .... If you have a specific version in mind, you can use this construct: [source,shell] .... % git log --grep revision=XXXX .... to find the specific revision. The hex number after 'commit' is the hash you can use to refer to this commit. ==== Migrating from GitHub fork Note: as of this writing, https://github.com/freebsd/freebsd-src is mirroring all official branches, along with a `master` branch which is the legacy svn2git result. The `master` branch will not be updated anymore, and the link:https://github.com/freebsd/freebsd-src/commit/de1aa3dab23c06fec962a14da3e7b4755c5880cf[last commit] contains the instructions for migrating to the new `main` branch. We'll retain the `master` branch for a certain time, but in the future it will only be kept in the link:https://github.com/freebsd/freebsd-legacy[freebsd-legacy] repository. When migrating branches from a GitHub fork from the old GitHub mirror to the official repo, the process is straight forward. This assumes that you have a `freebsd` upstream pointing to GitHub, adjust if necessary. This also assumes a clean tree before starting... ===== Add the new `freebsd` upstream repository: [source,shell] .... % git remote add freebsd https://git.freebsd.org/src.git % git fetch freebsd % git checkout --track freebsd/main .... ===== Rebase all your WIP branches. For each branch FOO, do the following after fetching the `freebsd` sources and creating a local `main` branch with the above checkout: [source,shell] .... % git rebase -i freebsd/master FOO --onto main .... And you'll now be tracking the official repository. You can then follow the `Keeping Current` section above to stay up to date. If you need to then commit work to FreeBSD, you can do so following the `Time to push changes upstream` instructions. You'll need to do the following once to update the push URL if you are a FreeBSD committer: [source,shell] .... % git remote set-url --push freebsd ssh://git@gitrepo.freebsd.org/src.git .... (note that gitrepo.freebsd.org will be change to repo.freebsd.org in the future.) You will also need to add `freebsd` as the location to push to. The author recommends that your upstream GitHub repository remain the default push location so that you only push things into FreeBSD you intend to by making it explicit. [[git-faq]] === Git FAQ This section provides a number of targeted answers to questions that are likely to come up often for users and developers. [NOTE] ==== We use the common convention of having the origin for the FreeBSD repository being 'freebsd' rather than the default 'origin' to allow people to use that for their own development and to minimize "whoopse" pushes to the wrong repository. ==== ==== Users ===== How do I track -current and -stable with only one copy of the repository? **Q:** Although disk space is not a huge issue, it's more efficient to use only one copy of the repository. With SVN mirroring, I could checkout multiple trees from the same repository. How do I do this with Git? **A:** You can use Git worktrees. There's a number of ways to do this, but the simplest way is to use a clone to track -current, and a worktree to track stable releases. While using a 'bare repository' has been put forward as a way to cope, it's more complicated and will not be documented here. First, you need to clone the FreeBSD repository, shown here cloning into `freebsd-current` to reduce confusion. $URL is whatever mirror works best for you: [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' $URL freebsd-current .... then once that's cloned, you can simply create a worktree from it: [source,shell] .... % cd freebsd-current % git worktree add ../freebsd-stable-12 stable/12 .... this will checkout `stable/12` into a directory named `freebsd-stable-12` that's a peer to the `freebsd-current` directory. Once created, it's updated very similarly to how you might expect: [source,shell] .... % cd freebsd-current % git checkout main % git pull --ff-only # changes from upstream now local and current tree updated % cd ../freebsd-stable-12 % git merge --ff-only freebsd/stable/12 # now your stable/12 is up to date too .... I recommend using `--ff-only` because it's safer and you avoid accidentally getting into a 'merge nightmare' where you have an extra change in your tree, forcing a complicated merge rather than a simple one. Here's https://adventurist.me/posts/00296[a good writeup] that goes into more detail. ==== Developers ===== Ooops! I committed to `main` instead of a branch. **Q:** From time to time, I goof up and commit to main instead of to a branch. What do I do? **A:** First, don't panic. Second, don't push. In fact, you can fix almost anything if you haven't pushed. All the answers in this section assume no push has happened. The following answer assumes you committed to `main` and want to create a branch called `issue`: [source,shell] .... % git branch issue # Create the 'issue' branch % git reset --hard freebsd/main # Reset 'main' back to the official tip % git checkout issue # Back to where you were .... ===== Ooops! I committed something to the wrong branch! **Q:** I was working on feature on the `wilma` branch, but accidentally committed a change relevant to the `fred` branch in 'wilma'. What do I do? **A:** The answer is similar to the previous one, but with cherry picking. This assumes there's only one commit on wilma, but will generalize to more complicated situations. It also assumes that it's the last commit on wilma (hence using wilma in the `git cherry-pick` command), but that too can be generalized. [source,shell] .... # We're on branch wilma % git checkout fred # move to fred branch % git cherry-pick wilma # copy the misplaced commit % git checkout wilma # go back to wilma branch % git reset --hard HEAD^ # move what wilma refers to back 1 commit .... Git experts would first rewind the wilma branch by 1 commit, switch over to fred and then use `git reflog` to see what that 1 deleted commit was and cherry-pick it over. **Q:** But what if I want to commit a few changes to `main`, but keep the rest in `wilma` for some reason? **A:** The same technique above also works if you are wanting to 'land' parts of the branch you are working on into `main` before the rest of the branch is ready (say you noticed an unrelated typo, or fixed an incidental bug). You can cherry pick those changes into main, then push to the parent repository. Once you've done that, cleanup couldn't be simpler: just `git rebase -i`. Git will notice you've done this and skip the common changes automatically (even if you had to change the commit message or tweak the commit slightly). There's no need to switch back to wilma to adjust it: just rebase! **Q:** I want to split off some changes from branch `wilma` into branch `fred` **A:** The more general answer would be the same as the previous. You'd checkout/create the `fred` branch, cherry pick the changes you want from `wilma` one at a time, then rebase `wilma` to remove those changes you cherry picked. `git rebase -i main wilma` will toss you into an editor, and remove the `pick` lines that correspond to the commits you copied to `fred`. If all goes well, and there are no conflicts, you're done. If not, you'll need to resolve the conflicts as you go. The other way to do this would be to checkout `wilma` and then create the branch `fred` to point to the same point in the tree. You can then `git rebase -i` both these branches, selecting the changes you want in `fred` or `wilma` by retaining the pick likes, and deleting the rest from the editor. Some people would create a tag/branch called `pre-split` before starting in case something goes wrong in the split. You can undo it with the following sequence: [source,shell] .... % git checkout pre-split # Go back % git branch -D fred # delete the fred branch % git checkout -B wilma # reset the wilma branch % git branch -d pre-split # Pretend it didn't happen .... The last step is optional. If you are going to try again to split, you'd omit it. **Q:** But I did things as I read along and didn't see your advice at the end to create a branch, and now `fred` and `wilma` are all screwed up. How do I find what `wilma` was before I started. I don't know how many times I moved things around. **A:** All is not lost. You can figure out it, so long as it hasn't been too long, or too many commits (hundreds). So I created a wilma branch and committed a couple of things to it, then decided I wanted to split it into fred and wilma. Nothing weird happened when I did that, but let's say it did. The way to look at what you've done is with the `git reflog`: [source,shell] .... % git reflog 6ff9c25 (HEAD -> wilma) HEAD@{0}: rebase -i (finish): returning to refs/heads/wilma 6ff9c25 (HEAD -> wilma) HEAD@{1}: rebase -i (start): checkout main 869cbd3 HEAD@{2}: rebase -i (start): checkout wilma a6a5094 (fred) HEAD@{3}: rebase -i (finish): returning to refs/heads/fred a6a5094 (fred) HEAD@{4}: rebase -i (pick): Encourage contributions 1ccd109 (freebsd/main, main) HEAD@{5}: rebase -i (start): checkout main 869cbd3 HEAD@{6}: rebase -i (start): checkout fred 869cbd3 HEAD@{7}: checkout: moving from wilma to fred 869cbd3 HEAD@{8}: commit: Encourage contributions ... % .... Here we see the changes I've made. You can use it to figure out where things went wrong. I'll just point out a few things here. The first one is that HEAD@{X} is a 'commitish' thing, so you can use that as an argument to a command. Although if that command commits anything to the repository, the X numbers change. You can also use the hash (first column). Next, 'Encourage contributions' was the last commit I made to `wilma` before I decided to split things up. You can also see the same hash is there when I created the `fred` branch to do that. I started by rebasing `fred` and you see the 'start', each step, and the 'finish' for that process. While we don't need it here, you can figure out exactly what happened. Fortunately, to fix this, you can follow the prior answer's steps, but with the hash `869cbd3` instead of `pre-split`. While that seems a bit verbose, it's easy to remember since you're doing one thing at a time. You can also stack: [source,shell] .... % git checkout -B wilma 869cbd3 % git branch -D fred .... and you are ready to try again. The 'checkout -B' with the hash combines checking out and creating a branch for it. The -B instead of -b forces the movement of a pre-existing branch. Either way works, which is what's great (and awful) about Git. One reason I tend to use `git checkout -B xxxx hash` instead of checking out the hash, and then creating / moving the branch is purely to avoid the slightly distressing message about detached heads: [source,shell] .... % git checkout 869cbd3 M faq.md Note: checking out '869cbd3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 869cbd3 Encourage contributions % git checkout -B wilma .... this produces the same effect, but I have to read a lot more and severed heads aren't an image I like to contemplate. ===== Ooops! I did a `git pull` and it created a merge commit, what do I do? **Q:** I was on autopilot and did a `git pull` for my development tree and that created a merge commit on the mainline. How do I recover? **A:** This can happen when you invoke the pull with your development branch checked out. Right after the pull, you will have the new merge commit checked out. Git supports a `HEAD^#` syntax to examine the parents of a merge commit: [source,shell] .... git log --oneline HEAD^1 # Look at the first parent's commits git log --oneline HEAD^2 # Look at the second parent's commits .... From those logs, you can easily identify which commit is your development work. Then you simply reset your branch to the corresponding `HEAD^#`: [source,shell] .... git reset --hard HEAD^2 .... **Q:** But I also need to fix my `main` branch. How do I do that? **A:** Git keeps track of the remote repository branches in a `freebsd/` namespace. To fix your `main` branch, just make it point to the remote's `main`: [source,shell] .... git branch -f main freebsd/main .... There's nothing magical about branches in Git: they are just labels on a graph that are automatically moved forward by making commits. So the above works because you're just moving a label. There's no metadata about the branch that needs to be preserved due to this. ===== Mixing and matching branches **Q:** So I have two branches `worker` and `async` that I'd like to combine into one branch called `feature` while maintaining the commits in both. **A:** This is a job for cherry pick. [source,shell] .... % git checkout worker % git checkout -b feature # create a new branch % git cherry-pick main..async # bring in the changes .... You now have a new branch called `feature`. This branch combines commits from both branches. You can further curate it with `git rebase`. **Q:** I have a branch called `driver` and I'd like to break it up into `kernel` and `userland` so I can evolve them separately and commit each branch as it becomes ready. **A:** This takes a little bit of prep work, but `git rebase` will do the heavy lifting here. [source,shell] .... % git checkout driver # Checkout the driver % git checkout -b kernel # Create kernel branch % git checkout -b userland # Create userland branch .... Now you have two identical branches. So, it's time to separate out the commits. We'll assume first that all the commits in `driver` go into either the `kernel` or the `userland` branch, but not both. [source,shell] .... % git rebase -i main kernel .... and just include the changes you want (with a 'p' or 'pick' line) and just delete the commits you don't (this sounds scary, but if worse comes to worse, you can throw this all away and start over with the `driver` branch since you've not yet moved it). [source,shell] .... % git rebase -i main userland .... and do the same thing you did with the `kernel` branch. **Q:** Oh great! I followed the above and forgot a commit in the `kernel` branch. How do I recover? **A:** You can use the `driver` branch to find the hash of the commit is missing and cherry pick it. [source,shell] .... % git checkout kernel % git log driver % git cherry-pick $HASH .... **Q:** OK. I have the same situation as the above, but my commits are all mixed up. I need parts of one commit to go to one branch and the rest to go to the other. In fact, I have several. Your rebase method to select sounds tricky. **A:** In this situation, you'd be better off to curate the original branch to separate out the commits, and then use the above method to split the branch. So let's assume that there's just one commit with a clean tree. You can either use `git rebase` with an `edit` line, or you can use this with the commit on the tip. The steps are the same either way. The first thing we need to do is to back up one commit while leaving the changes uncommitted in the tree: [source,shell] .... % git reset HEAD^ .... Note: Do not, repeat do not, add `--hard` here since that also removes the changes from your tree. Now, if you are lucky, the change needing to be split up falls entirely along file lines. In that case you can just do the usual `git add` for the files in each group than do a `git commit`. Note: when you do this, you'll lose the commit message when you do the reset, so if you need it for some reason, you should save a copy (though `git log $HASH` can recover it). If you are not lucky, you'll need to split apart files. There's another tool to do that which you can apply one file at a time. [source,shell] .... git add -i foo/bar.c .... will step through the diffs, prompting you, one at time, whether to include or exclude the hunk. Once you're done, `git commit` and you'll have the remainder in your tree. You can run it multiple times as well, and even over multiple files (though I find it easier to do one file at a time and use the `git rebase -i` to fold the related commits together). ==== Cloning and Mirroring **Q:** I'd like to mirror the entire Git repository, how do I do that? **A:** If all you want to do is mirror, then [source,shell] .... % git clone --mirror $URL .... will do the trick. However, there are two disadvantages to this if you want to use it for anything other than a mirror you'll reclone. First, this is a 'bare repository' which has the repository database, but no checked out worktree. This is great for mirroring, but terrible for day to day work. There's a number of ways around this with 'git worktree': [source,shell] .... % git clone --mirror https://git.freebsd.org/ports.git ports.git % cd ports.git % git worktree add ../ports main % git worktree add ../quarterly branches/2020Q4 % cd ../ports .... But if you aren't using your mirror for further local clones, then it's a poor match. The second disadvantage is that Git normally rewrites the refs (branch name, tags, etc) from upstream so that your local refs can evolve independently of upstream. This means that you'll lose changes if you are committing to this repository on anything other than private project branches. **Q:** So what can I do instead? **A:** Well, you can stuff all of the upstream repository's refs into a private namespace in your local repository. Git clones everything via a 'refspec' and the default refspec is: [source,shell] .... fetch = +refs/heads/*:refs/remotes/freebsd/* .... which says just fetch the branch refs. However, the FreeBSD repository has a number of other things in it. To see those, you can add explicit refspecs for each ref namespace, or you can fetch everything. To setup your repository to do that: [source,shell] .... git config --add remote.freebsd.fetch '+refs/*:refs/freebsd/*' .... which will put everything in the upstream repository into your local repository's 'refs/freebsd/' namespace. Please note, that this also grabs all the unconverted vendor branches and the number of refs associated with them is quite large. You'll need to refer to these 'refs' with their full name because they aren't in and of Git's regular namespaces. [source,shell] .... git log refs/freebsd/vendor/zlib/1.2.10 .... would look at the log for the vendor branch for zlib starting at 1.2.10. === Collaborating with others One of the keys to good software development on a project as large as FreeBSD is the ability to collaborate with others before you push your changes to the tree. The FreeBSD project's Git repositories do not, yet, allow user-created branches to be pushed to the repository, and therefore if you wish to share your changes with others you must use another mechanism, such as a hosted GitLab or GitHub, in order to share changes in a user-generated branch. The following instructions show how to set up a user-generated branch, based on the FreeBSD main branch, and push it to GitHub. Before you begin, make sure that your local Git repo is up to date and has the correct origins set <> [source,shell] ```` % git remote -v freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) ```` The first step is to create a fork of https://github.com/freebsd/freebsd-src[FreeBSD] on GitHub following these https://docs.github.com/en/github/getting-started-with-github/fork-a-repo[guidelines]. The destination of the fork should be your own, personal, GitHub account (gvnn3 in my case). Now add a remote on your local system that points to your fork: [source,shell] .... % git remote add github git@github.com:gvnn3/freebsd-src.git % git remote -v github git@github.com:gvnn3/freebsd-src.git (fetch) github git@github.com:gvnn3/freebsd-src.git (push) freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) .... With this in place you can create a branch <> [source,shell] .... % git checkout -b gnn-pr2001-fix .... Make whatever modifications you wish in your branch. Build, test, and once you're ready to collaborate with others it's time to push your changes into your hosted branch. Before you can push you'll have to set the appropriate upstream, as Git will tell you the first time you try to push to your +github+ remote: [source,shell] .... % git push github fatal: The current branch gnn-pr2001-fix has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream github gnn-pr2001-fix .... Setting the push as +git+ advises allows it to succeed: [source,shell] .... % git push --set-upstream github gnn-feature Enumerating objects: 20486, done. Counting objects: 100% (20486/20486), done. Delta compression using up to 8 threads Compressing objects: 100% (12202/12202), done. Writing objects: 100% (20180/20180), 56.25 MiB | 13.15 MiB/s, done. Total 20180 (delta 11316), reused 12972 (delta 7770), pack-reused 0 remote: Resolving deltas: 100% (11316/11316), completed with 247 local objects. remote: remote: Create a pull request for 'gnn-feature' on GitHub by visiting: remote: https://github.com/gvnn3/freebsd-src/pull/new/gnn-feature remote: To github.com:gvnn3/freebsd-src.git * [new branch] gnn-feature -> gnn-feature Branch 'gnn-feature' set up to track remote branch 'gnn-feature' from 'github'. .... Subsequent changes to the same branch will push correctly by default: [source,shell] .... % git push Enumerating objects: 4, done. Counting objects: 100% (4/4), done. Delta compression using up to 8 threads Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 314 bytes | 1024 bytes/s, done. Total 3 (delta 1), reused 1 (delta 0), pack-reused 0 remote: Resolving deltas: 100% (1/1), completed with 1 local object. To github.com:gvnn3/freebsd-src.git 9e5243d7b659..cf6aeb8d7dda gnn-feature -> gnn-feature .... At this point your work is now in your branch on +GitHub+ and you can share the link with other collaborators. [[vcs-history]] == Version Control History The project has moved to <>. The FreeBSD source repository switched from CVS to Subversion on May 31st, 2008. The first real SVN commit is __r179447__. The source repository switched from Subversion to Git on December 23rd, 2020. The last real svn commit is __r368820__. The first real git commit hash is __5ef5f51d2bef80b0ede9b10ad5b0e9440b60518c__ The FreeBSD `doc/www` repository switched from CVS to Subversion on May 19th, 2012. The first real SVN commit is __r38821__. The documentation repository switched from Subversion to Git on December 8th, 2020. The last SVN commit is __r54737__. The first real git commit hash is __3be01a475855e7511ad755b2defd2e0da5d58bbe__. The FreeBSD `ports` repository switched from CVS to Subversion on July 14th, 2012. The first real SVN commit is __r300894__. The ports repository switched from Subversion to Git on April 6, 2021. The last SVN commit is __r569609__ The first real git commit hash is __ed8d3eda309dd863fb66e04bccaa513eee255cbf__. [[conventions]] == Setup, Conventions, and Traditions There are a number of things to do as a new developer. The first set of steps is specific to committers only. These steps must be done by a mentor for those who are not committers. [[conventions-committers]] === For New Committers Those who have been given commit rights to the FreeBSD repositories must follow these steps. * Get mentor approval before committing each of these changes! * All [.filename]#src# commits go to FreeBSD-CURRENT first before being merged to FreeBSD-STABLE. The FreeBSD-STABLE branch must maintain ABI and API compatibility with earlier versions of that branch. Do not merge changes that break this compatibility. [[commit-steps]] [.procedure] ==== *Procedure 1. Steps for New Committers* . Add an Author Entity + [.filename]#doc/shared/authors.adoc# - Add an author entity. Later steps depend on this entity, and missing this step will cause the [.filename]#doc/# build to fail. This is a relatively easy task, but remains a good first test of version control skills. . Update the List of Developers and Contributors + [.filename]#doc/documentation/content/en/articles/contributors/contrib-committers.adoc# - Add an entry, which will then appear in the "Developers" section of the link:{contributors}#staff-committers[Contributors List]. Entries are sorted by last name. + [.filename]#doc/documentation/content/en/articles/contributors/contrib-additional.adoc# - _Remove_ the entry. Entries are sorted by first name. . Add a News Item + [.filename]#doc/website/data/en/news/news.toml# - Add an entry. Look for the other entries that announce new committers and follow the format. Use the date from the commit bit approval email from mailto:core@FreeBSD.org[core@FreeBSD.org]. . Add a PGP Key + `{des}` has written a shell script ([.filename]#doc/documentation/tools/addkey.sh#) to make this easier. See the https://cgit.freebsd.org/doc/plain/documentation/static/pgpkeys/README[README] file for more information. + Use [.filename]#doc/documentation/tools/checkkey.sh# to verify that keys meet minimal best-practices standards. + After adding and checking a key, add both updated files to source control and then commit them. Entries in this file are sorted by last name. + [NOTE] ====== It is very important to have a current PGP/GnuPG key in the repository. The key may be required for positive identification of a committer. For example, the `{admins}` might need it for account recovery. A complete keyring of `FreeBSD.org` users is available for download from link:https://www.FreeBSD.org/doc/pgpkeyring.txt[https://www.FreeBSD.org/doc/pgpkeyring.txt]. ====== . Update Mentor and Mentee Information + [.filename]#src/share/misc/committers-.dot# - Add an entry to the current committers section, where _repository_ is `doc`, `ports`, or `src`, depending on the commit privileges granted. + Add an entry for each additional mentor/mentee relationship in the bottom section. . Generate a Kerberos Password + See <> to generate or set a Kerberos for use with other FreeBSD services like the bug tracking database. . Optional: Enable Wiki Account + https://wiki.freebsd.org[FreeBSD Wiki] Account - A wiki account allows sharing projects and ideas. Those who do not yet have an account can follow instructions on the https://wiki.freebsd.org/AboutWiki[AboutWiki Page] to obtain one. Contact mailto:wiki-admin@FreeBSD.org[wiki-admin@FreeBSD.org] if you need help with your Wiki account. . Optional: Update Wiki Information + Wiki Information - After gaining access to the wiki, some people add entries to the https://wiki.freebsd.org/HowWeGotHere[How We Got Here], https://wiki.freebsd.org/IRC/Nicknames[IRC Nicks], and https://wiki.freebsd.org/Community/Dogs[Dogs of FreeBSD] pages. . Optional: Update Ports with Personal Information + [.filename]#ports/astro/xearth/files/freebsd.committers.markers# and [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd# - Some people add entries for themselves to these files to show where they are located or the date of their birthday. . Optional: Prevent Duplicate Mailings + Subscribers to {dev-commits-doc-all}, {dev-commits-ports-all} or {dev-commits-src-all} might wish to unsubscribe to avoid receiving duplicate copies of commit messages and followups. ==== [[conventions-everyone]] === For Everyone [[conventions-everyone-steps]] [.procedure] ==== . Introduce yourself to the other developers, otherwise no one will have any idea who you are or what you are working on. The introduction need not be a comprehensive biography, just write a paragraph or two about who you are, what you plan to be working on as a developer in FreeBSD, and who will be your mentor. Email this to the {developers-name} and you will be on your way! . Log into `freefall.FreeBSD.org` and create a [.filename]#/var/forward/user# (where _user_ is your username) file containing the e-mail address where you want mail addressed to _yourusername_@FreeBSD.org to be forwarded. This includes all of the commit messages as well as any other mail addressed to the {committers-name} and the {developers-name}. Really large mailboxes which have taken up permanent residence on `freefall` may get truncated without warning if space needs to be freed, so forward it or save it elsewhere. + [NOTE] ====== If your e-mail system uses SPF with strict rules, you should whitelist `mx2.FreeBSD.org` from SPF checks. ====== + Due to the severe load dealing with SPAM places on the central mail servers that do the mailing list processing, the front-end server does do some basic checks and will drop some messages based on these checks. At the moment proper DNS information for the connecting host is the only check in place but that may change. Some people blame these checks for bouncing valid email. To have these checks turned off for your email, create a file named [.filename]#~/.spam_lover# on `freefall.FreeBSD.org`. + [NOTE] ====== Those who are developers but not committers will not be subscribed to the committers or developers mailing lists. The subscriptions are derived from the access rights. ====== ==== [[smtp-setup]] ==== SMTP Access Setup For those willing to send e-mail messages through the FreeBSD.org infrastructure, follow the instructions below: [.procedure] ==== . Point your mail client at `smtp.FreeBSD.org:587`. . Enable STARTTLS. . Ensure your `From:` address is set to `_yourusername_@FreeBSD.org`. . For authentication, you can use your FreeBSD Kerberos username and password (see <>). The `_yourusername_/mail` principal is preferred, as it is only valid for authenticating to mail resources. + [NOTE] ====== Do not include `@FreeBSD.org` when entering in your username. ====== + .Additional Notes [NOTE] ====== * Will only accept mail from `_yourusername_@FreeBSD.org`. If you are authenticated as one user, you are not permitted to send mail from another. * A header will be appended with the SASL username: (`Authenticated sender: _username_`). * Host has various rate limits in place to cut down on brute force attempts. ====== ==== [[smtp-setup-local-mta]] ===== Using a Local MTA to Forward Emails to the FreeBSD.org SMTP Service It is also possible to use a local MTA to forward locally sent emails to the FreeBSD.org SMTP servers. [[smtp-setup-local-postfix]] .Using Postfix [example] ==== To tell a local Postfix instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#main.cf#: [.programlisting] .... sender_dependent_relayhost_maps = hash:/usr/local/etc/postfix/relayhost_maps smtp_sasl_auth_enable = yes smtp_sasl_security_options = noanonymous smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd smtp_use_tls = yes .... Create [.filename]#/usr/local/etc/postfix/relayhost_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org [smtp.freebsd.org]:587 .... Create [.filename]#/usr/local/etc/postfix/sasl_passwd# with the following content: [.programlisting] .... [smtp.freebsd.org]:587 yourusername:yourpassword .... If the email server is used by other people, you may want to prevent them from sending e-mails from your address. To achieve this, add this to your [.filename]#main.cf#: [.programlisting] .... smtpd_sender_login_maps = hash:/usr/local/etc/postfix/sender_login_maps smtpd_sender_restrictions = reject_known_sender_login_mismatch .... Create [.filename]#/usr/local/etc/postfix/sender_login_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org yourlocalusername .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of Postfix. ==== [[smtp-setup-local-opensmtpd]] .Using OpenSMTPD [example] ==== To tell a local OpenSMTPD instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#smtpd.conf#: [.programlisting] .... action "freebsd" relay host smtp+tls://freebsd@smtp.freebsd.org:587 auth match from any auth yourlocalusername mail-from "_yourusername_@freebsd.org" for any action "freebsd" .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of OpenSMTPD. Create [.filename]#/usr/local/etc/mail/secrets# with the following content: [.programlisting] .... freebsd yourusername:yourpassword .... ==== [[mentors]] === Mentors All new developers have a mentor assigned to them for the first few months. A mentor is responsible for teaching the mentee the rules and conventions of the project and guiding their first steps in the developer community. The mentor is also personally responsible for the mentee's actions during this initial period. For committers: do not commit anything without first getting mentor approval. Document that approval with an `Approved by:` line in the commit message. When the mentor decides that a mentee has learned the ropes and is ready to commit on their own, the mentor announces it with a commit to [.filename]#conf/mentors#. This file is in the [.filename]#svnadmin# branch of each repository: [.informaltable] [cols="1,1", frame="none"] |=== |`src` |[.filename]#base/svnadmin/conf/mentors# |`doc` |[.filename]#doc/svnadmin/conf/mentors# |`ports` |[.filename]#ports/svnadmin/conf/mentors# |=== [[pre-commit-review]] == Pre-Commit Review Code review is one way to increase the quality of software. The following guidelines apply to commits to the `head` (-CURRENT) branch of the `src` repository. Other branches and the `ports` and `docs` trees have their own review policies, but these guidelines generally apply to commits requiring review: * All non-trivial changes should be reviewed before they are committed to the repository. * Reviews may be conducted by email, in Bugzilla, in Phabricator, or by another mechanism. Where possible, reviews should be public. * The developer responsible for a code change is also responsible for making all necessary review-related changes. * Code review can be an iterative process, which continues until the patch is ready to be committed. Specifically, once a patch is sent out for review, it should receive an explicit "looks good" before it is committed. So long as it is explicit, this can take whatever form makes sense for the review method. * Timeouts are not a substitute for review. Sometimes code reviews will take longer than you would hope for, especially for larger features. Accepted ways to speed up review times for your patches are: * Review other people's patches. If you help out, everybody will be more willing to do the same for you; goodwill is our currency. * Ping the patch. If it is urgent, provide reasons why it is important to you to get this patch landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you are asking for valuable time from other professional developers. * Ask for help on mailing lists, IRC, etc. Others may be able to either help you directly, or suggest a reviewer. * Split your patch into multiple smaller patches that build on each other. The smaller your patch, the higher the probability that somebody will take a quick look at it. + When making large changes, it is helpful to keep this in mind from the beginning of the effort as breaking large changes into smaller ones is often difficult after the fact. Developers should participate in code reviews as both reviewers and reviewees. If someone is kind enough to review your code, you should return the favor for someone else. Note that while anyone is welcome to review and give feedback on a patch, only an appropriate subject-matter expert can approve a change. This will usually be a committer who works with the code in question on a regular basis. In some cases, no subject-matter expert may be available. In those cases, a review by an experienced developer is sufficient when coupled with appropriate testing. [[commit-log-message]] == Commit Log Messages This section contains some suggestions and traditions for how commit logs are formatted. === Why are commit messages important? When you commit a change in Git, Subversion, or another version control system (VCS), you're prompted to write some text describing the commit -- a commit message. How important is this commit message? Should you spend some significant effort writing it? Does it really matter if you write simply fixed a bug? Most projects have more than one developer and last for some length of time. Commit messages are a very important method of communicating with other developers, in the present and for the future. FreeBSD has hundreds of active developers and hundreds of thousands of commits spanning decades of history. Over that time the developer community has learned how valuable good commit messages are; sometimes these are hard-learned lessons. Commit messages serve at least three purposes: * Communicating with other developers + FreeBSD commits generate email to various mailing lists. These include the commit message along with a copy of the patch itself. Commit messages are also viewed through commands like git log. These serve to make other developers aware of changes that are ongoing; that other developer may want to test the change, may have an interest in the topic and will want to review in more detail, or may have their own projects underway that would benefit from interaction. * Making Changes Discoverable + In a large project with a long history it may be difficult to find changes of interest when investigating an issue or change in behaviour. Verbose, detailed commit messages allow searches for changes that might be relevant. For example, `git log --since 1year --grep 'USB timeout'`. * Providing historical documentation + Commit messages serve to document changes for future developers, perhaps years or decades later. This future developer may even be you, the original author. A change that seems obvious today may be decidedly not so much later on. The `git blame` command annotates each line of a source file with the change (hash and subject line) that brought it in. Having established the importance, here are elements of a good FreeBSD commit message: === Start with a subject line Commit messages should start with a single-line subject that briefly summarizes the change. The subject should, by itself, allow the reader to quickly determine if the change is of interest or not. === Keep subject lines short The subject line should be as short as possible while still retaining the required information. This is to make browsing Git log more efficient, and so that git log --oneline can display the short hash and subject on a single 80-column line. A good rule of thumb is to stay below 63 characters, and aim for about 50 or fewer if possible. === Prefix the subject line with a component, if applicable If the change relates to a specific component the subject line may be prefixed with that component name and a colon (:). ✓ `foo: Add -k option to keep temporary data` Include the prefix in the 63-character limit suggested above, so that `git log --oneline` avoids wrapping. === Capitalize the first letter of the subject Capitalize the first letter of the subject itself. The prefix, if any, is not capitalized unless necessary (e.g., `USB:` is capitalized). === Do not end the subject line with punctuation Do not end with a period or other punctuation. In this regard the subject line is like a newspaper headline. === Separate the subject and body with a blank line Separate the body from the subject with a blank line. Some trivial commits do not require a body, and will have only a subject. ✓ `ls: Fix typo in usage text` === Limit messages to 72 columns `git log` and `git format-patch` indent the commit message by four spaces. Wrapping at 72 columns provides a matching margin on the right edge. Limiting messages to 72 characters also keeps the commit message in formatted patches below RFC 2822's suggested email line length limit of 78 characters. This limit works well with a variety of tools that may render commit messages; line wrapping might be inconsistent with longer line length. === Use the present tense, imperative mood This facilitates short subject lines and provides consistency, including with automatically generated commit messages (e.g., as generated by git revert). This is important when reading a list of commit subjects. Think of the subject as finishing the sentence "when applied, this change will ...". ✓ `foo: Implement the -k (keep) option` + ✗ `foo: Implemented the -k option` + ✗ `This change implements the -k option in foo` + ✗ `-k option added` === Focus on what and why, not how Explain what the change accomplishes and why it is being done, rather than how. Do not assume that the reader is familiar with the issue. Explain the background and motivation for the change. Include benchmark data if you have it. If there are limitations or incomplete aspects of the change, describe them in the commit message. === Consider whether parts of the commit message could be code comments instead Sometimes while writing a commit message you may find yourself writing a sentence or two explaining some tricky or confusing aspect of the change. When this happens consider whether it would be valuable to have that explanation as a comment in the code itself. === Write commit messages for your future self While writing the commit message for a change you have all of the context in mind - what prompted the change, alternate approaches that were considered and rejected, limitations of the change, and so on. Imagine yourself revisiting the change a year or two in the future, and write the commit message in a way that would provide that necessary context. === Commit messages should stand alone You may include references to mailing list postings, benchmark result web sites, or code review links. However, the commit message should contain all of the relevant information in case these references are no longer available in the future. Similarly, a commit may refer to a previous commit, for example in the case of a bug fix or revert. In addition to the commit identifier (revision or hash), include the subject line from the referenced commit (or another suitable brief reference). With each VCS migration (from CVS to Subversion to Git) revision identifiers from previous systems may become difficult to follow. === Include appropriate metadata in a footer As well as including an informative message with each commit, some additional information may be needed. This information consists of one or more lines containing the key word or phrase, a colon, tabs for formatting, and then the additional information. The key words or phrases are: [.informaltable] [cols="20%,80%", frame="none"] |=== |`PR:` |The problem report (if any) which is affected (typically, by being closed) by this commit. Multiple PRs may be specified on one line, separated by commas or spaces. |`Reported by:` |The name and e-mail address of the person that reported the issue; for developers, just the username on the FreeBSD cluster. Typically used when there is no PR, for example if the issue was reported on a mailing list. |`Submitted by:` |The name and e-mail address of the person that submitted the fix; for developers, just the username on the FreeBSD cluster. Typically not used with Git; submitted patches should have the author set by using `git commit --author`. If the submitter is the maintainer of the port being committed, include "(maintainer)" after the email address. Avoid obfuscating the email address of the submitter as this adds additional work when searching logs. |`Reviewed by:` |The name and e-mail address of the person or people that reviewed the change; for developers, just the username on the FreeBSD cluster. If a patch was submitted to a mailing list for review, and the review was favorable, then just include the list name. |`Tested by:` |The name and e-mail address of the person or people that tested the change; for developers, just the username on the FreeBSD cluster. |`Approved by:` a| The name and e-mail address of the person or people that approved the change; for developers, just the username on the FreeBSD cluster. There are several cases where approval is customary: * while a new committer is under mentorship * commits to an area of the tree to which you do not usually commit * during a release cycle * committing to a repo where you do not hold a commit bit (e.g. src committer committing to docs) While under mentorship, get mentor approval before the commit. Enter the mentor's username in this field, and note that they are a mentor: [source,shell] .... Approved by: username-of-mentor (mentor) .... If a team approved these commits then include the team name followed by the username of the approver in parentheses. For example: [source,shell] .... Approved by: re (username) .... |`Obtained from:` |The name of the project (if any) from which the code was obtained. Do not use this line for the name of an individual person. |`Fixes:` |The Git short hash and the title line of a commit that is fixed by this change as returned by `git log -n 1 --oneline GIT-COMMIT-HASH`. |`MFC after:` |To receive an e-mail reminder to MFC at a later date, specify the number of days, weeks, or months after which an MFC is planned. |`MFC to:` |If the commit should be merged to a subset of stable branches, specify the branch names. |`MFC with:` |If the commit should be merged together with a previous one in a single MFC commit (for example, where this commit corrects a bug in the previous change), specify the corresponding Git hash. |`MFH:` |If the commit is to be merged into a ports quarterly branch name, specify the quarterly branch. For example `2021Q2`. |`Relnotes:` |If the change is a candidate for inclusion in the release notes for the next release from the branch, set to `yes`. |`Security:` |If the change is related to a security vulnerability or security exposure, include one or more references or a description of the issue. If possible, include a VuXML URL or a CVE ID. |`Event:` |The description for the event where this commit was made. If this is a recurring event, add the year or even the month to it. For example, this could be `FooBSDcon 2019`. The idea behind this line is to put recognition to conferences, gatherings, and other types of meetups and to show that these are useful to have. Please do not use the `Sponsored by:` line for this as that is meant for organizations sponsoring certain features or developers working on them. |`Sponsored by:` |Sponsoring organizations for this change, if any. Separate multiple organizations with commas. If only a portion of the work was sponsored, or different amounts of sponsorship were provided to different authors, please give appropriate credit in parentheses after each sponsor name. For example, `Example.com (alice, code refactoring), Wormulon (bob), Momcorp (cindy)` shows that Alice was sponsored by Example.com to do code refactoring, while Wormulon sponsored Bob's work and Momcorp sponsored Cindy's work. Other authors were either not sponsored or chose not to list sponsorship. |`Differential Revision:` |The full URL of the Phabricator review. This line __must be the last line__. For example: `https://reviews.freebsd.org/D1708`. |`Signed-off-by:` |ID certifies compliance with https://developercertificate.org/ |=== .Commit Log for a Commit Based on a PR [example] ==== The commit is based on a patch from a PR submitted by John Smith. The commit message "PR" field is filled. [.programlisting] .... ... PR: 12345 .... The committer sets the author of the patch with `git commit --author "John Smith "`. ==== .Commit Log for a Commit Needing Review [example] ==== The virtual memory system is being changed. After posting patches to the appropriate mailing list (in this case, `freebsd-arch`) and the changes have been approved. [.programlisting] .... ... Reviewed by: -arch .... ==== .Commit Log for a Commit Needing Approval [example] ==== Commit a port, after working with the listed MAINTAINER, who said to go ahead and commit. [.programlisting] .... ... Approved by: abc (maintainer) .... Where _abc_ is the account name of the person who approved. ==== .Commit Log for a Commit Bringing in Code from OpenBSD [example] ==== Committing some code based on work done in the OpenBSD project. [.programlisting] .... ... Obtained from: OpenBSD .... ==== .Commit Log for a Change to FreeBSD-CURRENT with a Planned Commit to FreeBSD-STABLE to Follow at a Later Date. [example] ==== Committing some code which will be merged from FreeBSD-CURRENT into the FreeBSD-STABLE branch after two weeks. [.programlisting] .... ... MFC after: 2 weeks .... Where _2_ is the number of days, weeks, or months after which an MFC is planned. The _weeks_ option may be `day`, `days`, `week`, `weeks`, `month`, `months`. ==== It is often necessary to combine these. Consider the situation where a user has submitted a PR containing code from the NetBSD project. Looking at the PR, the developer sees it is not an area of the tree they normally work in, so they have the change reviewed by the `arch` mailing list. Since the change is complex, the developer opts to MFC after one month to allow adequate testing. The extra information to include in the commit would look something like .Example Combined Commit Log [example] ==== [.programlisting] .... PR: 54321 Reviewed by: -arch Obtained from: NetBSD MFC after: 1 month Relnotes: yes .... ==== [[pref-license]] == Preferred License for New Files The FreeBSD Project's full license policy can be found at link:https://www.FreeBSD.org/internal/software-license/[https://www.FreeBSD.org/internal/software-license]. The rest of this section is intended to help you get started. As a rule, when in doubt, ask. It is much easier to give advice than to fix the source tree. The FreeBSD Project suggests and uses this text as the preferred license scheme: [.programlisting] .... /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) [year] [your name] * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * [id for your version control system, if any] */ .... The FreeBSD project strongly discourages the so-called "advertising clause" in new code. Due to the large number of contributors to the FreeBSD project, complying with this clause for many commercial vendors has become difficult. If you have code in the tree with the advertising clause, please consider removing it. In fact, please consider using the above license for your code. The FreeBSD project discourages completely new licenses and variations on the standard licenses. New licenses require the approval of the {core-email} to reside in the main repository. The more different licenses that are used in the tree, the more problems that this causes to those wishing to utilize this code, typically from unintended consequences from a poorly worded license. Project policy dictates that code under some non-BSD licenses must be placed only in specific sections of the repository, and in some cases, compilation must be conditional or even disabled by default. For example, the GENERIC kernel must be compiled under only licenses identical to or substantially similar to the BSD license. GPL, APSL, CDDL, etc, licensed software must not be compiled into GENERIC. Developers are reminded that in open source, getting "open" right is just as important as getting "source" right, as improper handling of intellectual property has serious consequences. Any questions or concerns should immediately be brought to the attention of the core team. [[tracking.license.grants]] == Keeping Track of Licenses Granted to the FreeBSD Project Various software or data exist in the repositories where the FreeBSD project has been granted a special licence to be able to use them. A case in point are the Terminus fonts for use with man:vt[4]. Here the author Dimitar Zhekov has allowed us to use the "Terminus BSD Console" font under a 2-clause BSD license rather than the regular Open Font License he normally uses. It is clearly sensible to keep a record of any such license grants. To that end, the {core-email} has decided to keep an archive of them. Whenever the FreeBSD project is granted a special license we require the {core-email} to be notified. Any developers involved in arranging such a license grant, please send details to the {core-email} including: * Contact details for people or organizations granting the special license. * What files, directories etc. in the repositories are covered by the license grant including the revision numbers where any specially licensed material was committed. * The date the license comes into effect from. Unless otherwise agreed, this will be the date the license was issued by the authors of the software in question. * The license text. * A note of any restrictions, limitations or exceptions that apply specifically to FreeBSD's usage of the licensed material. * Any other relevant information. Once the {core-email} is satisfied that all the necessary details have been gathered and are correct, the secretary will send a PGP-signed acknowledgement of receipt including the license details. This receipt will be persistently archived and serve as our permanent record of the license grant. The license archive should contain only details of license grants; this is not the place for any discussions around licensing or other subjects. Access to data within the license archive will be available on request to the {core-email}. [[spdx.tags]] == SPDX Tags in the tree The project uses https://spdx.dev[SPDX] tags in our source base. At present, these tags are indented to help automated tools reconstruct license requirements mechanically. All _SPDX-License-Identifier_ tags in the tree should be considered to be informative. All files in the FreeBSD source tree with these tags also have a copy of the license which governs use of that file. In the event of a discrepency, the verbatim license is controlling. The project tries to follow the https://spdx.github.io/spdx-spec/[SPDX Specification, Version 2.2]. How to mark source files and valid algebraic expressions are found in https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/[Appendix IV] and https://spdx.github.io/spdx-spec/appendix-V-using-SPDX-short-identifiers-in-source-files/[Appendix V]. The project draws identifiers from SPDX's list of valid https://spdx.org/licenses/[short license identifiers]. The project uses only the _SPDX-License-Identifier_ tag. As of March 2021, approximately 25,000 out of 90,000 files in the tree have been marked. [[developer.relations]] == Developer Relations When working directly on your own code or on code which is already well established as your responsibility, then there is probably little need to check with other committers before jumping in with a commit. Working on a bug in an area of the system which is clearly orphaned (and there are a few such areas, to our shame), the same applies. When modifying parts of the system which are maintained, formally, or informally, consider asking for review just as a developer would have before becoming a committer. For ports, contact the listed `MAINTAINER` in the [.filename]#Makefile#. To determine if an area of the tree is maintained, check the MAINTAINERS file at the root of the tree. If nobody is listed, scan the revision history to see who has committed changes in the past. An example script that lists each person who has committed to a given file along with the number of commits each person has made can be found at on `freefall` at [.filename]#~eadler/bin/whodid#. If queries go unanswered or the committer otherwise indicates a lack of interest in the area affected, go ahead and commit it. [IMPORTANT] ==== Avoid sending private emails to maintainers. Other people might be interested in the conversation, not just the final output. ==== If there is any doubt about a commit for any reason at all, have it reviewed before committing. Better to have it flamed then and there rather than when it is part of the repository. If a commit does results in controversy erupting, it may be advisable to consider backing the change out again until the matter is settled. Remember, with a version control system we can always change it back. Do not impugn the intentions of others. If they see a different solution to a problem, or even a different problem, it is probably not because they are stupid, because they have questionable parentage, or because they are trying to destroy hard work, personal image, or FreeBSD, but basically because they have a different outlook on the world. Different is good. Disagree honestly. Argue your position from its merits, be honest about any shortcomings it may have, and be open to seeing their solution, or even their vision of the problem, with an open mind. Accept correction. We are all fallible. When you have made a mistake, apologize and get on with life. Do not beat up yourself, and certainly do not beat up others for your mistake. Do not waste time on embarrassment or recrimination, just fix the problem and move on. Ask for help. Seek out (and give) peer reviews. One of the ways open source software is supposed to excel is in the number of eyeballs applied to it; this does not apply if nobody will review code. [[if-in-doubt]] == If in Doubt... When unsure about something, whether it be a technical issue or a project convention be sure to ask. If you stay silent you will never make progress. If it relates to a technical issue ask on the public mailing lists. Avoid the temptation to email the individual person that knows the answer. This way everyone will be able to learn from the question and the answer. For project specific or administrative questions ask, in order: * Your mentor or former mentor. * An experienced committer on IRC, email, etc. * Any team with a "hat", as they can give you a definitive answer. * If still not sure, ask on {developers-name}. Once your question is answered, if no one pointed you to documentation that spelled out the answer to your question, document it, as others will have the same question. [[bugzilla]] == Bugzilla The FreeBSD Project utilizes Bugzilla for tracking bugs and change requests. Be sure that if you commit a fix or suggestion found in the PR database to close it. It is also considered nice if you take time to close any PRs associated with your commits, if appropriate. Committers with non-``FreeBSD.org`` Bugzilla accounts can have the old account merged with the `FreeBSD.org` account by following these steps: [.procedure] ==== . Log in using your old account. . Open new bug. Choose `Services` as the Product, and `Bug Tracker` as the Component. In bug description list accounts you wish to be merged. . Log in using `FreeBSD.org` account and post comment to newly opened bug to confirm ownership. See <> for more details on how to generate or set a password for your `FreeBSD.org` account. . If there are more than two accounts to merge, post comments from each of them. ==== You can find out more about Bugzilla at: * link:{pr-guidelines}[FreeBSD Problem Report Handling Guidelines] * link:https://www.FreeBSD.org/support/[https://www.FreeBSD.org/support] [[phabricator]] == Phabricator The FreeBSD Project utilizes https://reviews.freebsd.org[Phabricator] for code review requests. See the https://wiki.freebsd.org/CodeReview[CodeReview] wiki page for details. Committers with non-``FreeBSD.org`` Phabricator accounts can have the old account renamed to the ``FreeBSD.org`` account by following these steps: [.procedure] ==== . Change your Phabricator account email to your `FreeBSD.org` email. . Open new bug on our bug tracker using your `FreeBSD.org` account, see <> for more information. Choose `Services` as the Product, and `Code Review` as the Component. In bug description request that your Phabricator account be renamed, and provide a link to your Phabricator user. For example, `https://reviews.freebsd.org/p/bob_example.com/` ==== [IMPORTANT] ==== Phabricator accounts cannot be merged, please do not open a new account. ==== [[people]] == Who's Who Besides the repository meisters, there are other FreeBSD project members and teams whom you will probably get to know in your role as a committer. Briefly, and by no means all-inclusively, these are: `{doceng}`:: doceng is the group responsible for the documentation build infrastructure, approving new documentation committers, and ensuring that the FreeBSD website and documentation on the FTP site is up to date with respect to the Subversion tree. It is not a conflict resolution body. The vast majority of documentation related discussion takes place on the {freebsd-doc}. More details regarding the doceng team can be found in its https://www.FreeBSD.org/internal/doceng/[charter]. Committers interested in contributing to the documentation should familiarize themselves with the link:{fdp-primer}[Documentation Project Primer]. `{re-members}`:: These are the members of the `{re}`. This team is responsible for setting release deadlines and controlling the release process. During code freezes, the release engineers have final authority on all changes to the system for whichever branch is pending release status. If there is something you want merged from FreeBSD-CURRENT to FreeBSD-STABLE (whatever values those may have at any given time), these are the people to talk to about it. `{so}`:: `{so-name}` is the link:https://www.FreeBSD.org/security/[FreeBSD Security Officer] and oversees the `{security-officer}`. `{wollman}`:: If you need advice on obscure network internals or are not sure of some potential change to the networking subsystem you have in mind, Garrett is someone to talk to. Garrett is also very knowledgeable on the various standards applicable to FreeBSD. {committers-name}:: {svn-src-all}, {svn-ports-all} and {svn-doc-all} are the mailing lists that the version control system uses to send commit messages to. _Never_ send email directly to these lists. Only send replies to this list when they are short and are directly related to a commit. {developers-name}:: All committers are subscribed to -developers. This list was created to be a forum for the committers "community" issues. Examples are Core voting, announcements, etc. + The {developers-name} is for the exclusive use of FreeBSD committers. To develop FreeBSD, committers must have the ability to openly discuss matters that will be resolved before they are publicly announced. Frank discussions of work in progress are not suitable for open publication and may harm FreeBSD. + All FreeBSD committers are expected not to not publish or forward messages from the {developers-name} outside the list membership without permission of all of the authors. Violators will be removed from the {developers-name}, resulting in a suspension of commit privileges. Repeated or flagrant violations may result in permanent revocation of commit privileges. + This list is _not_ intended as a place for code reviews or for any technical discussion. In fact using it as such hurts the FreeBSD Project as it gives a sense of a closed list where general decisions affecting all of the FreeBSD using community are made without being "open". Last, but not least __never, never ever, email the {developers-name} and CC:/BCC: another FreeBSD list__. Never, ever email another FreeBSD email list and CC:/BCC: the {developers-name}. Doing so can greatly diminish the benefits of this list. [[ssh.guide]] == SSH Quick-Start Guide [.procedure] ==== . If you do not wish to type your password in every time you use man:ssh[1], and you use keys to authenticate, man:ssh-agent[1] is there for your convenience. If you want to use man:ssh-agent[1], make sure that you run it before running other applications. X users, for example, usually do this from their [.filename]#.xsession# or [.filename]#.xinitrc#. See man:ssh-agent[1] for details. . Generate a key pair using man:ssh-keygen[1]. The key pair will wind up in your [.filename]#$HOME/.ssh/# directory. + [IMPORTANT] ====== Only ECDSA, Ed25519 or RSA keys are supported. ====== . Send your public key ([.filename]#$HOME/.ssh/id_ecdsa.pub#, [.filename]#$HOME/.ssh/id_ed25519.pub#, or [.filename]#$HOME/.ssh/id_rsa.pub#) to the person setting you up as a committer so it can be put into [.filename]#yourlogin# in [.filename]#/etc/ssh-keys/# on `freefall`. ==== Now man:ssh-add[1] can be used for authentication once per session. It prompts for the private key's pass phrase, and then stores it in the authentication agent (man:ssh-agent[1]). Use `ssh-add -d` to remove keys stored in the agent. Test with a simple remote command: `ssh freefall.FreeBSD.org ls /usr`. For more information, see package:security/openssh-portable[], man:ssh[1], man:ssh-add[1], man:ssh-agent[1], man:ssh-keygen[1], and man:scp[1]. For information on adding, changing, or removing man:ssh[1] keys, see https://wiki.freebsd.org/clusteradm/ssh-keys[this article]. [[coverity]] == Coverity(R) Availability for FreeBSD Committers All FreeBSD developers can obtain access to Coverity analysis results of all FreeBSD Project software. All who are interested in obtaining access to the analysis results of the automated Coverity runs, can sign up at http://scan.coverity.com/[Coverity Scan]. The FreeBSD wiki includes a mini-guide for developers who are interested in working with the Coverity(R) analysis reports: https://wiki.freebsd.org/CoverityPrevent[https://wiki.freebsd.org/CoverityPrevent]. Please note that this mini-guide is only readable by FreeBSD developers, so if you cannot access this page, you will have to ask someone to add you to the appropriate Wiki access list. Finally, all FreeBSD developers who are going to use Coverity(R) are always encouraged to ask for more details and usage information, by posting any questions to the mailing list of the FreeBSD developers. [[rules]] == The FreeBSD Committers' Big List of Rules Everyone involved with the FreeBSD project is expected to abide by the _Code of Conduct_ available from link:https://www.FreeBSD.org/internal/code-of-conduct/[https://www.FreeBSD.org/internal/code-of-conduct]. As committers, you form the public face of the project, and how you behave has a vital impact on the public perception of it. This guide expands on the parts of the _Code of Conduct_ specific to committers. . Respect other committers. . Respect other contributors. . Discuss any significant change _before_ committing. . Respect existing maintainers (if listed in the `MAINTAINER` field in [.filename]#Makefile# or in [.filename]#MAINTAINER# in the top-level directory). . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined for the maintainer in rule #5. . Do not fight in public with other committers; it looks bad. . Respect all code freezes and read the `committers` and `developers` mailing lists in a timely manner so you know when a code freeze is in effect. . When in doubt on any procedure, ask first! . Test your changes before committing them. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. As noted, breaking some of these rules can be grounds for suspension or, upon repeated offense, permanent removal of commit privileges. Individual members of core have the power to temporarily suspend commit privileges until core as a whole has the chance to review the issue. In case of an "emergency" (a committer doing damage to the repository), a temporary suspension may also be done by the repository meisters. Only a 2/3 majority of core has the authority to suspend commit privileges for longer than a week or to remove them permanently. This rule does not exist to set core up as a bunch of cruel dictators who can dispose of committers as casually as empty soda cans, but to give the project a kind of safety fuse. If someone is out of control, it is important to be able to deal with this immediately rather than be paralyzed by debate. In all cases, a committer whose privileges are suspended or revoked is entitled to a "hearing" by core, the total duration of the suspension being determined at that time. A committer whose privileges are suspended may also request a review of the decision after 30 days and every 30 days thereafter (unless the total suspension period is less than 30 days). A committer whose privileges have been revoked entirely may request a review after a period of 6 months has elapsed. This review policy is _strictly informal_ and, in all cases, core reserves the right to either act on or disregard requests for review if they feel their original decision to be the right one. In all other aspects of project operation, core is a subset of committers and is bound by the __same rules__. Just because someone is in core this does not mean that they have special dispensation to step outside any of the lines painted here; core's "special powers" only kick in when it acts as a group, not on an individual basis. As individuals, the core team members are all committers first and core second. === Details [[respect]] . Respect other committers. + This means that you need to treat other committers as the peer-group developers that they are. Despite our occasional attempts to prove the contrary, one does not get to be a committer by being stupid and nothing rankles more than being treated that way by one of your peers. Whether we always feel respect for one another or not (and everyone has off days), we still have to _treat_ other committers with respect at all times, on public forums and in private email. + Being able to work together long term is this project's greatest asset, one far more important than any set of changes to the code, and turning arguments about code into issues that affect our long-term ability to work harmoniously together is just not worth the trade-off by any conceivable stretch of the imagination. + To comply with this rule, do not send email when you are angry or otherwise behave in a manner which is likely to strike others as needlessly confrontational. First calm down, then think about how to communicate in the most effective fashion for convincing the other persons that your side of the argument is correct, do not just blow off some steam so you can feel better in the short term at the cost of a long-term flame war. Not only is this very bad "energy economics", but repeated displays of public aggression which impair our ability to work well together will be dealt with severely by the project leadership and may result in suspension or termination of your commit privileges. The project leadership will take into account both public and private communications brought before it. It will not seek the disclosure of private communications, but it will take it into account if it is volunteered by the committers involved in the complaint. + All of this is never an option which the project's leadership enjoys in the slightest, but unity comes first. No amount of code or good advice is worth trading that away. . Respect other contributors. + You were not always a committer. At one time you were a contributor. Remember that at all times. Remember what it was like trying to get help and attention. Do not forget that your work as a contributor was very important to you. Remember what it was like. Do not discourage, belittle, or demean contributors. Treat them with respect. They are our committers in waiting. They are every bit as important to the project as committers. Their contributions are as valid and as important as your own. After all, you made many contributions before you became a committer. Always remember that. + Consider the points raised under <> and apply them also to contributors. . Discuss any significant change _before_ committing. + The repository is not where changes are initially submitted for correctness or argued over, that happens first in the mailing lists or by use of the Phabricator service. The commit will only happen once something resembling consensus has been reached. This does not mean that permission is required before correcting every obvious syntax error or manual page misspelling, just that it is good to develop a feel for when a proposed change is not quite such a no-brainer and requires some feedback first. People really do not mind sweeping changes if the result is something clearly better than what they had before, they just do not like being _surprised_ by those changes. The very best way of making sure that things are on the right track is to have code reviewed by one or more other committers. + When in doubt, ask for review! . Respect existing maintainers if listed. + Many parts of FreeBSD are not "owned" in the sense that any specific individual will jump up and yell if you commit a change to "their" area, but it still pays to check first. One convention we use is to put a maintainer line in the [.filename]#Makefile# for any package or subtree which is being actively maintained by one or more people; see link:{developers-handbook}#policies[Source Tree Guidelines and Policies] for documentation on this. Where sections of code have several maintainers, commits to affected areas by one maintainer need to be reviewed by at least one other maintainer. In cases where the "maintainer-ship" of something is not clear, look at the repository logs for the files in question and see if someone has been working recently or predominantly in that area. . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. + This may be hard to swallow in times of conflict (when each side is convinced that they are in the right, of course) but a version control system makes it unnecessary to have an ongoing dispute raging when it is far easier to simply reverse the disputed change, get everyone calmed down again and then try to figure out what is the best way to proceed. If the change turns out to be the best thing after all, it can be easily brought back. If it turns out not to be, then the users did not have to live with the bogus change in the tree while everyone was busily debating its merits. People _very_ rarely call for back-outs in the repository since discussion generally exposes bad or controversial changes before the commit even happens, but on such rare occasions the back-out should be done without argument so that we can get immediately on to the topic of figuring out whether it was bogus or not. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined in rule #5. + This is another "do not argue about it" issue since it is the release engineer who is ultimately responsible (and gets beaten up) if a change turns out to be bad. Please respect this and give the release engineer your full cooperation when it comes to the FreeBSD-STABLE branch. The management of FreeBSD-STABLE may frequently seem to be overly conservative to the casual observer, but also bear in mind the fact that conservatism is supposed to be the hallmark of FreeBSD-STABLE and different rules apply there than in FreeBSD-CURRENT. There is also really no point in having FreeBSD-CURRENT be a testing ground if changes are merged over to FreeBSD-STABLE immediately. Changes need a chance to be tested by the FreeBSD-CURRENT developers, so allow some time to elapse before merging unless the FreeBSD-STABLE fix is critical, time sensitive or so obvious as to make further testing unnecessary (spelling fixes to manual pages, obvious bug/typo fixes, etc.) In other words, apply common sense. + Changes to the security branches (for example, `releng/9.3`) must be approved by a member of the `{security-officer}`, or in some cases, by a member of the `{re}`. . Do not fight in public with other committers; it looks bad. + This project has a public image to uphold and that image is very important to all of us, especially if we are to continue to attract new members. There will be occasions when, despite everyone's very best attempts at self-control, tempers are lost and angry words are exchanged. The best thing that can be done in such cases is to minimize the effects of this until everyone has cooled back down. Do not air angry words in public and do not forward private correspondence or other private communications to public mailing lists, mail aliases, instant messaging channels or social media sites. What people say one-to-one is often much less sugar-coated than what they would say in public, and such communications therefore have no place there - they only serve to inflame an already bad situation. If the person sending a flame-o-gram at least had the grace to send it privately, then have the grace to keep it private yourself. If you feel you are being unfairly treated by another developer, and it is causing you anguish, bring the matter up with core rather than taking it public. Core will do its best to play peace makers and get things back to sanity. In cases where the dispute involves a change to the codebase and the participants do not appear to be reaching an amicable agreement, core may appoint a mutually-agreeable third party to resolve the dispute. All parties involved must then agree to be bound by the decision reached by this third party. . Respect all code freezes and read the `committers` and `developers` mailing list on a timely basis so you know when a code freeze is in effect. + Committing unapproved changes during a code freeze is a really big mistake and committers are expected to keep up-to-date on what is going on before jumping in after a long absence and committing 10 megabytes worth of accumulated stuff. People who abuse this on a regular basis will have their commit privileges suspended until they get back from the FreeBSD Happy Reeducation Camp we run in Greenland. . When in doubt on any procedure, ask first! + Many mistakes are made because someone is in a hurry and just assumes they know the right way of doing something. If you have not done it before, chances are good that you do not actually know the way we do things and really need to ask first or you are going to completely embarrass yourself in public. There is no shame in asking "how in the heck do I do this?" We already know you are an intelligent person; otherwise, you would not be a committer. . Test your changes before committing them. + This may sound obvious, but if it really were so obvious then we probably would not see so many cases of people clearly not doing this. If your changes are to the kernel, make sure you can still compile both GENERIC and LINT. If your changes are anywhere else, make sure you can still make world. If your changes are to a branch, make sure your testing occurs with a machine which is running that code. If you have a change which also may break another architecture, be sure and test on all supported architectures. Please refer to the https://www.FreeBSD.org/internal/[FreeBSD Internal Page] for a list of available resources. As other architectures are added to the FreeBSD supported platforms list, the appropriate shared testing resources will be made available. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. + Contributed software is anything under the [.filename]#src/contrib#, [.filename]#src/crypto#, or [.filename]#src/sys/contrib# trees. + The trees mentioned above are for contributed software usually imported onto a vendor branch. Committing something there may cause unnecessary headaches when importing newer versions of the software. As a general consider sending patches upstream to the vendor. Patches may be committed to FreeBSD first with permission of the maintainer. + Reasons for modifying upstream software range from wanting strict control over a tightly coupled dependency to lack of portability in the canonical repository's distribution of their code. Regardless of the reason, effort to minimize the maintenance burden of fork is helpful to fellow maintainers. Avoid committing trivial or cosmetic changes to files since it makes every merge thereafter more difficult: such patches need to be manually re-verified every import. + If a particular piece of software lacks a maintainer, you are encouraged to take up ownership. If you are unsure of the current maintainership email {freebsd-arch} and ask. === Policy on Multiple Architectures FreeBSD has added several new architecture ports during recent release cycles and is truly no longer an i386(TM) centric operating system. In an effort to make it easier to keep FreeBSD portable across the platforms we support, core has developed this mandate: [.blockquote] Our 32-bit reference platform is i386, and our 64-bit reference platform is amd64. Major design work (including major API and ABI changes) must prove itself on at least one 32-bit and at least one 64-bit platform, preferably the primary reference platforms, before it may be committed to the source tree. The i386 and amd64 platforms were chosen due to being more readily available to developers and as representatives of more diverse processor and system designs - big versus little endian, register file versus register stack, different DMA and cache implementations, hardware page tables versus software TLB management etc. We will continue to re-evaluate this policy as cost and availability of the 64-bit platforms change. Developers should also be aware of our Tier Policy for the long term support of hardware architectures. The rules here are intended to provide guidance during the development process, and are distinct from the requirements for features and architectures listed in that section. The Tier rules for feature support on architectures at release-time are more strict than the rules for changes during the development process. === Other Suggestions When committing documentation changes, use a spell checker before committing. For all XML docs, verify that the formatting directives are correct by running `make lint` and package:textproc/igor[]. For manual pages, run package:sysutils/manck[] and package:textproc/igor[] over the manual page to verify all of the cross references and file references are correct and that the man page has all of the appropriate `MLINKS` installed. Do not mix style fixes with new functionality. A style fix is any change which does not modify the functionality of the code. Mixing the changes obfuscates the functionality change when asking for differences between revisions, which can hide any new bugs. Do not include whitespace changes with content changes in commits to [.filename]#doc/#. The extra clutter in the diffs makes the translators' job much more difficult. Instead, make any style or whitespace changes in separate commits that are clearly labeled as such in the commit message. === Deprecating Features When it is necessary to remove functionality from software in the base system, follow these guidelines whenever possible: . Mention is made in the manual page and possibly the release notes that the option, utility, or interface is deprecated. Use of the deprecated feature generates a warning. . The option, utility, or interface is preserved until the next major (point zero) release. . The option, utility, or interface is removed and no longer documented. It is now obsolete. It is also generally a good idea to note its removal in the release notes. === Privacy and Confidentiality . Most FreeBSD business is done in public. + FreeBSD is an _open_ project. Which means that not only can anyone use the source code, but that most of the development process is open to public scrutiny. . Certain sensitive matters must remain private or held under embargo. + There unfortunately cannot be complete transparency. As a FreeBSD developer you will have a certain degree of privileged access to information. Consequently you are expected to respect certain requirements for confidentiality. Sometimes the need for confidentiality comes from external collaborators or has a specific time limit. Mostly though, it is a matter of not releasing private communications. . The Security Officer has sole control over the release of security advisories. + Where there are security problems that affect many different operating systems, FreeBSD frequently depends on early access to be able to prepare advisories for coordinated release. Unless FreeBSD developers can be trusted to maintain security, such early access will not be made available. The Security Officer is responsible for controlling pre-release access to information about vulnerabilities, and for timing the release of all advisories. He may request help under condition of confidentiality from any developer with relevant knowledge to prepare security fixes. . Communications with Core are kept confidential for as long as necessary. + Communications to core will initially be treated as confidential. Eventually however, most of Core's business will be summarized into the monthly or quarterly core reports. Care will be taken to avoid publicising any sensitive details. Records of some particularly sensitive subjects may not be reported on at all and will be retained only in Core's private archives. . Non-disclosure Agreements may be required for access to certain commercially sensitive data. + Access to certain commercially sensitive data may only be available under a Non-Disclosure Agreement. The FreeBSD Foundation legal staff must be consulted before any binding agreements are entered into. . Private communications must not be made public without permission. + Beyond the specific requirements above there is a general expectation not to publish private communications between developers without the consent of all parties involved. Ask permission before forwarding a message onto a public mailing list, or posting it to a forum or website that can be accessed by other than the original correspondents. . Communications on project-only or restricted access channels must be kept private. + Similarly to personal communications, certain internal communications channels, including FreeBSD Committer only mailing lists and restricted access IRC channels are considered private communications. Permission is required to publish material from these sources. . Core may approve publication. + Where it is impractical to obtain permission due to the number of correspondents or where permission to publish is unreasonably withheld, Core may approve release of such private matters that merit more general publication. [[archs]] == Support for Multiple Architectures FreeBSD is a highly portable operating system intended to function on many different types of hardware architectures. Maintaining clean separation of Machine Dependent (MD) and Machine Independent (MI) code, as well as minimizing MD code, is an important part of our strategy to remain agile with regards to current hardware trends. Each new hardware architecture supported by FreeBSD adds substantially to the cost of code maintenance, toolchain support, and release engineering. It also dramatically increases the cost of effective testing of kernel changes. As such, there is strong motivation to differentiate between classes of support for various architectures while remaining strong in a few key architectures that are seen as the FreeBSD "target audience". === Statement of General Intent The FreeBSD Project targets "production quality commercial off-the-shelf (COTS) workstation, server, and high-end embedded systems". By retaining a focus on a narrow set of architectures of interest in these environments, the FreeBSD Project is able to maintain high levels of quality, stability, and performance, as well as minimize the load on various support teams on the project, such as the ports team, documentation team, security officer, and release engineering teams. Diversity in hardware support broadens the options for FreeBSD consumers by offering new features and usage opportunities, but these benefits must always be carefully considered in terms of the real-world maintenance cost associated with additional platform support. The FreeBSD Project differentiates platform targets into four tiers. Each tier includes a list of guarantees consumers may rely on as well as obligations by the Project and developers to fulfill those guarantees. These lists define the minimum guarantees for each tier. The Project and developers may provide additional levels of support beyond the minimum guarantees for a given tier, but such additional support is not guaranteed. Each platform target is assigned to a specific tier for each stable branch. As a result, a platform target might be assigned to different tiers on concurrent stable branches. === Platform Targets Support for a hardware platform consists of two components: kernel support and userland Application Binary Interfaces (ABIs). Kernel platform support includes things needed to run a FreeBSD kernel on a hardware platform such as machine-dependent virtual memory management and device drivers. A userland ABI specifies an interface for user processes to interact with a FreeBSD kernel and base system libraries. A userland ABI includes system call interfaces, the layout and semantics of public data structures, and the layout and semantics of arguments passed to subroutines. Some components of an ABI may be defined by specifications such as the layout of C++ exception objects or calling conventions for C functions. A FreeBSD kernel also uses an ABI (sometimes referred to as the Kernel Binary Interface (KBI)) which includes the semantics and layouts of public data structures and the layout and semantics of arguments to public functions within the kernel itself. A FreeBSD kernel may support multiple userland ABIs. For example, FreeBSD's amd64 kernel supports FreeBSD amd64 and i386 userland ABIs as well as Linux x86_64 and i386 userland ABIs. A FreeBSD kernel should support a "native" ABI as the default ABI. The native "ABI" generally shares certain properties with the kernel ABI such as the C calling convention, sizes of basic types, etc. Tiers are defined for both kernels and userland ABIs. In the common case, a platform's kernel and FreeBSD ABIs are assigned to the same tier. === Tier 1: Fully-Supported Architectures Tier 1 platforms are the most mature FreeBSD platforms. They are supported by the security officer, release engineering, and port management teams. Tier 1 architectures are expected to be Production Quality with respect to all aspects of the FreeBSD operating system, including installation and development environments. The FreeBSD Project provides the following guarantees to consumers of Tier 1 platforms: * Official FreeBSD release images will be provided by the release engineering team. * Binary updates and source patches for Security Advisories and Errata Notices will be provided for supported releases. * Source patches for Security Advisories will be provided for supported branches. * Binary updates and source patches for cross-platform Security Advisories will typically be provided at the time of the announcement. * Changes to userland ABIs will generally include compatibility shims to ensure correct operation of binaries compiled against any stable branch where the platform is Tier 1. These shims might not be enabled in the default install. If compatibility shims are not provided for an ABI change, the lack of shims will be clearly documented in the release notes. * Changes to certain portions of the kernel ABI will include compatibility shims to ensure correct operation of kernel modules compiled against the oldest supported release on the branch. Note that not all parts of the kernel ABI are protected. * Official binary packages for third party software will be provided by the ports team. For embedded architectures, these packages may be cross-built from a different architecture. * Most relevant ports should either build or have the appropriate filters to prevent inappropriate ones from building. * New features which are not inherently platform-specific will be fully functional on all Tier 1 architectures. * Features and compatibility shims used by binaries compiled against older stable branches may be removed in newer major versions. Such removals will be clearly documented in the release notes. * Tier 1 platforms should be fully documented. Basic operations will be documented in the FreeBSD Handbook. * Tier 1 platforms will be included in the source tree. * Tier 1 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. To maintain maturity of Tier 1 platforms, the FreeBSD Project will maintain the following resources to support development: * Build and test automation support either in the FreeBSD.org cluster or some other location easily available for all developers. Embedded platforms may substitute an emulator available in the FreeBSD.org cluster for actual hardware. * Inclusion in the `make universe` and `make tinderbox` targets. * Dedicated hardware in one of the FreeBSD clusters for package building (either natively or via qemu-user). Collectively, developers are required to provide the following to maintain the Tier 1 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 1 platform. * Tier 1 architectures must have a mature, healthy ecosystem of users and active developers. * Developers should be able to build packages on commonly available, non-embedded Tier 1 systems. This can mean either native builds if non-embedded systems are commonly available for the platform in question, or it can mean cross-builds hosted on some other Tier 1 architecture. * Changes cannot break the userland ABI. If an ABI change is required, ABI compatibility for existing binaries should be provided via use of symbol versioning or shared library version bumps. * Changes merged to stable branches cannot break the protected portions of the kernel ABI. If a kernel ABI change is required, the change should be modified to preserve functionality of existing kernel modules. === Tier 2: Developmental and Niche Architectures Tier 2 platforms are functional, but less mature FreeBSD platforms. They are not supported by the security officer, release engineering, and port management teams. Tier 2 platforms may be Tier 1 platform candidates that are still under active development. Architectures reaching end of life may also be moved from Tier 1 status to Tier 2 status as the availability of resources to continue to maintain the system in a Production Quality state diminishes. Well-supported niche architectures may also be Tier 2. The FreeBSD Project provides the following guarantees to consumers of Tier 2 platforms: * The ports infrastructure should include basic support for Tier 2 architectures sufficient to support building ports and packages. This includes support for basic packages such as ports-mgmt/pkg, but there is no guarantee that arbitrary ports will be buildable or functional. * New features which are not inherently platform-specific should be feasible on all Tier 2 architectures if not implemented. * Tier 2 platforms will be included in the source tree. * Tier 2 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. * Tier 2 platforms should provide functional kernels and userlands even if an official release distribution is not provided. To maintain maturity of Tier 2 platforms, the FreeBSD Project will maintain the following resources to support development: * Inclusion in the `make universe` and `make tinderbox` targets. Collectively, developers are required to provide the following to maintain the Tier 2 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 2 platform. * Tier 2 architectures must have an active ecosystem of users and developers. * While changes are permitted to break the userland ABI, the ABI should not be broken gratuitously. Significant userland ABI changes should be restricted to major versions. * New features that are not yet implemented on Tier 2 architectures should provide a means of disabling them on those architectures. === Tier 3: Experimental Architectures Tier 3 platforms have at least partial FreeBSD support. They are _not_ supported by the security officer, release engineering, and port management teams. Tier 3 platforms are architectures in the early stages of development, for non-mainstream hardware platforms, or which are considered legacy systems unlikely to see broad future use. Initial support for Tier 3 platforms may exist in a separate repository rather than the main source repository. The FreeBSD Project provides no guarantees to consumers of Tier 3 platforms and is not committed to maintaining resources to support development. Tier 3 platforms may not always be buildable, nor are any kernel or userland ABIs considered stable. === Tier 4: Unsupported Architectures Tier 4 platforms are not supported in any form by the project. All systems not otherwise classified are Tier 4 systems. When a platform transitions to Tier 4, all support for the platform is removed from the source and ports trees. Note that ports support should remain as long as the platform is supported in a branch supported by ports. === Policy on Changing the Tier of an Architecture Systems may only be moved from one tier to another by approval of the FreeBSD Core Team, which shall make that decision in collaboration with the Security Officer, Release Engineering, and ports management teams. For a platform to be promoted to a higher tier, any missing support guarantees must be satisfied before the promotion is completed. [[ports]] == Ports Specific FAQ [[ports-qa-adding]] === Adding a New Port [[ports-qa-add-new]] ==== How do I add a new port? First, please read the section about repository copies. The easiest way to add a new port is the `addport` script located in the [.filename]#ports/Tools/scripts# directory. It adds a port from the directory specified, determining the category automatically from the port [.filename]#Makefile#. It also adds an entry to the port's category [.filename]#Makefile#. It was written by `{mharo}`, `{will}`, and `{garga}`. When sending questions about this script to the {freebsd-ports}, please also CC `{crees}`, the current maintainer. [[ports-qa-add-new-extra]] ==== Any other things I need to know when I add a new port? Check the port, preferably to make sure it compiles and packages correctly. The link:{porters-handbook}testing[Porters Handbook's Testing Chapter] contains more detailed instructions. See the link:{porters-handbook}testing#testing-portclippy[Portclippy / Portfmt] and the link:{porters-handbook}testing#testing-poudriere[Poudriere] sections. You do not necessarily have to eliminate all warnings but make sure you have fixed the simple ones. If the port came from a submitter who has not contributed to the Project before, add that person's name to the link:{contributors}#contrib-additional[Additional Contributors] section of the FreeBSD Contributors List. Close the PR if the port came in as a PR. To close a PR, change the state to `Issue Resolved` and the resolution as `Fixed`. [NOTE] ==== If for some reason using link:{porters-handbook}testing#testing-poudriere[Poudriere] to test the new port is not possible, the bare minimum of testing includes this sequence: [source,shell] .... # make install # make package # make deinstall # pkg add package you built above # make deinstall # make reinstall # make package .... Note that poudriere is the reference for package building, it the port does not build in poudriere, it will be removed. ==== [[ports-qa-removing]] === Removing an Existing Port [[ports-qa-remove-one]] ==== How do I remove an existing port? First, please read the section about repository copies. Before you remove the port, you have to verify there are no other ports depending on it. * Make sure there is no dependency on the port in the ports collection: ** The port's PKGNAME appears in exactly one line in a recent INDEX file. ** No other ports contains any reference to the port's directory or PKGNAME in their Makefiles + [TIP] ==== When using Git, consider using man:git-grep[1], it is much faster than `grep -r`. ==== + * Then, remove the port: + [.procedure] ==== * Remove the port's files and directory with `git rm`. * Remove the `SUBDIR` listing of the port in the parent directory [.filename]#Makefile#. * Add an entry to [.filename]#ports/MOVED#. * Search for entries in [.filename]#ports/security/vuxml/vuln.xml# and adjust them accordingly. In particular, check for previous packages with the new name which version could include the new port. * Remove the port from [.filename]#ports/LEGAL# if it is there. ==== Alternatively, you can use the rmport script, from [.filename]#ports/Tools/scripts#. This script was written by {vd}. When sending questions about this script to the {freebsd-ports}, please also CC {crees}, the current maintainer. [[ports-qa-move-port]] === How do I move a port to a new location? [.procedure] ==== . Perform a thorough check of the ports collection for any dependencies on the old port location/name, and update them. Running `grep` on [.filename]#INDEX# is not enough because some ports have dependencies enabled by compile-time options. A full man:git-grep[1] of the ports collection is recommended. . Remove the `SUBDIR` entry from the old category Makefile and add a `SUBDIR` entry to the new category Makefile. . Add an entry to [.filename]#ports/MOVED#. . Move the port with `git mv`. . Commit the changes. ==== [[ports-qa-copy-port]] === How do I copy a port to a new location? [.procedure] ==== . Copy port with `cp -R old-cat/old-port new-cat/new-port`. . Add the new port to the [.filename]#new-cat/Makefile#. . Change stuff in [.filename]#new-cat/new-port#. . Commit the changes. ==== [[ports-qa-freeze]] === Ports Freeze [[ports-qa-freeze-what]] ==== What is a “ports freeze”? A “ports freeze” was a restricted state the ports tree was put in before a release. It was used to ensure a higher quality for the packages shipped with a release. It usually lasted a couple of weeks. During that time, build problems were fixed, and the release packages were built. This practice is no longer used, as the packages for the releases are built from the current stable, quarterly branch. For more information on how to merge commits to the quarterly branch, see <>. [[ports-qa-quarterly]] === Quarterly Branches [[ports-qa-misc-request-mfh]] ==== What is the procedure to request authorization for merging a commit to the quarterly branch? As of November 30, 2020, there is no need to seek explicit approval to commit to the quarterly branch. [[ports-qa-misc-commit-mfh]] ==== What is the procedure for merging commits to the quarterly branch? Merging commits to the quarterly branch is very similar to MFC'ing a commit in the src repository, so basically: [source,shell] .... % git checkout 2021Q2 % git cherry-pick -x $HASH (verify everything is OK, for example by doing a build test) % git push .... where '$HASH' is the hash of the commit you want to copy over to the quarterly branch. The -x parameter ensures the hash '$HASH' of the main branch is included in the new commit message of the quarterly branch. [[ports-qa-new-category]] === Creating a New Category [[ports-qa-new-category-how]] ==== What is the procedure for creating a new category? Please see link:{porters-handbook}#proposing-categories[Proposing a New Category] in the Porter's Handbook. Once that procedure has been followed and the PR has been assigned to the {portmgr}, it is their decision whether or not to approve it. If they do, it is their responsibility to: [.procedure] ==== . Perform any needed moves. (This only applies to physical categories.) . Update the `VALID_CATEGORIES` definition in [.filename]#ports/Mk/bsd.port.mk#. . Assign the PR back to you. ==== [[ports-qa-new-category-physical]] ==== What do I need to do to implement a new physical category? [.procedure] ==== . Upgrade each moved port's [.filename]#Makefile#. Do not connect the new category to the build yet. + To do this, you will need to: + [.procedure] ====== . Change the port's `CATEGORIES` (this was the point of the exercise, remember?) The new category is listed first. This will help to ensure that the PKGORIGIN is correct. . Run a `make describe`. Since the top-level `make index` that you will be running in a few steps is an iteration of `make describe` over the entire ports hierarchy, catching any errors here will save you having to re-run that step later on. . If you want to be really thorough, now might be a good time to run man:portlint[1]. ====== + . Check that the ``PKGORIGIN``s are correct. The ports system uses each port's `CATEGORIES` entry to create its `PKGORIGIN`, which is used to connect installed packages to the port directory they were built from. If this entry is wrong, common port tools like man:pkg_version[1] and man:portupgrade[1] fail. + To do this, use the [.filename]#chkorigin.sh# tool: `env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh`. This will check every port in the ports tree, even those not connected to the build, so you can run it directly after the move operation. Hint: do not forget to look at the ``PKGORIGIN``s of any slave ports of the ports you just moved! . On your own local system, test the proposed changes: first, comment out the SUBDIR entries in the old ports' categories' [.filename]##Makefile##s; then enable building the new category in [.filename]#ports/Makefile#. Run make checksubdirs in the affected category directories to check the SUBDIR entries. Next, in the [.filename]#ports/# directory, run make index. This can take over 40 minutes on even modern systems; however, it is a necessary step to prevent problems for other people. . Once this is done, you can commit the updated [.filename]#ports/Makefile# to connect the new category to the build and also commit the [.filename]#Makefile# changes for the old category or categories. . Add appropriate entries to [.filename]#ports/MOVED#. . Update the documentation by modifying: ** the link:{porters-handbook}#PORTING-CATEGORIES[list of categories] in the Porter's Handbook + . Only once all the above have been done, and no one is any longer reporting problems with the new ports, should the old ports be deleted from their previous locations in the repository. ==== ==== What do I need to do to implement a new virtual category? This is much simpler than a physical category. Only a few modifications are needed: * the link:{porters-handbook}#PORTING-CATEGORIES[list of categories] in the Porter's Handbook [[ports-qa-misc-questions]] === Miscellaneous Questions [[ports-qa-misc-blanket-approval]] ==== Are there changes that can be committed without asking the maintainer for approval? Blanket approval for most ports applies to these types of fixes: * Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new `USES` macros, enabling verbose builds, and switching to new ports system syntaxes. * Trivial and _tested_ build and runtime fixes. * Documentations or metadata changes to ports, like [.filename]#pkg-descr# or `COMMENT`. [IMPORTANT] ==== Exceptions to this are anything maintained by the {portmgr}, or the {security-officer}. No unauthorized commits may ever be made to ports maintained by those groups. ==== [[ports-qa-misc-correctly-building]] ==== How do I know if my port is building correctly or not? The packages are built multiple times each week. If a port fails, the maintainer will receive an email from `pkg-fallout@FreeBSD.org`. Reports for all the package builds (official, experimental, and non-regression) are aggregated at link:pkg-status.FreeBSD.org[pkg-status.FreeBSD.org]. [[ports-qa-misc-INDEX]] ==== I added a new port. Do I need to add it to the [.filename]#INDEX#? No. The file can either be generated by running `make index`, or a pre-generated version can be downloaded with `make fetchindex`. [[ports-qa-misc-no-touch]] ==== Are there any other files I am not allowed to touch? Any file directly under [.filename]#ports/#, or any file under a subdirectory that starts with an uppercase letter ([.filename]#Mk/#, [.filename]#Tools/#, etc.). In particular, the {portmgr} is very protective of [.filename]#ports/Mk/bsd.port*.mk# so do not commit changes to those files unless you want to face their wrath. [[ports-qa-misc-updated-distfile]] ==== What is the proper procedure for updating the checksum for a port distfile when the file changes without a version change? When the checksum for a distribution file is updated due to the author updating the file without changing the port revision, the commit message includes a summary of the relevant diffs between the original and new distfile to ensure that the distfile has not been corrupted or maliciously altered. If the current version of the port has been in the ports tree for a while, a copy of the old distfile will usually be available on the ftp servers; otherwise the author or maintainer should be contacted to find out why the distfile has changed. [[ports-exp-run]] ==== How can an experimental test build of the ports tree (exp-run) be requested? An exp-run must be completed before patches with a significant ports impact are committed. The patch can be against the ports tree or the base system. Full package builds will be done with the patches provided by the submitter, and the submitter is required to fix detected problems _(fallout)_ before commit. [.procedure] ==== . Go to the link:https://bugs.freebsd.org/submit[Bugzilla new PR page]. . Select the product your patch is about. . Fill in the bug report as normal. Remember to attach the patch. . If at the top it says “Show Advanced Fields” click on it. It will now say “Hide Advanced Fields”. Many new fields will be available. If it already says “Hide Advanced Fields”, no need to do anything. . In the “Flags” section, set the “exp-run” one to `?`. As for all other fields, hovering the mouse over any field shows more details. . Submit. Wait for the build to run. . {portmgr} will reply with a possible fallout. . Depending on the fallout: ** If there is no fallout, the procedure stops here, and the change can be committed, pending any other approval required. ... If there is fallout, it _must_ be fixed, either by fixing the ports directly in the ports tree, or adding to the submitted patch. ... When this is done, go back to step 6 saying the fallout was fixed and wait for the exp-run to be run again. Repeat as long as there are broken ports. ==== [[non-committers]] == Issues Specific to Developers Who Are Not Committers A few people who have access to the FreeBSD machines do not have commit bits. Almost all of this document will apply to these developers as well (except things specific to commits and the mailing list memberships that go with them). In particular, we recommend that you read: * <> * <> + [NOTE] ==== Get your mentor to add you to the "Additional Contributors" ([.filename]#~/documentation/content/en/articles/contributors/contrib-additional.adoc#), if you are not already listed there. ==== * <> * <> * <> [[google-analytics]] == Information About Google Analytics As of December 12, 2012, Google Analytics was enabled on the FreeBSD Project website to collect anonymized usage statistics regarding usage of the site. The information collected is valuable to the FreeBSD Documentation Project, to identify various problems on the FreeBSD website. [[google-analytics-policy]] === Google Analytics General Policy The FreeBSD Project takes visitor privacy very seriously. As such, the FreeBSD Project website honors the "Do Not Track" header _before_ fetching the tracking code from Google. For more information, please see the https://www.FreeBSD.org/privacy/[FreeBSD Privacy Policy]. Google Analytics access is _not_ arbitrarily allowed - access must be requested, voted on by the `{doceng}`, and explicitly granted. Requests for Google Analytics data must include a specific purpose. For example, a valid reason for requesting access would be "to see the most frequently used web browsers when viewing FreeBSD web pages to ensure page rendering speeds are acceptable." Conversely, "to see what web browsers are most frequently used" (without stating __why__) would be rejected. All requests must include the timeframe for which the data would be required. For example, it must be explicitly stated if the requested data would be needed for a timeframe covering a span of 3 weeks, or if the request would be one-time only. Any request for Google Analytics data without a clear, reasonable reason beneficial to the FreeBSD Project will be rejected. [[google-analytics-data]] === Data Available Through Google Analytics A few examples of the types of Google Analytics data available include: * Commonly used web browsers * Page load times * Site access by language [[misc]] == Miscellaneous Questions === Are there changes that can be committed without asking the maintainer for approval? Blanket approval for most ports applies to these types of fixes: * Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new `USES` macros, enabling verbose builds, and switching to new ports system syntaxes. * Trivial and _tested_ build and runtime fixes. * Documentations or metadata changes to ports, like [.filename]#pkg-descr# or `COMMENT`. === How do I access people.FreeBSD.org to put up personal or project information? `people.FreeBSD.org` is the same as `freefall.FreeBSD.org`. Just create a [.filename]#public_html# directory. Anything you place in that directory will automatically be visible under https://people.FreeBSD.org/[https://people.FreeBSD.org/]. === Where are the mailing list archives stored? The mailing lists are archived under [.filename]#/local/mail# on `freefall.FreeBSD.org`. === I would like to mentor a new committer. What process do I need to follow? See the https://www.freebsd.org/internal/new-account/[New Account Creation Procedure] document on the internal pages. [[benefits]] == Benefits and Perks for FreeBSD Committers [[benefits-recognition]] === Recognition Recognition as a competent software engineer is the longest lasting value. In addition, getting a chance to work with some of the best people that every engineer would dream of meeting is a great perk! [[benefits-freebsdmall]] === FreeBSD Mall FreeBSD committers can get a free 4-CD or DVD set at conferences from http://www.freebsdmall.com[FreeBSD Mall, Inc.]. [[benefits-irc]] === IRC In addition, developers may request a cloaked hostmask for their account on the Freenode IRC network in the form of `freebsd/developer/` or `freebsd/developer/`. To request a cloak, send an email to `{irc-email}` with your requested hostmask and NickServ account name. See the https://wiki.freebsd.org/IRC/Cloaks[IRC Cloaks] wiki page for more details. [[benefits-gandi]] === `Gandi.net` https://gandi.net[Gandi] provides website hosting, cloud computing, domain registration, and X.509 certificate services. Gandi offers an E-rate discount to all FreeBSD developers. In order to streamline the process of getting the discount first set up a Gandi account, fill in the billing information and select the currency. Then send an mail to mailto:non-profit@gandi.net[non-profit@gandi.net] using your `@freebsd.org` mail address, and indicate your Gandi handle. [[benefits-rsync]] === `rsync.net` https://rsync.net[rsync.net] provides cloud storage for offsite backup that is optimized for UNIX users. Their service runs entirely on FreeBSD and ZFS. rsync.net offers a free-forever 500 GB account to FreeBSD developers. Simply sign up at https://www.rsync.net/freebsd.html[https://www.rsync.net/freebsd.html] using your `@freebsd.org` address to receive this free account. diff --git a/documentation/content/en/articles/contributing/_index.adoc b/documentation/content/en/articles/contributing/_index.adoc index 280ae8cb6a..2f4c6e20a5 100644 --- a/documentation/content/en/articles/contributing/_index.adoc +++ b/documentation/content/en/articles/contributing/_index.adoc @@ -1,575 +1,587 @@ --- title: Contributing to FreeBSD authors: - author: Jordan Hubbard - author: Sam Lawrance - author: Mark Linimon description: How to contribute to the FreeBSD Project trademarks: ["freebsd", "ieee", "general"] weight: 15 tags: ["Contributing", "FreeBSD", "Non-Programmer Tasks", "Programmer Tasks"] --- = Contributing to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article describes the different ways in which an individual or organization may contribute to the FreeBSD Project. ''' toc::[] So you want to contribute to FreeBSD? That is great! FreeBSD _relies_ on the contributions of its user base to survive. Your contributions are not only appreciated, they are vital to FreeBSD's continued growth. A large and growing number of international contributors, of greatly varying ages and areas of technical expertise, develop FreeBSD. There is always more work to be done than there are people available to do it, and more help is always appreciated. As a volunteer, what you do is limited only by what you want to do. However, we do ask that you are aware of what other members of the FreeBSD community will expect of you. You may want to take this into account before deciding to volunteer. The FreeBSD project is responsible for an entire operating system environment, rather than just a kernel or a few scattered utilities. As such, our [.filename]#TODO# lists span a very wide range of tasks: from documentation, beta testing and presentation, to the system installer and highly specialized types of kernel development. People of any skill level, in almost any area, can almost certainly help the project. Commercial entities engaged in FreeBSD-related enterprises are also encouraged to contact us. Do you need a special extension to make your product work? You will find us receptive to your requests, given that they are not too outlandish. Are you working on a value-added product? Please let us know! We may be able to work cooperatively on some aspect of it. The free software world is challenging many existing assumptions about how software is developed, sold, and maintained, and we urge you to at least give it a second look. [[contrib-what]] == What Is Needed The following list of tasks and sub-projects represents something of an amalgam of various [.filename]#TODO# lists and user requests. [[non-programmer-tasks]] === Ongoing Non-Programmer Tasks Many people who are involved in FreeBSD are not programmers. The Project includes documentation writers, Web designers, and support people. All that these people need to contribute is an investment of time and a willingness to learn. . Read through the FAQ and Handbook periodically. If anything is poorly explained, ambiguous, out of date or incorrect, let us know. Even better, send us a fix (Docbook is not difficult to learn, but there is no objection to ASCII submissions). . Help translate FreeBSD documentation into your native language. If documentation already exists for your language, you can help translate additional documents or verify that the translations are up-to-date and correct. First take a look at the link:{fdp-primer}#translations[Translations FAQ] in the FreeBSD Documentation Project Primer. You are not committing yourself to translating every single FreeBSD document by doing this - as a volunteer, you can do as much or as little translation as you desire. Once someone begins translating, others almost always join the effort. If you only have the time or energy to translate one part of the documentation, please translate the installation instructions. . Read the {freebsd-questions} occasionally (or even regularly). It can be very satisfying to share your expertise and help people solve their problems; sometimes you may even learn something new yourself! These forums can also be a source of ideas for things to improve upon. [[ongoing-programmer-tasks]] === Ongoing Programmer Tasks Most of the tasks listed here may require a considerable investment of time, an in-depth knowledge of the FreeBSD kernel, or both. However, there are also many useful tasks which are suitable for "weekend hackers". . If you run FreeBSD-CURRENT and have a good Internet connection, there is a machine `current.FreeBSD.org` which builds a full release once a day-every now and again, try to install the latest release from it and report any failures in the process. . Read the {freebsd-bugs}. There may be a problem you can comment constructively on or with patches you can test. Or you could even try to fix one of the problems yourself. . If you know of any bug fixes which have been successfully applied to -CURRENT but have not been merged into -STABLE after a decent interval (normally a couple of weeks), send the committer a polite reminder. . Move contributed software to [.filename]#src/contrib# in the source tree. . Make sure code in [.filename]#src/contrib# is up to date. . Build the source tree (or just part of it) with extra warnings enabled and clean up the warnings. A list of build warnings can also be found from our https://ci.freebsd.org[CI] by selecting a build and checking "LLVM/Clang Warnings". . Fix warnings for ports which do deprecated things like using `gets()` or including [.filename]#malloc.h#. . If you have contributed any ports and you had to make FreeBSD-specific changes, send your patches back to the original authors (this will make your life easier when they bring out the next version). . Get copies of formal standards like POSIX(R). Compare FreeBSD's behavior to that required by the standard. If the behavior differs, particularly in subtle or obscure corners of the specification, send in a PR about it. If you are able, figure out how to fix it and include a patch in the PR. If you think the standard is wrong, ask the standards body to consider the question. . Suggest further tasks for this list! === Work through the PR Database The https://bugs.FreeBSD.org/search/[FreeBSD PR list] shows all the current active problem reports and requests for enhancement that have been submitted by FreeBSD users. The PR database includes both programmer and non-programmer tasks. Look through the open PRs, and see if anything there takes your interest. Some of these might be very simple tasks that just need an extra pair of eyes to look over them and confirm that the fix in the PR is a good one. Others might be much more complex, or might not even have a fix included at all. Start with the PRs that have not been assigned to anyone else. If a PR is assigned to someone else, but it looks like something you can handle, email the person it is assigned to and ask if you can work on it-they might already have a patch ready to be tested, or further ideas that you can discuss with them. === Ongoing Ports Tasks The Ports Collection is a perpetual work in progress. We want to provide our users with an easy to use, up to date, high quality repository of third party software. We need people to donate some of their time and effort to help us achieve this goal. Anyone can get involved, and there are lots of different ways to do so. Contributing to ports is an excellent way to help "give back" something to the project. Whether you are looking for an ongoing role, or a fun challenge for a rainy day, we would love to have your help! There are a number of easy ways you can contribute to keeping the ports tree up to date and in good working order: * Find some cool or useful software and link:{porters-handbook}[create a port] for it. * There are a large number of ports that have no maintainer. Become a maintainer and <>. * If you have created or adopted a port, be aware of <>. * When you are looking for a quick challenge you could <>. === Pick one of the items from the Ideas page The https://wiki.freebsd.org/IdeasPage[FreeBSD list of projects and ideas for volunteers] is also available for people willing to contribute to the FreeBSD project. The list is being regularly updated and contains items for both programmers and non-programmers with information about each project. [[contrib-how]] == How to Contribute Contributions to the system generally fall into one or more of the following 5 categories: [[contrib-general]] === Bug Reports and General Commentary An idea or suggestion of _general_ technical interest should be mailed to the {freebsd-hackers}. Likewise, people with an interest in such things (and a tolerance for a _high_ volume of mail!) may subscribe to the {freebsd-hackers}. See link:{handbook}#eresources-mail[The FreeBSD Handbook] for more information about this and other mailing lists. If you find a bug or are submitting a specific change, please report it using the https://bugs.FreeBSD.org/submit/[bug submission form]. Try to fill-in each field of the bug report. Unless they exceed 65KB, include any patches directly in the report. If the patch is suitable to be applied to the source tree put `[PATCH]` in the synopsis of the report. When including patches, _do not_ use cut-and-paste because cut-and-paste turns tabs into spaces and makes them unusable. When patches are a lot larger than 20KB, consider compressing them (eg. with man:gzip[1] or man:bzip2[1]) prior to uploading them. After filing a report, you should receive confirmation along with a tracking number. Keep this tracking number so that you can update us with details about the problem. See also link:{problem-reports}[this article] on how to write good problem reports. === Changes to the Documentation Changes to the documentation are overseen by the {freebsd-doc}. Please look at the link:{fdp-primer}[FreeBSD Documentation Project Primer] for complete instructions. Send submissions and changes (even small ones are welcome!) using the same method as any other bug report. === Changes to Existing Source Code An addition or change to the existing source code is a somewhat trickier affair and depends a lot on how far out of date you are with the current state of FreeBSD development. There is a special on-going release of FreeBSD known as "FreeBSD-CURRENT" which is made available in a variety of ways for the convenience of developers working actively on the system. See link:{handbook}#current-stable[The FreeBSD Handbook] for more information about getting and using FreeBSD-CURRENT. Working from older sources unfortunately means that your changes may sometimes be too obsolete or too divergent for easy re-integration into FreeBSD. Chances of this can be minimized somewhat by subscribing to the {freebsd-announce} and the {freebsd-current} lists, where discussions on the current state of the system take place. Assuming that you can manage to secure fairly up-to-date sources to base your changes on, the next step is to produce a set of diffs to send to the FreeBSD maintainers. This is done with the man:diff[1] command. The preferred man:diff[1] format for submitting patches is the unified output format generated by `diff -u`. [source,shell] .... % diff -u oldfile newfile .... or [source,shell] .... % diff -u -r -N olddir newdir .... would generate a set of unified diffs for the given source file or directory hierarchy. See man:diff[1] for more information. Once you have a set of diffs (which you may test with the man:patch[1] command), you should submit them for inclusion with FreeBSD as a bug report. _Do not_ just send the diffs to the {freebsd-hackers} or they will get lost! We greatly appreciate your submission (this is a volunteer project!); because we are busy, we may not be able to address it immediately, but it will remain in the PR database until we do. Indicate your submission by including `[PATCH]` in the synopsis of the report. If you feel it appropriate (e.g. you have added, deleted, or renamed files), bundle your changes into a `tar` file. Archives created with man:shar[1] are also welcome. If your change is of a potentially sensitive nature, such as if you are unsure of copyright issues governing its further distribution then you should send it to {core-email} directly rather than submitting as a bug report. The {core-email} reaches a much smaller group of people who do much of the day-to-day work on FreeBSD. Note that this group is also _very busy_ and so you should only send mail to them where it is truly necessary. Please refer to man:intro[9] and man:style[9] for some information on coding style. We would appreciate it if you were at least aware of this information before submitting code. === New Code or Major Value-Added Packages In the case of a significant contribution of a large body work, or the addition of an important new feature to FreeBSD, it becomes almost always necessary to either send changes as tar files or upload them to a web or FTP site for other people to access. If you do not have access to a web or FTP site, ask on an appropriate FreeBSD mailing list for someone to host the changes for you. When working with large amounts of code, the touchy subject of copyrights also invariably comes up. FreeBSD prefers free software licenses such as BSD or ISC. Copyleft licenses such as GPLv2 are sometimes permitted. The complete listing can be found on the link:https://www.FreeBSD.org/internal/software-license/[core team licensing policy] page. === Money or Hardware We are always very happy to accept donations to further the cause of the FreeBSD Project and, in a volunteer effort like ours, a little can go a long way! Donations of hardware are also very important to expanding our list of supported peripherals since we generally lack the funds to buy such items ourselves. [[donations]] ==== Donating Funds The https://www.freebsdfoundation.org[FreeBSD Foundation] is a non-profit, tax-exempt foundation established to further the goals of the FreeBSD Project. As a 501(c)3 entity, the Foundation is generally exempt from US federal income tax as well as Colorado State income tax. Donations to a tax-exempt entity are often deductible from taxable federal income. Donations may be sent in check form to: [.address] **** The FreeBSD Foundation + P.O. Box 20247, + Boulder, + CO 80308 + USA **** The FreeBSD Foundation is also able to accept https://www.freebsdfoundation.org/donate/[online donations] through various payment options. More information about the FreeBSD Foundation can be found in https://people.FreeBSD.org/~jdp/foundation/announcement.html[The FreeBSD Foundation -- an Introduction]. To contact the Foundation by email, write to mailto:info@FreeBSDFoundation.org[info@FreeBSDFoundation.org]. ==== Donating Hardware The FreeBSD Project happily accepts donations of hardware that it can find good use for. If you are interested in donating hardware, please contact the link:https://www.FreeBSD.org/donations/[Donations Liaison Office]. [[ports-contributing]] == Contributing to ports [[adopt-port]] === Adopting an unmaintained port ==== Choosing an unmaintained port Taking over maintainership of ports that are unmaintained is a great way to get involved. Unmaintained ports are only updated and fixed when somebody volunteers to work on them. There are a large number of unmaintained ports. It is a good idea to start with adopting a port that you use regularly. Unmaintained ports have their `MAINTAINER` set to `ports@FreeBSD.org`. A list of unmaintained ports and their current errors and problem reports can be seen at the http://portsmon.FreeBSD.org/portsconcordanceformaintainer.py?maintainer=ports%40FreeBSD.org[FreeBSD Ports Monitoring System]. On https://portsfallout.com/fallout?port=&maintainer=ports%40FreeBSD.org[PortsFallout] can be seen a list of unmaintained ports with errors. Many unmaintained ports can have pending updates, this can be seen at the https://portscout.freebsd.org/ports@freebsd.org.html[FreeBSD Ports distfile scanner]. Some ports affect a large number of others due to dependencies and slave port relationships. Generally, we want people to have some experience before they maintain such ports. You can find out whether or not a port has dependencies or slave ports by looking at a master index of ports called [.filename]#INDEX#. (The name of the file varies by release of FreeBSD; for instance, [.filename]#INDEX-8#.) Some ports have conditional dependencies that are not included in a default [.filename]#INDEX# build. We expect you to be able to recognize such ports by looking through other ports' [.filename]#Makefile#'s. [NOTE] ====== The FreeBSD Ports Monitoring System (portsmon) is currently not working due to latest Python updates. ====== ==== How to adopt the port First make sure you understand your <>. Also read the link:{porters-handbook}[Porter's Handbook]. _Please do not commit yourself to more than you feel you can comfortably handle._ You may request maintainership of any unmaintained port as soon as you wish. Simply set `MAINTAINER` to your own email address and send a PR (Problem Report) with the change. If the port has build errors or needs updating, you may wish to include any other changes in the same PR. This will help because many committers are less willing to assign maintainership to someone who does not have a known track record with FreeBSD. Submitting PRs that fix build errors or update ports are the best ways to establish one. File your PR with category `ports` and class `change-request`. A committer will examine your PR, commit the changes, and finally close the PR. Sometimes this process can take a little while (committers are volunteers, too :). [[maintain-port]] === The challenge for port maintainers This section will give you an idea of why ports need to be maintained and outline the responsibilities of a port maintainer. [[why-maintenance]] ==== Why ports require maintenance Creating a port is a once-off task. Ensuring that a port is up to date and continues to build and run requires an ongoing maintenance effort. Maintainers are the people who dedicate some of their time to meeting these goals. The foremost reason ports need maintenance is to bring the latest and greatest in third party software to the FreeBSD community. An additional challenge is to keep individual ports working within the Ports Collection framework as it evolves. As a maintainer, you will need to manage the following challenges: * *New software versions and updates.* New versions and updates of existing ported software become available all the time, and these need to be incorporated into the Ports Collection in order to provide up-to-date software. * *Changes to dependencies.* If significant changes are made to the dependencies of your port, it may need to be updated so that it will continue to work correctly. * *Changes affecting dependent ports.* If other ports depend on a port that you maintain, changes to your port may require coordination with other maintainers. * *Interaction with other users, maintainers and developers.* Part of being a maintainer is taking on a support role. You are not expected to provide general support (but we welcome it if you choose to do so). What you should provide is a point of coordination for FreeBSD-specific issues regarding your ports. * *Bug hunting.* A port may be affected by bugs which are specific to FreeBSD. You will need to investigate, find, and fix these bugs when they are reported. Thoroughly testing a port to identify problems before they make their way into the Ports Collection is even better. * *Changes to ports infrastructure and policy.* Occasionally the systems that are used to build ports and packages are updated or a new recommendation affecting the infrastructure is made. You should be aware of these changes in case your ports are affected and require updating. * *Changes to the base system.* FreeBSD is under constant development. Changes to software, libraries, the kernel or even policy changes can cause flow-on change requirements to ports. ==== Maintainer responsibilities ===== Keep your ports up to date This section outlines the process to follow to keep your ports up to date. This is an overview. More information about upgrading a port is available in the link:{porters-handbook}[Porter's Handbook]. [.procedure] ==== . Watch for updates + Monitor the upstream vendor for new versions, updates and security fixes for the software. Announcement mailing lists or news web pages are useful for doing this. Sometimes users will contact you and ask when your port will be updated. If you are busy with other things or for any reason just cannot update it at the moment, ask if they will help you by submitting an update. + You may also receive automated email from the `FreeBSD Ports Version Check` informing you that a newer version of your port's distfile is available. More information about that system (including how to stop future emails) will be provided in the message. . Incorporate changes + When they become available, incorporate the changes into the port. You need to be able to generate a patch between the original port and your updated port. . Review and test + Thoroughly review and test your changes: ** Build, install and test your port on as many platforms and architectures as you can. It is common for a port to work on one branch or platform and fail on another. ** Make sure your port's dependencies are complete. The recommended way of doing this is by installing your own ports tinderbox. See <> for more information. ** Check that the packing list is up to date. This involves adding in any new files and directories and removing unused entries. ** Verify your port using man:portlint[1] as a guide. See <> for important information about using portlint. ** Consider whether changes to your port might cause any other ports to break. If this is the case, coordinate the changes with the maintainers of those ports. This is especially important if your update changes the shared library version; in this case, at the very least, the dependent ports will need to get a `PORTREVISION` bump so that they will automatically be upgraded by automated tools such as portmaster or man:portupgrade[1]. . Submit changes + Send your update by submitting a PR with an explanation of the changes and a patch containing the differences between the original port and the updated one. Please refer to link:{problem-reports}[Writing FreeBSD Problem Reports] for information on how to write a really good PR. + [NOTE] ====== Please do not submit a man:shar[1] archive of the entire port; instead, use man:diff[1] `-ruN`. In this way, committers can much more easily see exactly what changes are being made. The Porter's Handbook section on link:{porters-handbook}#port-upgrading[Upgrading] has more information. ====== . Wait + At some stage a committer will deal with your PR. It may take minutes, or it may take weeks - so please be patient. . Give feedback + If a committer finds a problem with your changes, they will most likely refer it back to you. A prompt response will help get your PR committed faster, and is better for maintaining a thread of conversation when trying to resolve any problems. . And Finally + Your changes will be committed and your port will have been updated. The PR will then be closed by the committer. That's it! ==== ===== Ensure your ports continue to build correctly This section is about discovering and fixing problems that stop your ports from building correctly. FreeBSD only guarantees that the Ports Collection works on the `-STABLE` branches. In theory, you should be able to get by with running the latest release of each stable branch (since the ABIs are not supposed to change) but if you can run the branch, that is even better. Since the majority of FreeBSD installations run on PC-compatible machines (what is termed the `i386` architecture), we expect you to keep the port working on that architecture. We prefer that ports also work on the `amd64` architecture running native. It is completely fair to ask for help if you do not have one of these machines. [NOTE] ==== The usual failure modes for non-`x86` machines are that the original programmers assumed that, for instance, pointers are `int`-s, or that a relatively lax older gcc compiler was being used. More and more, application authors are reworking their code to remove these assumptions - but if the author is not actively maintaining their code, you may need to do this yourself. ==== These are the tasks you need to perform to ensure your port is able to be built: [.procedure] ==== . Watch for build failures + Check your mail for mail from `pkg-fallout@FreeBSD.org` and the http://portscout.FreeBSD.org[distfiles scanner] to see if any of the port which are failing to build are out of date. . Collect information + Once you are aware of a problem, collect information to help you fix it. Build errors reported by `pkg-fallout` are accompanied by logs which will show you where the build failed. If the failure was reported to you by a user, ask them to send you information which may help in diagnosing the problem, such as: ** Build logs ** The commands and options used to build the port (including options set in [.filename]#/etc/make.conf#) ** A list of packages installed on their system as shown by man:pkg-info[8] ** The version of FreeBSD they are running as shown by man:uname[1] `-a` ** When their ports collection was last updated ** When their ports tree and [.filename]#INDEX# was last updated . Investigate and find a solution + Unfortunately there is no straightforward process to follow to do this. Remember, though: if you are stuck, ask for help! The {freebsd-ports} is a good place to start, and the upstream developers are often very helpful. . Submit changes + Just as with updating a port, you should now incorporate changes, review and test, submit your changes in a PR, and provide feedback if required. . Send patches to upstream authors + In some cases, you will have to make patches to the port to make it run on FreeBSD. Some (but not all) upstream authors will accept such patches back into their code for the next release. If so, this may even help their users on other BSD-based systems as well and perhaps save duplicated effort. Please consider sending any applicable patches to the authors as a courtesy. ==== ===== Investigate bug reports and PRs related to your port This section is about discovering and fixing bugs. FreeBSD-specific bugs are generally caused by assumptions about the build and runtime environments that do not apply to FreeBSD. You are less likely to encounter a problem of this type, but it can be more subtle and difficult to diagnose. These are the tasks you need to perform to ensure your port continues to work as intended: [.procedure] ==== . Respond to bug reports + Bugs may be reported to you through email via the https://bugs.FreeBSD.org/search/[Problem Report database]. Bugs may also be reported directly to you by users. + You should respond to PRs and other reports within 14 days, but please try not to take that long. Try to respond as soon as possible, even if it is just to say you need some more time before you can work on the PR. + If you have not responded after 14 days, any committer may commit from a PR that you have not responded to via a `maintainer-timeout`. . Collect information + If the person reporting the bug has not also provided a fix, you need to collect the information that will allow you to generate one. + If the bug is reproducible, you can collect most of the required information yourself. If not, ask the person who reported the bug to collect the information for you, such as: ** A detailed description of their actions, expected program behavior and actual behavior ** Copies of input data used to trigger the bug ** Information about their build and execution environment - for example, a list of installed packages and the output of man:env[1] ** Core dumps ** Stack traces . Eliminate incorrect reports + Some bug reports may be incorrect. For example, the user may have simply misused the program; or their installed packages may be out of date and require updating. Sometimes a reported bug is not specific to FreeBSD. In this case report the bug to the upstream developers. If the bug is within your capabilities to fix, you can also patch the port so that the fix is applied before the next upstream release. . Find a solution + As with build errors, you will need to sort out a fix to the problem. Again, remember to ask if you are stuck! . Submit or approve changes + Just as with updating a port, you should now incorporate changes, review and test, and submit your changes in a PR (or send a follow-up if a PR already exists for the problem). If another user has submitted changes in the PR, you can also send a follow-up saying whether or not you approve the changes. ==== ===== Providing support Part of being a maintainer is providing support - not for the software in general - but for the port and any FreeBSD-specific quirks and problems. Users may contact you with questions, suggestions, problems and patches. Most of the time their correspondence will be specific to FreeBSD. Occasionally you may have to invoke your skills in diplomacy, and kindly point users seeking general support to the appropriate resources. Less frequently you will encounter a person asking why the `RPMS` are not up to date or how can they get the software to run under Foo Linux. Take the opportunity to tell them that your port is up to date (if it is, of course!), and suggest that they try FreeBSD. Sometimes users and developers will decide that you are a busy person whose time is valuable and do some of the work for you. For example, they might: * submit a PR or send you patches to update your port, * investigate and perhaps provide a fix to a PR, or * otherwise submit changes to your port. In these cases your main obligation is to respond in a timely manner. Again, the timeout for non-responsive maintainers is 14 days. After this period changes may be committed unapproved. They have taken the trouble to do this for you; so please try to at least respond promptly. Then review, approve, modify or discuss their changes with them as soon as possible. If you can make them feel that their contribution is appreciated (and it should be) you will have a better chance persuading them to do more things for you in the future :-). [[fix-broken]] === Finding and fixing a broken port There are some really good places to find a port that needs some attention. You can use the https://bugs.freebsd.org/search[web interface] to the Problem Report database to search through and view unresolved PRs. The majority of ports PRs are updates, but with a little searching and skimming over synopses you should be able to find something interesting to work on (the `sw-bug` class is a good place to start). The other place is the http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System]. In particular look for unmaintained ports with build errors and ports that are marked `BROKEN`. https://portsfallout.com/[PortsFallout] shows port issues gathered from the FreeBSD package building. It is OK to send changes for a maintained port as well, but remember to ask the maintainer in case they are already working on the problem. Once you have found a bug or problem, collect information, investigate and fix! If there is an existing PR, follow up to that. Otherwise create a new PR. Your changes will be reviewed and, if everything checks out, committed. [NOTE] ====== The FreeBSD Ports Monitoring System (portsmon) is currently not working due to latest Python updates. ====== [[mortal-coil]] === When to call it quits As your interests and commitments change, you may find that you no longer have time to continue some (or all) of your ports contributions. That is fine! Please let us know if you are no longer using a port or have otherwise lost time or interest in being a maintainer. In this way we can go ahead and allow other people to try to work on existing problems with the port without waiting for your response. Remember, FreeBSD is a volunteer project, so if maintaining a port is no fun any more, it is probably time to let someone else do it! In any case, the Ports Management Team (`portmgr`) reserves the right to reset your maintainership if you have not actively maintained your port in some time. (Currently, this is set to 3 months.) By this, we mean that there are unresolved problems or pending updates that have not been worked on during that time. [[resources]] === Resources for ports maintainers and contributors The link:{porters-handbook}[Porter's Handbook] is your hitchhiker's guide to the ports system. Keep it handy! link:{problem-reports}[Writing FreeBSD Problem Reports] describes how to best formulate and submit a PR. In 2005 more than eleven thousand ports PRs were submitted! Following this article will greatly assist us in reducing the time needed to handle your PRs. The https://bugs.freebsd.org/bugzilla/query.cgi[Problem Report database]. The http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System (portsmon)] can show you cross-referenced information about ports such as build errors and problem reports. If you are a maintainer you can use it to check on the build status of your ports. As a contributor you can use it to find broken and unmaintained ports that need to be fixed. The http://portscout.FreeBSD.org[FreeBSD Ports distfile scanner (portscout)] can show you ports for which the distfiles are not fetchable. You can check on your own ports or use it to find ports that need their `MASTER_SITES` updated. package:ports-mgmt/poudriere[] is the most thorough way to test a port through the entire cycle of installation, packaging, and deinstallation. Documentation is located at the https://github.com/freebsd/poudriere[poudriere github repository] man:portlint[1] is an application which can be used to verify that your port conforms to many important stylistic and functional guidelines. portlint is a simple heuristic application, so you should use it __only as a guide__. If portlint suggests changes which seem unreasonable, consult the link:{porters-handbook}[Porter's Handbook] or ask for advice. The {freebsd-ports} is for general ports-related discussion. It is a good place to ask for help. You can https://lists.freebsd.org/mailman/listinfo[subscribe, or read and search the list archives]. Reading the archives of the {freebsd-ports-bugs} and the {svn-ports-head} may also be of interest. https://portsfallout.com/[PortsFallout] is a place to help in searching for the https://lists.freebsd.org/pipermail/freebsd-pkg-fallout/[FreeBSD package-fallout archive]. [[ideas-contributing]] == Getting Started in Other Areas Looking for something interesting to get started that is not mentioned elsewhere in this article? The FreeBSD Project has several Wiki pages containing areas within which new contributors can get ideas on how to get started. The https://wiki.freebsd.org/JuniorJobs[Junior Jobs] page has a list of projects that might be of interest to people just getting started in FreeBSD, and want to work on interesting things to get their feet wet. The https://wiki.freebsd.org/IdeasPage[Ideas Page] contains various "nice to have" or "interesting" things to work on in the Project. diff --git a/documentation/content/en/articles/contributors/_index.adoc b/documentation/content/en/articles/contributors/_index.adoc index 7921971d75..a3f7fd0652 100644 --- a/documentation/content/en/articles/contributors/_index.adoc +++ b/documentation/content/en/articles/contributors/_index.adoc @@ -1,167 +1,179 @@ --- title: Contributors to FreeBSD description: A list of organizations and individuals who have contributed to FreeBSD trademarks: ["freebsd", "sun", "general"] tags: ["Contributors", "FreeBSD", "individuals", "organizations"] --- = Contributors to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :source-highlighter: rouge :experimental: :sectnumlevels: 6 +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article lists individuals and organizations who have made a contribution to FreeBSD. ''' toc::[] [[donors]] == Donors Gallery [NOTE] ==== As of 2010, the following section is several years out-of-date. Donations from the past several years appear https://www.FreeBSD.org/donations/donors/[here]. ==== The FreeBSD Project is indebted to the following donors and would like to publicly thank them here! * _Contributors to the central server project:_ The following individuals and businesses made it possible for the FreeBSD Project to build a new central server machine, which has replaced `freefall.FreeBSD.org` at one point, by donating the following items: ** {mbarkah} and his employer, http://www.hemi.com/[ Hemisphere Online], donated a _Pentium Pro (P6) 200MHz CPU_ ** http://www.asacomputers.com/[ASA Computers] donated a _Tyan 1662 motherboard_. ** Joe McGuckin mailto:joe@via.net[joe@via.net] of http://www.via.net/[ViaNet Communications] donated a _Kingston ethernet controller._ ** Jack O'Neill mailto:jack@diamond.xtalwind.net[jack@diamond.xtalwind.net] donated an _NCR 53C875 SCSI controller card_. ** Ulf Zimmermann mailto:ulf@Alameda.net[ulf@Alameda.net] of http://www.Alameda.net/[Alameda Networks] donated _128MB of memory_, a _4 Gb disk drive and the case._ * _Direct funding:_ + The following individuals and businesses have generously contributed direct funding to the project: ** Annelise Anderson mailto:ANDRSN@HOOVER.STANFORD.EDU[ANDRSN@HOOVER.STANFORD.EDU] ** {dillon} ** http://www.bluemountain.com/[Blue Mountain Arts] ** http://www.epilogue.com/[Epilogue Technology Corporation] ** {sef} ** http://www.gta.com/[Global Technology Associates, Inc] ** Don Scott Wilde ** Gianmarco Giovannelli mailto:gmarco@masternet.it[gmarco@masternet.it] ** Josef C. Grosch mailto:joeg@truenorth.org[joeg@truenorth.org] ** Robert T. Morris ** {chuckr} ** Kenneth P. Stox mailto:ken@stox.sa.enteract.com[ken@stox.sa.enteract.com] of http://www.imagescape.com/[Imaginary Landscape, LLC.] ** Dmitry S. Kohmanyuk mailto:dk@dog.farm.org[dk@dog.farm.org] ** http://www.cdrom.co.jp/[Laser5] of Japan (a portion of the profits from sales of their various FreeBSD CDROMs). ** http://www.mmjp.or.jp/fuki/[Fuki Shuppan Publishing Co.] donated a portion of their profits from _Hajimete no FreeBSD_ (FreeBSD, Getting started) to the FreeBSD and XFree86 projects. ** http://www.ascii.co.jp/[ASCII Corp.] donated a portion of their profits from several FreeBSD-related books to the FreeBSD project. ** http://www.yokogawa.co.jp/[Yokogawa Electric Corp] has generously donated significant funding to the FreeBSD project. ** http://www.buffnet.net/[BuffNET] ** http://www.pacificsolutions.com/[Pacific Solutions] ** http://www.siemens.de/[Siemens AG] via Andre Albsmeier mailto:andre.albsmeier@mchp.siemens.de[andre.albsmeier@mchp.siemens.de] ** Chris Silva mailto:ras@interaccess.com[ras@interaccess.com] * _Hardware contributors:_ + The following individuals and businesses have generously contributed hardware for testing and device driver development/support: ** BSDi for providing the Pentium P5-90 and 486/DX2-66 EISA/VL systems that are being used for our development work, to say nothing of the network access and other donations of hardware resources. ** http://www.compaq.com[Compaq] has donated a variety of Alpha systems to the FreeBSD Project. Among the many generous donations are 4 AlphaStation DS10s, an AlphaServer DS20, AlphaServer 2100s, an AlphaServer 4100, 8 500Mhz Personal Workstations, 4 433Mhz Personal Workstations, and more! These machines are used for release engineering, package building, SMP development, and general development on the Alpha architecture. ** TRW Financial Systems, Inc. provided 130 PCs, three 68 GB file servers, twelve Ethernets, two routers and an ATM switch for debugging the diskless code. ** Dermot McDonnell donated the Toshiba XM3401B CDROM drive currently used in freefall. ** Chuck Robey mailto:chuckr@glue.umd.edu[chuckr@glue.umd.edu] contributed his floppy tape streamer for experimental work. ** Larry Altneu mailto:larry@ALR.COM[larry@ALR.COM], and {wilko}, provided Wangtek and Archive QIC-02 tape drives in order to improve the [.filename]#wt# driver. ** Ernst Winter (http://berklix.org/ewinter/[Deceased]) contributed a 2.88 MB floppy drive to the project. This will hopefully increase the pressure for rewriting the floppy disk driver. ** http://www.tekram.com/[Tekram Technologies] sent one each of their DC-390, DC-390U and DC-390F FAST and ULTRA SCSI host adapter cards for regression testing of the NCR and AMD drivers with their cards. They are also to be applauded for making driver sources for free operating systems available from their FTP server link:ftp://ftp.tekram.com/scsi/FreeBSD/[ftp://ftp.tekram.com/scsi/FreeBSD/]. ** Larry M. Augustin contributed not only a Symbios Sym8751S SCSI card, but also a set of data books, including one about the forthcoming Sym53c895 chip with Ultra-2 and LVD support, and the latest programming manual with information on how to safely use the advanced features of the latest Symbios SCSI chips. Thanks a lot! ** {kuku} donated an FX120 12 speed Mitsumi CDROM drive for IDE CDROM driver development. ** Mike Tancsa mailto:mike@sentex.ca[mike@sentex.ca] donated four various ATM PCI cards in order to help increase support of these cards as well as help support the development effort of the netatm ATM stack. * _Special contributors:_ ** http://www.osd.bsdi.com/[BSDi] (formerly Walnut Creek CDROM) has donated almost more than we can say (see the 'About the FreeBSD Project' section of the link:{handbook}[FreeBSD Handbook] for more details). In particular, we would like to thank them for the original hardware used for `freefall.FreeBSD.org`, our primary development machine, and for `thud.FreeBSD.org`, a testing and build box. We are also indebted to them for funding various contributors over the years and providing us with unrestricted use of their T1 connection to the Internet. ** The http://www.interface-business.de/[interface business GmbH, Dresden] has been patiently supporting {joerg} who has often preferred FreeBSD work over paid work, and used to fall back to their (quite expensive) EUnet Internet connection whenever his private connection became too slow or flaky to work with it... ** http://www.bsdi.com/[Berkeley Software Design, Inc.] has contributed their DOS emulator code to the remaining BSD world, which is used in the _doscmd_ command. [[staff-committers]] == The FreeBSD Developers These are the people who have commit privileges and do the engineering work on the FreeBSD source tree. All core team members are also developers. (in alphabetical order by last name): include::content/en/articles/contributors/contrib-committers.adoc[] [[contrib-corealumni]] == Core Team Alumni The following people were members of the FreeBSD core team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-corealumni.adoc[] [[contrib-develalumni]] == Development Team Alumni The following people were members of the FreeBSD development team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-develalumni.adoc[] [[contrib-portmgralumni]] == Ports Management Team Alumni The following people were members of the FreeBSD portmgr team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-portmgralumni.adoc[] [[contrib-develinmemoriam]] == Development Team: In Memoriam During the many years that the FreeBSD Project has been in existence, sadly, some of our developers have passed away. Here are some remembrances. _In rough reverse chronological order of their passing:_ include::content/en/articles/contributors/contrib-develinmemoriam.adoc[] [[contrib-derived]] == Derived Software Contributors This software was originally derived from William F. Jolitz's 386BSD release 0.1, though almost none of the original 386BSD specific code remains. This software has been essentially re-implemented from the 4.4BSD-Lite release provided by the Computer Science Research Group (CSRG) at the University of California, Berkeley and associated academic contributors. There are also portions of NetBSD and OpenBSD that have been integrated into FreeBSD as well, and we would therefore like to thank all the contributors to NetBSD and OpenBSD for their work. [[contrib-additional]] == Additional FreeBSD Contributors (in alphabetical order by first name): include::content/en/articles/contributors/contrib-additional.adoc[] [[contrib-386bsd]] == 386BSD Patch Kit Patch Contributors (in alphabetical order by first name): include::content/en/articles/contributors/contrib-386bsd.adoc[] diff --git a/documentation/content/en/articles/filtering-bridges/_index.adoc b/documentation/content/en/articles/filtering-bridges/_index.adoc index f8033f08f4..c729c8f934 100644 --- a/documentation/content/en/articles/filtering-bridges/_index.adoc +++ b/documentation/content/en/articles/filtering-bridges/_index.adoc @@ -1,275 +1,285 @@ --- title: Filtering Bridges authors: - author: Alex Dupre email: ale@FreeBSD.org description: Configuring firewalls and filtering on FreeBSD hosts acting as bridges rather than routers trademarks: ["freebsd", "3com", "intel", "general"] tags: ["network", "filtering", "bridges", "FreeBSD"] --- = Filtering Bridges :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :source-highlighter: rouge :experimental: :sectnumlevels: 6 +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract Often it is useful to divide one physical network (like an Ethernet) into two separate segments without having to create subnets, and use a router to link them together. The device that connects the two networks in this way is called a bridge. A FreeBSD system with two network interfaces is enough in order to act as a bridge. A bridge works by scanning the addresses of MAC level (Ethernet addresses) of the devices connected to each of its network interfaces and then forwarding the traffic between the two networks only if the source and the destination are on different segments. Under many points of view a bridge is similar to an Ethernet switch with only two ports. ''' toc::[] [[filtering-bridges-why]] == Why use a filtering bridge? More and more frequently, thanks to the lowering costs of broad band Internet connections (xDSL) and also because of the reduction of available IPv4 addresses, many companies are connected to the Internet 24 hours on 24 and with few (sometimes not even a power of 2) IP addresses. In these situations it is often desirable to have a firewall that filters incoming and outgoing traffic from and towards Internet, but a packet filtering solution based on router may not be applicable, either due to subnetting issues, the router is owned by the connectivity supplier (ISP), or because it does not support such functionalities. In these scenarios the use of a filtering bridge is highly advised. A bridge-based firewall can be configured and inserted between the xDSL router and your Ethernet hub/switch without any IP numbering issues. [[filtering-bridges-how]] == How to Install Adding bridge functionalities to a FreeBSD system is not difficult. Since 4.5 release it is possible to load such functionalities as modules instead of having to rebuild the kernel, simplifying the procedure a great deal. In the following subsections I will explain both installation ways. [IMPORTANT] ==== _Do not_ follow both instructions: a procedure _excludes_ the other one. Select the best choice according to your needs and abilities. ==== Before going on, be sure to have at least two Ethernet cards that support the promiscuous mode for both reception and transmission, since they must be able to send Ethernet packets with any address, not just their own. Moreover, to have a good throughput, the cards should be PCI bus mastering cards. The best choices are still the Intel EtherExpress(TM) Pro, followed by the 3Com(R) 3c9xx series. To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) in order to distinguish clearly which interface is connected to the router and which to the inner network. [[filtering-bridges-kernel]] === Kernel Configuration So you have decided to use the older but well tested installation method. To begin, you have to add the following rows to your kernel configuration file: [.programlisting] .... options BRIDGE options IPFIREWALL options IPFIREWALL_VERBOSE .... The first line is to compile the bridge support, the second one is the firewall and the third one is the logging functions of the firewall. Now it is necessary to build and install the new kernel. You may find detailed instructions in the link:{handbook}#kernelconfig-building[Building and Installing a Custom Kernel] section of the FreeBSD Handbook. [[filtering-bridges-modules]] === Modules Loading If you have chosen to use the new and simpler installation method, the only thing to do now is add the following row to [.filename]#/boot/loader.conf#: [.programlisting] .... bridge_load="YES" .... In this way, during the system startup, the [.filename]#bridge.ko# module will be loaded together with the kernel. It is not required to add a similar row for the [.filename]#ipfw.ko# module, since it will be loaded automatically after the execution of the steps in the following section. [[filtering-bridges-finalprep]] == Final Preparation Before rebooting in order to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file. The default rule of the firewall is to reject all IP packets. Initially we will set up an `open` firewall, in order to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network). Put these lines in [.filename]#/etc/rc.conf#: [.programlisting] .... firewall_enable="YES" firewall_type="open" firewall_quiet="YES" firewall_logging="YES" .... The first row will enable the firewall (and will load the module [.filename]#ipfw.ko# if it is not compiled in the kernel), the second one to set up it in `open` mode (as explained in [.filename]#/etc/rc.firewall#), the third one to not show rules loading and the fourth one to enable logging support. About the configuration of the network interfaces, the most used way is to assign an IP to only one of the network cards, but the bridge will work equally even if both interfaces or none has a configured IP. In the last case (IP-less) the bridge machine will be still more hidden, as inaccessible from the network: to configure it, you have to login from console or through a third network interface separated from the bridge. Sometimes, during the system startup, some programs require network access, say for domain resolution: in this case it is necessary to assign an IP to the external interface (the one connected to Internet, where DNS server resides), since the bridge will be activated at the end of the startup procedure. It means that the [.filename]#fxp0# interface (in our case) must be mentioned in the ifconfig section of the [.filename]#/etc/rc.conf# file, while the [.filename]#xl0# is not. Assigning an IP to both the network cards does not make much sense, unless, during the start procedure, applications should access to services on both Ethernet segments. There is another important thing to know. When running IP over Ethernet, there are actually two Ethernet protocols in use: one is IP, the other is ARP. ARP does the conversion of the IP address of a host into its Ethernet address (MAC layer). In order to allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets. Such protocol is not included in the IP layer, since it exists only with IP over Ethernet. The FreeBSD firewall filters exclusively on the IP layer and therefore all non-IP packets (ARP included) will be forwarded without being filtered, even if the firewall is configured to not permit anything. Now it is time to reboot the system and use it as before: there will be some new messages about the bridge and the firewall, but the bridge will not be activated and the firewall, being in `open` mode, will not avoid any operations. If there are any problems, you should sort them out now before proceeding. [[filtering-bridges-enabling]] == Enabling the Bridge At this point, to enable the bridge, you have to execute the following commands (having the shrewdness to replace the names of the two network interfaces [.filename]#fxp0# and [.filename]#xl0# with your own ones): [source,shell] .... # sysctl net.link.ether.bridge.config=fxp0:0,xl0:0 # sysctl net.link.ether.bridge.ipfw=1 # sysctl net.link.ether.bridge.enable=1 .... The first row specifies which interfaces should be activated by the bridge, the second one will enable the firewall on the bridge and finally the third one will enable the bridge. At this point you should be able to insert the machine between two sets of hosts without compromising any communication abilities between them. If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, in order to have them execute at startup. [[filtering-bridges-ipfirewall]] == Configuring The Firewall Now it is time to create your own file with custom firewall rules, in order to secure the inside network. There will be some complication in doing this because not all of the firewall functionalities are available on bridged packets. Furthermore, there is a difference between the packets that are in the process of being forwarded and packets that are being received by the local machine. In general, incoming packets are run through the firewall only once, not twice as is normally the case; in fact they are filtered only upon receipt, so rules that use `out` or `xmit` will never match. Personally, I use `in via` which is an older syntax, but one that has a sense when you read it. Another limitation is that you are restricted to use only `pass` or `drop` commands for packets filtered by a bridge. Sophisticated things like `divert`, `forward` or `reject` are not available. Such options can still be used, but only on traffic to or from the bridge machine itself (if it has an IP address). New in FreeBSD 4.0, is the concept of stateful filtering. This is a big improvement for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and destination reversed, of course). For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic as a single session. But with a firewall that can "remember" an outgoing UDP packet and, for the next few minutes, allow a response, handling UDP services is trivial. The following example shows how to do it. It is possible to do the same thing with TCP packets. This allows you to avoid some denial of service attacks and other nasty tricks, but it also typically makes your state table grow quickly in size. Let's look at an example setup. Note first that at the top of [.filename]#/etc/rc.firewall# there are already standard rules for the loopback interface [.filename]#lo0#, so we should not have to care for them anymore. Custom rules should be put in a separate file (say [.filename]#/etc/rc.firewall.local#) and loaded at system startup, by modifying the row of [.filename]#/etc/rc.conf# where we defined the `open` firewall: [.programlisting] .... firewall_type="/etc/rc.firewall.local" .... [IMPORTANT] ==== You have to specify the _full_ path, otherwise it will not be loaded with the risk to remain isolated from the network. ==== For our example imagine to have the [.filename]#fxp0# interface connected towards the outside (Internet) and the [.filename]#xl0# towards the inside (LAN). The bridge machine has the IP `1.2.3.4` (it is not possible that your ISP can give you an address quite like this, but for our example it is good). [.programlisting] .... # Things that we have kept state on before get to go through in a hurry add check-state # Throw away RFC 1918 networks add drop all from 10.0.0.0/8 to any in via fxp0 add drop all from 172.16.0.0/12 to any in via fxp0 add drop all from 192.168.0.0/16 to any in via fxp0 # Allow the bridge machine to say anything it wants # (if the machine is IP-less do not include these rows) add pass tcp from 1.2.3.4 to any setup keep-state add pass udp from 1.2.3.4 to any keep-state add pass ip from 1.2.3.4 to any # Allow the inside hosts to say anything they want add pass tcp from any to any in via xl0 setup keep-state add pass udp from any to any in via xl0 keep-state add pass ip from any to any in via xl0 # TCP section # Allow SSH add pass tcp from any to any 22 in via fxp0 setup keep-state # Allow SMTP only towards the mail server add pass tcp from any to relay 25 in via fxp0 setup keep-state # Allow zone transfers only by the slave name server [dns2.nic.it] add pass tcp from 193.205.245.8 to ns 53 in via fxp0 setup keep-state # Pass ident probes. It is better than waiting for them to timeout add pass tcp from any to any 113 in via fxp0 setup keep-state # Pass the "quarantine" range add pass tcp from any to any 49152-65535 in via fxp0 setup keep-state # UDP section # Allow DNS only towards the name server add pass udp from any to ns 53 in via fxp0 keep-state # Pass the "quarantine" range add pass udp from any to any 49152-65535 in via fxp0 keep-state # ICMP section # Pass 'ping' add pass icmp from any to any icmptypes 8 keep-state # Pass error messages generated by 'traceroute' add pass icmp from any to any icmptypes 3 add pass icmp from any to any icmptypes 11 # Everything else is suspect add drop log all from any to any .... Those of you who have set up firewalls before may notice some things missing. In particular, there are no anti-spoofing rules, in fact we did _not_ add: [.programlisting] .... add deny all from 1.2.3.4/8 to any in via fxp0 .... That is, drop packets that are coming in from the outside claiming to be from our network. This is something that you would commonly do to be sure that someone does not try to evade the packet filter, by generating nefarious packets that look like they are from the inside. The problem with that is that there is _at least_ one host on the outside interface that you do not want to ignore: the router. But usually, the ISP anti-spoofs at their router, so we do not need to bother that much. The last rule seems to be an exact duplicate of the default rule, that is, do not let anything pass that is not specifically allowed. But there is a difference: all suspected traffic will be logged. There are two rules for passing SMTP and DNS traffic towards the mail server and the name server, if you have them. Obviously the whole rule set should be flavored to personal taste, this is only a specific example (rule format is described accurately in the man:ipfw[8] man page). Note that in order for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled. This is an example of making sure that you set the IP on the correct network card. Alternatively it is possible to specify the IP address instead of the host name (required if the machine is IP-less). People that are used to setting up firewalls are probably also used to either having a `reset` or a `forward` rule for ident packets (TCP port 113). Unfortunately, this is not an applicable option with the bridge, so the best thing is to simply pass them to their destination. As long as that destination machine is not running an ident daemon, this is relatively harmless. The alternative is dropping connections on port 113, which creates some problems with services like IRC (the ident probe must timeout). The only other thing that is a little weird that you may have noticed is that there is a rule to let the bridge machine speak, and another for internal hosts. Remember that this is because the two sets of traffic will take different paths through the kernel and into the packet filter. The inside net will go through the bridge, while the local machine will use the normal IP stack to speak. Thus the two rules to handle the different cases. The `in via fxp0` rules work for both paths. In general, if you use `in via` rules throughout the filter, you will need to make an exception for locally generated packets, because they did not come in via any of our interfaces. [[filtering-bridges-contributors]] == Contributors Many parts of this article have been taken, updated and adapted from an old text about bridging, edited by Nick Sayer. A pair of inspirations are due to an introduction on bridging by Steve Peterson. A big thanks to Luigi Rizzo for the implementation of the bridge code in FreeBSD and for the time he has dedicated to me answering all of my related questions. A thanks goes out also to Tom Rhodes who looked over my job of translation from Italian (the original language of this article) into English. diff --git a/documentation/content/en/articles/freebsd-questions/_index.adoc b/documentation/content/en/articles/freebsd-questions/_index.adoc index d8c5f3b7bf..1edcd2ecf0 100644 --- a/documentation/content/en/articles/freebsd-questions/_index.adoc +++ b/documentation/content/en/articles/freebsd-questions/_index.adoc @@ -1,257 +1,269 @@ --- title: How to get Best Results from the FreeBSD-questions Mailing List authors: - author: Greg Lehey email: grog@FreeBSD.org description: How to get Best Results from the FreeBSD-questions Mailing List trademarks: ["freebsd", "microsoft", "opengroup", "qualcomm", "general"] tags: ["questions", "mailing", "FreeBSD"] --- = How to get Best Results from the FreeBSD-questions Mailing List :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This document provides useful information for people looking to prepare an e-mail to the FreeBSD-questions mailing list. Advice and hints are given that will maximize the chance that the reader will receive useful replies. This document is regularly posted to the FreeBSD-questions mailing list. ''' toc::[] == Introduction `FreeBSD-questions` is a mailing list maintained by the FreeBSD project to help people who have questions about the normal use of FreeBSD. Another group, `FreeBSD-hackers`, discusses more advanced questions such as future development work. [NOTE] ==== The term "hacker" has nothing to do with breaking into other people's computers. The correct term for the latter activity is "cracker", but the popular press has not found out yet. The FreeBSD hackers disapprove strongly of cracking security, and have nothing to do with it. For a longer description of hackers, see Eric Raymond's http://www.catb.org/~esr/faqs/hacker-howto.html[How To Become A Hacker] ==== This is a regular posting aimed to help both those seeking advice from FreeBSD-questions (the "newcomers"), and also those who answer the questions (the "hackers"). Inevitably there is some friction, which stems from the different viewpoints of the two groups. The newcomers accuse the hackers of being arrogant, stuck-up, and unhelpful, while the hackers accuse the newcomers of being stupid, unable to read plain English, and expecting everything to be handed to them on a silver platter. Of course, there is an element of truth in both these claims, but for the most part these viewpoints come from a sense of frustration. In this document, I would like to do something to relieve this frustration and help everybody get better results from FreeBSD-questions. In the following section, I recommend how to submit a question; after that, we will look at how to answer one. == How to Subscribe to FreeBSD-questions FreeBSD-questions is a mailing list, so you need mail access. Point your WWW browser to the {freebsd-questions}. In the section titled "Subscribing to freebsd-questions" fill in the "Your email address" field; the other fields are optional. [NOTE] ==== The password fields in the subscription form provide only mild security, but should prevent others from messing with your subscription. _Do not use a valuable password_ as it will occasionally be emailed back to you in cleartext. ==== You will receive a confirmation message from mailman; follow the included instructions to complete your subscription. Finally, when you get the "Welcome" message from mailman telling you the details of the list and subscription area password, __please save it__. If you ever should want to leave the list, you will need the information there. See the next section for more details. == How to Unsubscribe from FreeBSD-questions When you subscribed to FreeBSD-questions, you got a welcome message from mailman. In this message, amongst other things, it told you how to unsubscribe. Here is a typical message: .... Welcome to the freebsd-questions@freebsd.org mailing list! To post to this list, send your email to: freebsd-questions@freebsd.org General information about the mailing list is at: https://lists.freebsd.org/mailman/listinfo/freebsd-questions If you ever want to unsubscribe or change your options (e.g., switch to or from digest mode, change your password, etc.), visit your subscription page at: https://lists.freebsd.org/mailman/options/freebsd-questions/grog%40lemsi.de You can also make such adjustments via email by sending a message to: freebsd-questions-request@freebsd.org with the word 'help' in the subject or body (do not include the quotes), and you will get back a message with instructions. You must know your password to change your options (including changing the password, itself) or to unsubscribe. It is: 12345 Normally, Mailman will remind you of your freebsd.org mailing list passwords once every month, although you can disable this if you prefer. This reminder will also include instructions on how to unsubscribe or change your account options. There is also a button on your options page that will email your current password to you. .... From the URL specified in your "Welcome" message you may visit the "Account management page" and enter a request to "Unsubscribe" you from FreeBSD-questions mailing list. A confirmation message will be sent to you from mailman; follow the included instructions to finish unsubscribing. If you have done this, and you still cannot figure out what is going on, send a message to mailto:freebsd-questions-request@FreeBSD.org[freebsd-questions-request@FreeBSD.org], and they will sort things out for you. _Do not_ send a message to FreeBSD-questions: they cannot help you. == Should I ask `-questions` or `-hackers`? Two mailing lists handle general questions about FreeBSD, `FreeBSD-questions` and `FreeBSD-hackers`. In some cases, it is not really clear which group you should ask. The following criteria should help for 99% of all questions, however: . If the question is of a general nature, ask `FreeBSD-questions`. Examples might be questions about installing FreeBSD or the use of a particular UNIX(R) utility. . If you think the question relates to a bug, but you are not sure, or you do not know how to look for it, send the message to `FreeBSD-questions`. . If the question relates to a bug, and you are _sure_ that it is a bug (for example, you can pinpoint the place in the code where it happens, and you maybe have a fix), then send the message to `FreeBSD-hackers`. . If the question relates to enhancements to FreeBSD, and you can make suggestions about how to implement them, then send the message to `FreeBSD-hackers`. There are also a number of other link:{handbook}#eresources-mail[specialized mailing lists], which caters to more specific interests. The criteria above still apply, and it is in your interest to stick to them, since you are more likely to get good results that way. == Before Submitting a Question You can (and should) do some things yourself before asking a question on one of the mailing lists: * Try solving the problem on your own. If you post a question which shows that you have tried to solve the problem, your question will generally attract more positive attention from people reading it. Trying to solve the problem yourself will also enhance your understanding of FreeBSD, and will eventually let you use your knowledge to help others by answering questions posted to the mailing lists. * Read the manual pages, and the FreeBSD documentation (either installed in [.filename]#/usr/doc# or accessible via WWW at http://www.FreeBSD.org[http://www.FreeBSD.org]), especially the link:{handbook}[handbook] and the link:{faq}[FAQ]. * Browse and/or search the archives for the mailing list, to see if your question or a similar one has been asked (and possibly answered) on the list. You can browse and/or search the mailing list archives at https://www.FreeBSD.org/mail[https://www.FreeBSD.org/mail] and https://www.FreeBSD.org/search/#mailinglists[https://www.FreeBSD.org/search/#mailinglists] respectively. This can be done at other WWW sites as well, for example at http://marc.theaimsgroup.com[http://marc.theaimsgroup.com]. * Use a search engine such as http://www.google.com[Google] or http://www.yahoo.com[Yahoo] to find answers to your question. == How to Submit a Question When submitting a question to FreeBSD-questions, consider the following points: * Remember that nobody gets paid for answering a FreeBSD question. They do it of their own free will. You can influence this free will positively by submitting a well-formulated question supplying as much relevant information as possible. You can influence this free will negatively by submitting an incomplete, illegible, or rude question. It is perfectly possible to send a message to FreeBSD-questions and not get an answer even if you follow these rules. It is much more possible to not get an answer if you do not. In the rest of this document, we will look at how to get the most out of your question to FreeBSD-questions. * Not everybody who answers FreeBSD questions reads every message: they look at the subject line and decide whether it interests them. Clearly, it is in your interest to specify a subject. "FreeBSD problem" or "Help" are not enough. If you provide no subject at all, many people will not bother reading it. If your subject is not specific enough, the people who can answer it may not read it. * Format your message so that it is legible, and PLEASE DO NOT SHOUT!!!!!. We appreciate that a lot of people do not speak English as their first language, and we try to make allowances for that, but it is really painful to try to read a message written full of typos or without any line breaks. + Do not underestimate the effect that a poorly formatted mail message has, not just on the FreeBSD-questions mailing list. Your mail message is all people see of you, and if it is poorly formatted, one line per paragraph, badly spelt, or full of errors, it will give people a poor impression of you. + A lot of badly formatted messages come from http://www.lemis.com/email.html[bad mailers or badly configured mailers]. The following mailers are known to send out badly formatted messages without you finding out about them: ** Eudora(R) ** exmh ** Microsoft(R) Exchange ** Microsoft(R) Outlook(R) + Try not to use MIME: a lot of people use mailers which do not get on very well with MIME. * Make sure your time and time zone are set correctly. This may seem a little silly, since your message still gets there, but many of the people you are trying to reach get several hundred messages a day. They frequently sort the incoming messages by subject and by date, and if your message does not come before the first answer, they may assume they missed it and not bother to look. * Do not include unrelated questions in the same message. Firstly, a long message tends to scare people off, and secondly, it is more difficult to get all the people who can answer all the questions to read the message. * Specify as much information as possible. This is a difficult area, and we need to expand on what information you need to submit, but here is a start: ** In nearly every case, it is important to know the version of FreeBSD you are running. This is particularly the case for FreeBSD-CURRENT, where you should also specify the date of the sources, though of course you should not be sending questions about -CURRENT to FreeBSD-questions. ** With any problem which _could_ be hardware related, tell us about your hardware. In case of doubt, assume it is possible that it is hardware. What kind of CPU are you using? How fast? What motherboard? How much memory? What peripherals? + There is a judgement call here, of course, but the output of the man:dmesg[8] command can frequently be very useful, since it tells not just what hardware you are running, but what version of FreeBSD as well. ** If you get error messages, do not say "I get error messages", say (for example) "I get the error message 'No route to host'". ** If your system panics, do not say "My system panicked", say (for example) "my system panicked with the message 'free vnode isn't'". ** If you have difficulty installing FreeBSD, please tell us what hardware you have. In particular, it is important to know the IRQs and I/O addresses of the boards installed in your machine. ** If you have difficulty getting PPP to run, describe the configuration. Which version of PPP do you use? What kind of authentication do you have? Do you have a static or dynamic IP address? What kind of messages do you get in the log file? * A lot of the information you need to supply is the output of programs, such as man:dmesg[8], or console messages, which usually appear in [.filename]#/var/log/messages#. Do not try to copy this information by typing it in again; it is a real pain, and you are bound to make a mistake. To send log file contents, either make a copy of the file and use an editor to trim the information to what is relevant, or cut and paste into your message. For the output of programs like man:dmesg[8], redirect the output to a file and include that. For example, + [source,shell] .... % dmesg > /tmp/dmesg.out .... + This redirects the information to the file [.filename]#/tmp/dmesg.out#. * If you do all this, and you still do not get an answer, there could be other reasons. For example, the problem is so complicated that nobody knows the answer, or the person who does know the answer was offline. If you do not get an answer after, say, a week, it might help to re-send the message. If you do not get an answer to your second message, though, you are probably not going to get one from this forum. Resending the same message again and again will only make you unpopular. To summarize, let's assume you know the answer to the following question (yes, it is the same one in each case). You choose which of these two questions you would be more prepared to answer: .Message 1 [example] ==== .... Subject: HELP!!?!?? I just can't get hits damn silly FereBSD system to workd, and Im really good at this tsuff, but I have never seen anythign sho difficult to install, it jst wont work whatever I try so why don't you guys tell me what I doing wrong. .... ==== .Message 2 [example] ==== .... Subject: Problems installing FreeBSD I've just got the FreeBSD 2.1.5 CDROM from Walnut Creek, and I'm having a lot of difficulty installing it. I have a 66 MHz 486 with 16 MB of memory and an Adaptec 1540A SCSI board, a 1.2GB Quantum Fireball disk and a Toshiba 3501XA CDROM drive. The installation works just fine, but when I try to reboot the system, I get the message Missing Operating System. .... ==== == How to Follow up to a Question Often you will want to send in additional information to a question you have already sent. The best way to do this is to reply to your original message. This has three advantages: . You include the original message text, so people will know what you are talking about. Do not forget to trim unnecessary text out, though. . The text in the subject line stays the same (you did remember to put one in, did you not?). Many mailers will sort messages by subject. This helps group messages together. . The message reference numbers in the header will refer to the previous message. Some mailers, such as http://www.mutt.org/[mutt], can _thread_ messages, showing the exact relationships between the messages. == How to Answer a Question Before you answer a question to FreeBSD-questions, consider: . A lot of the points on submitting questions also apply to answering questions. Read them. . Has somebody already answered the question? The easiest way to check this is to sort your incoming mail by subject: then (hopefully) you will see the question followed by any answers, all together. + If somebody has already answered it, it does not automatically mean that you should not send another answer. But it makes sense to read all the other answers first. . Do you have something to contribute beyond what has already been said? In general, "Yeah, me too" answers do not help much, although there are exceptions, like when somebody is describing a problem they are having, and they do not know whether it is their fault or whether there is something wrong with the hardware or software. If you do send a "me too" answer, you should also include any further relevant information. . Are you sure you understand the question? Very frequently, the person who asks the question is confused or does not express themselves very well. Even with the best understanding of the system, it is easy to send a reply which does not answer the question. This does not help: you will leave the person who submitted the question more frustrated or confused than ever. If nobody else answers, and you are not too sure either, you can always ask for more information. . Are you sure your answer is correct? If not, wait a day or so. If nobody else comes up with a better answer, you can still reply and say, for example, "I do not know if this is correct, but since nobody else has replied, why don't you try replacing your ATAPI CDROM with a frog?". . Unless there is a good reason to do otherwise, reply to the sender and to FreeBSD-questions. Many people on the FreeBSD-questions are "lurkers": they learn by reading messages sent and replied to by others. If you take a message which is of general interest off the list, you are depriving these people of their information. Be careful with group replies; lots of people send messages with hundreds of CCs. If this is the case, be sure to trim the Cc: lines appropriately. . Include relevant text from the original message. Trim it to the minimum, but do not overdo it. It should still be possible for somebody who did not read the original message to understand what you are talking about. . Use some technique to identify which text came from the original message, and which text you add. I personally find that prepending "`>`" to the original message works best. Leaving white space after the "`> ;`" and leave empty lines between your text and the original text both make the result more readable. . Put your response in the correct place (after the text to which it replies). It is very difficult to read a thread of responses where each reply comes before the text to which it replies. . Most mailers change the subject line on a reply by prepending a text such as "Re: ". If your mailer does not do it automatically, you should do it manually. . If the submitter did not abide by format conventions (lines too long, inappropriate subject line) _please_ fix it. In the case of an incorrect subject line (such as "HELP!!??"), change the subject line to (say) "Re: Difficulties with sync PPP (was: HELP!!??)". That way other people trying to follow the thread will have less difficulty following it. + In such cases, it is appropriate to say what you did and why you did it, but try not to be rude. If you find you can not answer without being rude, do not answer. + If you just want to reply to a message because of its bad format, just reply to the submitter, not to the list. You can just send him this message in reply, if you like. diff --git a/documentation/content/en/articles/freebsd-update-server/_index.adoc b/documentation/content/en/articles/freebsd-update-server/_index.adoc index 90a3702717..dab8525828 100644 --- a/documentation/content/en/articles/freebsd-update-server/_index.adoc +++ b/documentation/content/en/articles/freebsd-update-server/_index.adoc @@ -1,613 +1,625 @@ --- title: Build Your Own FreeBSD Update Server authors: - author: Jason Helfman email: jgh@FreeBSD.org copyright: 2009-2011, 2013 Jason Helfman description: Building your own freebsd-update server allows a system administrator to perform fast updates for a number of machines from a local mirror trademarks: ["freebsd", "amd", "intel", "general"] tags: ["FreeBSD", "Update", "Server", "internal"] --- = Build Your Own FreeBSD Update Server :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article describes building an internal FreeBSD Update Server. The https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] is written by `{cperciva}`, Security Officer Emeritus of FreeBSD. For users that think it is convenient to update their systems against an official update server, building their own FreeBSD Update Server may help to extend its functionality by supporting manually-tweaked FreeBSD releases or by providing a local mirror that will allow faster updates for a number of machines. ''' toc::[] [[acknowledgments]] == Acknowledgments This article was subsequently printed at https://people.freebsd.org/~jgh/files/fus/BSD_03_2010_EN.pdf[BSD Magazine]. [[introduction]] == Introduction Experienced users or administrators are often responsible for several machines or environments. They understand the difficult demands and challenges of maintaining such an infrastructure. Running a FreeBSD Update Server makes it easier to deploy security and software patches to selected test machines before rolling them out to production. It also means a number of systems can be updated from the local network rather than a potentially slower Internet connection. This article outlines the steps involved in creating an internal FreeBSD Update Server. [[prerequisites]] == Prerequisites To build an internal FreeBSD Update Server some requirements should be met. * A running FreeBSD system. + [NOTE] ==== At a minimum, updates require building on a FreeBSD release greater than or equal to the target release version for distribution. ==== * A user account with at least 4 GB of available space. This will allow the creation of updates for 7.1 and 7.2, but the exact space requirements may change from version to version. * An man:ssh[1] account on a remote machine to upload distributed updates. * A web server, like link:{handbook}#network-apache[Apache], with over half of the space required for the build. For instance, test builds for 7.1 and 7.2 consume a total amount of 4 GB, and the webserver space needed to distribute these updates is 2.6 GB. * Basic knowledge of shell scripting with Bourne shell, man:sh[1]. [[Configuration]] == Configuration: Installation & Setup Download the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] software by installing package:devel/subversion[] and package:security/ca_root_nss[], and execute: [source,shell] .... % svn co https://svn.freebsd.org/base/user/cperciva/freebsd-update-build freebsd-update-server .... Update [.filename]#scripts/build.conf# appropriately. It is sourced during all build operations. Here is the default [.filename]#build.conf#, which should be modified to suit your environment. [.programlisting] .... # Main configuration file for FreeBSD Update builds. The # release-specific configuration data is lower down in # the scripts tree. # Location from which to fetch releases export FTP=ftp://ftp2.freebsd.org/pub/FreeBSD/releases <.> # Host platform export HOSTPLATFORM=`uname -m` # Host name to use inside jails export BUILDHOSTNAME=${HOSTPLATFORM}-builder.daemonology.net <.> # Location of SSH key export SSHKEY=/root/.ssh/id_dsa <.> # SSH account into which files are uploaded MASTERACCT=builder@wadham.daemonology.net <.> # Directory into which files are uploaded MASTERDIR=update-master.freebsd.org <.> .... Parameters for consideration would be: <.> This is the location where ISO images are downloaded from (by the `fetchiso()` subroutine of [.filename]#scripts/build.subr#). The location configured is not limited to FTP URIs. Any URI scheme supported by standard man:fetch[1] utility should work fine. Customizations to the `fetchiso()` code can be installed by copying the default [.filename]#build.subr# script to the release and architecture-specific area at [.filename]#scripts/RELEASE/ARCHITECTURE/build.subr# and applying local changes. <.> The name of the build host. This information will be displayed on updated systems when issuing: + [source,shell] .... % uname -v .... + <.> The SSH key for uploading files to the update server. A key pair can be created by typing `ssh-keygen -t dsa`. This parameter is optional; standard password authentication will be used as a fallback authentication method when `SSHKEY` is not defined. The man:ssh-keygen[1] manual page has more detailed information about SSH and the appropriate steps for creating and using one. <.> Account for uploading files to the update server. <.> Directory on the update server where files are uploaded to. The default [.filename]#build.conf# shipped with the freebsd-update-server sources is suitable for building i386 releases of FreeBSD. As an example of building an update server for other architectures, the following steps outline the configuration changes needed for amd64: [.procedure] ==== . Create a build environment for amd64: + [source,shell] .... % mkdir -p /usr/local/freebsd-update-server/scripts/7.2-RELEASE/amd64 .... . Install a [.filename]#build.conf# in the newly created build directory. The build configuration options for FreeBSD 7.2-RELEASE on amd64 should be similar to: + [.programlisting] .... # SHA256 hash of RELEASE disc1.iso image. export RELH=1ea1f6f652d7c5f5eab7ef9f8edbed50cb664b08ed761850f95f48e86cc71ef5 <.> # Components of the world, source, and kernels export WORLDPARTS="base catpages dict doc games info manpages proflibs lib32" export SOURCEPARTS="base bin contrib crypto etc games gnu include krb5 \ lib libexec release rescue sbin secure share sys tools \ ubin usbin cddl" export KERNELPARTS="generic" # EOL date export EOL=1275289200 <.> .... + <.> The man:sha256[1] hash key for the desired release, is published within the respective link:https://www.FreeBSD.org/releases/[release announcement]. <.> To generate the "End of Life" number for [.filename]#build.conf#, refer to the "Estimated EOL" posted on the link:https://www.FreeBSD.org/security/security/[FreeBSD Security Website]. The value of `EOL` can be derived from the date listed on the web site, using the man:date[1] utility, for example: + [source,shell] .... % date -j -f '%Y%m%d-%H%M%S' '20090401-000000' '+%s' .... ==== [[build]] == Building Update Code The first step is to run [.filename]#scripts/make.sh#. This will build some binaries, create directories, and generate an RSA signing key used for approving builds. In this step, a passphrase will have to be supplied for the final creation of the signing key. [source,shell] .... # sh scripts/make.sh cc -O2 -fno-strict-aliasing -pipe findstamps.c -o findstamps findstamps.c: In function 'usage': findstamps.c:45: warning: incompatible implicit declaration of built-in function 'exit' cc -O2 -fno-strict-aliasing -pipe unstamp.c -o unstamp install findstamps ../bin install unstamp ../bin rm -f findstamps unstamp Generating RSA private key, 4096 bit long modulus ................................................................................++ ...................++ e is 65537 (0x10001) Public key fingerprint: 27ef53e48dc869eea6c3136091cc6ab8589f967559824779e855d58a2294de9e Encrypting signing key for root enter aes-256-cbc encryption password: Verifying - enter aes-256-cbc encryption password: .... [NOTE] ==== Keep a note of the generated key fingerprint. This value is required in [.filename]#/etc/freebsd-update.conf# for binary updates. ==== At this point, we are ready to stage a build. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/init.sh amd64 7.2-RELEASE .... What follows is a sample of an _initial_ build run. [source,shell] .... # sh scripts/init.sh amd64 7.2-RELEASE Mon Aug 24 16:04:36 PDT 2009 Starting fetch for FreeBSD/amd64 7.2-RELEASE /usr/local/freebsd-update-server/work/7.2-RELE100 of 588 MB 359 kBps 00m00s Mon Aug 24 16:32:38 PDT 2009 Verifying disc1 hash for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:32:44 PDT 2009 Extracting components for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:34:05 PDT 2009 Constructing world+src image for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:35:57 PDT 2009 Extracting world+src for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 23:36:24 UTC 2009 Building world for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:31:29 UTC 2009 Distributing world for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:32:36 UTC 2009 Building and distributing kernels for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:44:44 UTC 2009 Constructing world components for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:44:56 UTC 2009 Distributing source for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:46:18 PDT 2009 Moving components into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:46:33 PDT 2009 Identifying extra documentation for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:47:13 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:47:18 PDT 2009 Indexing release for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:50:44 PDT 2009 Indexing world0 for FreeBSD/amd64 7.2-RELEASE Files built but not released: Files released but not built: Files which differ by more than contents: Files which differ between release and build: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel src|sys|/sys/conf/newvers.sh world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... .... Then the build of the world is performed again, with world patches. A more detailed explanation may be found in [.filename]#scripts/build.subr#. [WARNING] ==== During this second build cycle, the network time protocol daemon, man:ntpd[8], is turned off. Per `{cperciva}`, Security Officer Emeritus of FreeBSD, "the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] build code needs to identify timestamps which are stored in files so that they can be ignored when comparing builds to determine which files need to be updated. This timestamp-finding works by doing two builds 400 days apart and comparing the results." ==== [source,shell] .... Mon Aug 24 17:54:07 PDT 2009 Extracting world+src for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 00:54:34 UTC 2010 Building world for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 01:49:42 UTC 2010 Distributing world for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 01:50:50 UTC 2010 Building and distributing kernels for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 02:02:56 UTC 2010 Constructing world components for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 02:03:08 UTC 2010 Distributing source for FreeBSD/amd64 7.2-RELEASE Tue Sep 28 19:04:31 PDT 2010 Moving components into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:04:46 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:04:51 PDT 2009 Indexing world1 for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:08:04 PDT 2009 Locating build stamps for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:19 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:19 PDT 2009 Preparing to copy files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:20 PDT 2009 Copying data files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 12:16:57 PDT 2009 Copying metadata files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 12:16:59 PDT 2009 Constructing metadata index and tag for FreeBSD/amd64 7.2-RELEASE Files found which include build stamps: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/include/osreldate.h world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... .... Finally, the build completes. [source,shell] .... Values of build stamps, excluding library archive headers: v1.2 (Aug 25 2009 00:40:36) v1.2 (Aug 25 2009 00:38:22) @()FreeBSD 7.2-RELEASE 0: Tue Aug 25 00:38:29 UTC 2009 FreeBSD 7.2-RELEASE 0: Tue Aug 25 00:38:29 UTC 2009 root@server.myhost.com:/usr/obj/usr/src/sys/GENERIC 7.2-RELEASE Mon Aug 24 23:55:25 UTC 2009 Mon Aug 24 23:55:25 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 Mon Aug 24 23:46:47 UTC 2009 ntpq 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) * Copyright (c) 1992-2009 The FreeBSD Project. Mon Aug 24 23:46:47 UTC 2009 Mon Aug 24 23:55:40 UTC 2009 Aug 25 2009 ntpd 4.2.4p5-a Mon Aug 24 23:55:52 UTC 2009 (1) ntpdate 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) ntpdc 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) Tue Aug 25 00:21:21 UTC 2009 Tue Aug 25 00:21:21 UTC 2009 Tue Aug 25 00:21:21 UTC 2009 Mon Aug 24 23:46:47 UTC 2009 FreeBSD/amd64 7.2-RELEASE initialization build complete. Please review the list of build stamps printed above to confirm that they look sensible, then run sh -e approve.sh amd64 7.2-RELEASE to sign the release. .... Approve the build if everything is correct. More information on determining this can be found in the distributed source file named [.filename]#USAGE#. Execute [.filename]#scripts/approve.sh#, as directed. This will sign the release, and move components into a staging area suitable for uploading. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/mountkey.sh .... [source,shell] .... # sh -e scripts/approve.sh amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Signing build for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to patch source directories for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to upload staging area for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:07 PDT 2009 Updating databases for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:07 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.2-RELEASE .... After the approval process is complete, the upload procedure may be started. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/upload.sh amd64 7.2-RELEASE .... [NOTE] ==== In the event update code needs to be re-uploaded, this may be done by changing to the public distributions directory for the target release and updating attributes of the _uploaded_ file. [source,shell] .... # cd /usr/local/freebsd-update-server/pub/7.2-RELEASE/amd64 # touch -t 200801010101.01 uploaded .... ==== The uploaded files will need to be in the document root of the webserver in order for updates to be distributed. The exact configuration will vary depending on the web server used. For the Apache web server, please refer to the link:{handbook}#network-apache[Configuration of Apache servers] section in the Handbook. Update client's `KeyPrint` and `ServerName` in [.filename]#/etc/freebsd-update.conf#, and perform updates as instructed in the link:{handbook}#updating-upgrading-freebsdupdate[FreeBSD Update] section of the Handbook. [IMPORTANT] ==== In order for FreeBSD Update Server to work properly, updates for both the _current_ release and the release _one wants to upgrade to_ need to be built. This is necessary for determining the differences of files between releases. For example, when upgrading a FreeBSD system from 7.1-RELEASE to 7.2-RELEASE, updates will need to be built and uploaded to your distribution server for both versions. ==== For reference, the entire run of link:../../source/articles/freebsd-update-server/init.txt[init.sh] is attached. [[patch]] == Building a Patch Every time a link:https://www.FreeBSD.org/security/advisories/[security advisory] or link:https://www.FreeBSD.org/security/notices/[security notice] is announced, a patch update can be built. For this example, 7.1-RELEASE will be used. A couple of assumptions are made for a different release build: * Setup the correct directory structure for the initial build. * Perform an initial build for 7.1-RELEASE. Create the patch directory of the respective release under [.filename]#/usr/local/freebsd-update-server/patches/#. [source,shell] .... % mkdir -p /usr/local/freebsd-update-server/patches/7.1-RELEASE/ % cd /usr/local/freebsd-update-server/patches/7.1-RELEASE .... As an example, take the patch for man:named[8]. Read the advisory, and grab the necessary file from link:https://www.FreeBSD.org/security/advisories/[FreeBSD Security Advisories]. More information on interpreting the advisory, can be found in the link:{handbook}#security-advisories[FreeBSD Handbook]. In the https://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc[security brief], this advisory is called `SA-09:12.bind`. After downloading the file, it is required to rename the file to an appropriate patch level. It is suggested to keep this consistent with official FreeBSD patch levels, but its name may be freely chosen. For this build, let us follow the currently established practice of FreeBSD and call this `p7`. Rename the file: [source,shell] .... % cd /usr/local/freebsd-update-server/patches/7.1-RELEASE/; mv bind.patch 7-SA-09:12.bind .... [NOTE] ==== When running a patch level build, it is assumed that previous patches are in place. When a patch build is run, it will run all patches contained in the patch directory. There can be custom patches added to any build. Use the number zero, or any other number. ==== [WARNING] ==== It is up to the administrator of the FreeBSD Update Server to take appropriate measures to verify the authenticity of every patch. ==== At this point, a _diff_ is ready to be built. The software checks first to see if a [.filename]#scripts/init.sh# has been run on the respective release prior to running the diff build. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/diff.sh amd64 7.1-RELEASE 7 .... What follows is a sample of a _differential_ build run. [source,shell] .... # sh -e scripts/diff.sh amd64 7.1-RELEASE 7 Wed Aug 26 10:09:59 PDT 2009 Extracting world+src for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 17:10:25 UTC 2009 Building world for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:05:11 UTC 2009 Distributing world for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:06:16 UTC 2009 Building and distributing kernels for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:17:50 UTC 2009 Constructing world components for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:18:02 UTC 2009 Distributing source for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:23 PDT 2009 Moving components into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:37 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:42 PDT 2009 Indexing world0 for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:23:02 PDT 2009 Extracting world+src for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 18:23:29 UTC 2010 Building world for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:18:15 UTC 2010 Distributing world for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:19:18 UTC 2010 Building and distributing kernels for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:30:52 UTC 2010 Constructing world components for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:31:03 UTC 2010 Distributing source for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 12:32:25 PDT 2010 Moving components into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:32:39 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:32:43 PDT 2009 Indexing world1 for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:35:54 PDT 2009 Locating build stamps for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:36:58 PDT 2009 Reverting changes due to build stamps for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:14 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:14 PDT 2009 Preparing to copy files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:15 PDT 2009 Copying data files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:43:23 PDT 2009 Copying metadata files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:43:25 PDT 2009 Constructing metadata index and tag for FreeBSD/amd64 7.1-RELEASE-p7 ... Files found which include build stamps: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/include/osreldate.h world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... Values of build stamps, excluding library archive headers: v1.2 (Aug 26 2009 18:13:46) v1.2 (Aug 26 2009 18:11:44) @()FreeBSD 7.1-RELEASE-p7 0: Wed Aug 26 18:11:50 UTC 2009 FreeBSD 7.1-RELEASE-p7 0: Wed Aug 26 18:11:50 UTC 2009 root@server.myhost.com:/usr/obj/usr/src/sys/GENERIC 7.1-RELEASE-p7 Wed Aug 26 17:29:15 UTC 2009 Wed Aug 26 17:29:15 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 Wed Aug 26 17:20:39 UTC 2009 ntpq 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) * Copyright (c) 1992-2009 The FreeBSD Project. Wed Aug 26 17:20:39 UTC 2009 Wed Aug 26 17:29:30 UTC 2009 Aug 26 2009 ntpd 4.2.4p5-a Wed Aug 26 17:29:41 UTC 2009 (1) ntpdate 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) ntpdc 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:20:39 UTC 2009 ... .... Updates are printed, and approval is requested. [source,shell] .... New updates: kernel|generic|/GENERIC/kernel.symbols|f|0|0|0555|0|7c8dc176763f96ced0a57fc04e7c1b8d793f27e006dd13e0b499e1474ac47e10| kernel|generic|/GENERIC/kernel|f|0|0|0555|0|33197e8cf15bbbac263d17f39c153c9d489348c2c534f7ca1120a1183dec67b1| kernel|generic|/|d|0|0|0755|0|| src|base|/|d|0|0|0755|0|| src|bin|/|d|0|0|0755|0|| src|cddl|/|d|0|0|0755|0|| src|contrib|/contrib/bind9/bin/named/update.c|f|0|10000|0644|0|4d434abf0983df9bc47435670d307fa882ef4b348ed8ca90928d250f42ea0757| src|contrib|/contrib/bind9/lib/dns/openssldsa_link.c|f|0|10000|0644|0|c6805c39f3da2a06dd3f163f26c314a4692d4cd9a2d929c0acc88d736324f550| src|contrib|/contrib/bind9/lib/dns/opensslrsa_link.c|f|0|10000|0644|0|fa0f7417ee9da42cc8d0fd96ad24e7a34125e05b5ae075bd6e3238f1c022a712| ... FreeBSD/amd64 7.1-RELEASE update build complete. Please review the list of build stamps printed above and the list of updated files to confirm that they look sensible, then run sh -e approve.sh amd64 7.1-RELEASE to sign the build. .... Follow the same process as noted before for approving a build: [source,shell] .... # sh -e scripts/approve.sh amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Signing build for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to patch source directories for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to upload staging area for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:07 PDT 2009 Updating databases for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:07 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.1-RELEASE The FreeBSD/amd64 7.1-RELEASE update build has been signed and is ready to be uploaded. Remember to run sh -e umountkey.sh to unmount the decrypted key once you have finished signing all the new builds. .... After approving the build, upload the software: [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/upload.sh amd64 7.1-RELEASE .... For reference, the entire run of link:../../source/articles/freebsd-update-server/diff.txt[diff.sh] is attached. [[tips]] == Tips * If a custom release is built using the native `make release` link:{releng}#release-build[procedure], freebsd-update-server code will work from your release. As an example, a release without ports or documentation can be built by clearing functionality pertaining to documentation subroutines `findextradocs ()`, `addextradocs ()` and altering the download location in `fetchiso ()`, respectively, in [.filename]#scripts/build.subr#. As a last step, change the man:sha256[1] hash in [.filename]#build.conf# under your respective release and architecture and you are ready to build off your custom release. + [.programlisting] .... # Compare ${WORKDIR}/release and ${WORKDIR}/$1, identify which parts # of the world|doc subcomponent are missing from the latter, and # build a tarball out of them. findextradocs () { } # Add extra docs to ${WORKDIR}/$1 addextradocs () { } .... * Adding `-j _NUMBER_` flags to `buildworld` and `obj` targets in the [.filename]#scripts/build.subr# script may speed up processing depending on the hardware used, however it is not necessary. Using these flags in other targets is not recommended, as it may cause the build to become unreliable. + [.programlisting] .... # Build the world log "Building world" cd /usr/src && make -j 2 ${COMPATFLAGS} buildworld 2>&1 # Distribute the world log "Distributing world" cd /usr/src/release && make -j 2 obj && make ${COMPATFLAGS} release.1 release.2 2>&1 .... * Create an appropriate link:{handbook}#network-dns[DNS] SRV record for the update server, and put others behind it with variable weights. Using this facility will provide update mirrors, however this tip is not necessary unless you wish to provide a redundant service. + [.programlisting] .... _http._tcp.update.myserver.com. IN SRV 0 2 80 host1.myserver.com. IN SRV 0 1 80 host2.myserver.com. IN SRV 0 0 80 host3.myserver.com. .... diff --git a/documentation/content/en/articles/geom-class/_index.adoc b/documentation/content/en/articles/geom-class/_index.adoc index 77cff44e57..3d836c51f3 100644 --- a/documentation/content/en/articles/geom-class/_index.adoc +++ b/documentation/content/en/articles/geom-class/_index.adoc @@ -1,403 +1,413 @@ --- title: Writing a GEOM Class authors: - author: Ivan Voras email: ivoras@FreeBSD.org description: A guide to GEOM internals, and writing your own GEOM class trademarks: ["freebsd", "intel", "general"] tags: ["GEOM", "kernel", "modules", "FreeBSD"] --- = Writing a GEOM Class :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This text documents some starting points in developing GEOM classes, and kernel modules in general. It is assumed that the reader is familiar with C userland programming. ''' toc::[] [[intro]] == Introduction [[intro-docs]] === Documentation Documentation on kernel programming is scarce - it is one of few areas where there is nearly nothing in the way of friendly tutorials, and the phrase "use the source!" really holds true. However, there are some bits and pieces (some of them seriously outdated) floating around that should be studied before beginning to code: * The link:{developers-handbook}[FreeBSD Developer's Handbook] - part of the documentation project, it does not contain anything specific to kernel programming, but rather some general useful information. * The link:{arch-handbook}[FreeBSD Architecture Handbook] - also from the documentation project, contains descriptions of several low-level facilities and procedures. The most important chapter is 13, link:{arch-handbook}#driverbasics[Writing FreeBSD device drivers]. * The Blueprints section of http://www.freebsddiary.org[FreeBSD Diary] web site - contains several interesting articles on kernel facilities. * The man pages in section 9 - for important documentation on kernel functions. * The man:geom[4] man page and http://phk.freebsd.dk/pubs/[PHK's GEOM slides] - for general introduction of the GEOM subsystem. * Man pages man:g_bio[9], man:g_event[9], man:g_data[9], man:g_geom[9], man:g_provider[9], man:g_consumer[9], man:g_access[9] & others linked from those, for documentation on specific functionalities. * The man:style[9] man page - for documentation on the coding-style conventions which must be followed for any code which is to be committed to the FreeBSD tree. [[prelim]] == Preliminaries The best way to do kernel development is to have (at least) two separate computers. One of these would contain the development environment and sources, and the other would be used to test the newly written code by network-booting and network-mounting filesystems from the first one. This way if the new code contains bugs and crashes the machine, it will not mess up the sources (and other "live" data). The second system does not even require a proper display. Instead, it could be connected with a serial cable or KVM to the first one. But, since not everybody has two or more computers handy, there are a few things that can be done to prepare an otherwise "live" system for developing kernel code. This setup is also applicable for developing in a http://www.vmware.com/[VMWare] or http://www.qemu.org/[QEmu] virtual machine (the next best thing after a dedicated development machine). [[prelim-system]] === Modifying a System for Development For any kernel programming a kernel with `INVARIANTS` enabled is a must-have. So enter these in your kernel configuration file: [.programlisting] .... options INVARIANT_SUPPORT options INVARIANTS .... For more debugging you should also include WITNESS support, which will alert you of mistakes in locking: [.programlisting] .... options WITNESS_SUPPORT options WITNESS .... For debugging crash dumps, a kernel with debug symbols is needed: [.programlisting] .... makeoptions DEBUG=-g .... With the usual way of installing the kernel (`make installkernel`) the debug kernel will not be automatically installed. It is called [.filename]#kernel.debug# and located in [.filename]#/usr/obj/usr/src/sys/KERNELNAME/#. For convenience it should be copied to [.filename]#/boot/kernel/#. Another convenience is enabling the kernel debugger so you can examine a kernel panic when it happens. For this, enter the following lines in your kernel configuration file: [.programlisting] .... options KDB options DDB options KDB_TRACE .... For this to work you might need to set a sysctl (if it is not on by default): [.programlisting] .... debug.debugger_on_panic=1 .... Kernel panics will happen, so care should be taken with the filesystem cache. In particular, having softupdates might mean the latest file version could be lost if a panic occurs before it is committed to storage. Disabling softupdates yields a great performance hit, and still does not guarantee data consistency. Mounting filesystem with the "sync" option is needed for that. For a compromise, the softupdates cache delays can be shortened. There are three sysctl's that are useful for this (best to be set in [.filename]#/etc/sysctl.conf#): [.programlisting] .... kern.filedelay=5 kern.dirdelay=4 kern.metadelay=3 .... The numbers represent seconds. For debugging kernel panics, kernel core dumps are required. Since a kernel panic might make filesystems unusable, this crash dump is first written to a raw partition. Usually, this is the swap partition. This partition must be at least as large as the physical RAM in the machine. On the next boot, the dump is copied to a regular file. This happens after filesystems are checked and mounted, and before swap is enabled. This is controlled with two [.filename]#/etc/rc.conf# variables: [.programlisting] .... dumpdev="/dev/ad0s4b" dumpdir="/usr/core .... The `dumpdev` variable specifies the swap partition and `dumpdir` tells the system where in the filesystem to relocate the core dump on reboot. Writing kernel core dumps is slow and takes a long time so if you have lots of memory (>256M) and lots of panics it could be frustrating to sit and wait while it is done (twice - first to write it to swap, then to relocate it to filesystem). It is convenient then to limit the amount of RAM the system will use via a [.filename]#/boot/loader.conf# tunable: [.programlisting] .... hw.physmem="256M" .... If the panics are frequent and filesystems large (or you simply do not trust softupdates+background fsck) it is advisable to turn background fsck off via [.filename]#/etc/rc.conf# variable: [.programlisting] .... background_fsck="NO" .... This way, the filesystems will always get checked when needed. Note that with background fsck, a new panic could happen while it is checking the disks. Again, the safest way is not to have many local filesystems by using another computer as an NFS server. [[prelim-starting]] === Starting the Project For the purpose of creating a new GEOM class, an empty subdirectory has to be created under an arbitrary user-accessible directory. You do not have to create the module directory under [.filename]#/usr/src#. [[prelim-makefile]] === The Makefile It is good practice to create [.filename]#Makefiles# for every nontrivial coding project, which of course includes kernel modules. Creating the [.filename]#Makefile# is simple thanks to an extensive set of helper routines provided by the system. In short, here is how a minimal [.filename]#Makefile# looks for a kernel module: [.programlisting] .... SRCS=g_journal.c KMOD=geom_journal .include .... This [.filename]#Makefile# (with changed filenames) will do for any kernel module, and a GEOM class can reside in just one kernel module. If more than one file is required, list it in the `SRCS` variable, separated with whitespace from other filenames. [[kernelprog]] == On FreeBSD Kernel Programming [[kernelprog-memalloc]] === Memory Allocation See man:malloc[9]. Basic memory allocation is only slightly different than its userland equivalent. Most notably, `malloc`() and `free`() accept additional parameters as is described in the man page. A "malloc type" must be declared in the declaration section of a source file, like this: [.programlisting] .... static MALLOC_DEFINE(M_GJOURNAL, "gjournal data", "GEOM_JOURNAL Data"); .... To use this macro, [.filename]#sys/param.h#, [.filename]#sys/kernel.h# and [.filename]#sys/malloc.h# headers must be included. There is another mechanism for allocating memory, the UMA (Universal Memory Allocator). See man:uma[9] for details, but it is a special type of allocator mainly used for speedy allocation of lists comprised of same-sized items (for example, dynamic arrays of structs). [[kernelprog-lists]] === Lists and Queues See man:queue[3]. There are a LOT of cases when a list of things needs to be maintained. Fortunately, this data structure is implemented (in several ways) by C macros included in the system. The most used list type is TAILQ because it is the most flexible. It is also the one with largest memory requirements (its elements are doubly-linked) and also the slowest (although the speed variation is on the order of several CPU instructions more, so it should not be taken seriously). If data retrieval speed is very important, see man:tree[3] and man:hashinit[9]. [[kernelprog-bios]] === BIOs Structure `bio` is used for any and all Input/Output operations concerning GEOM. It basically contains information about what device ('provider') should satisfy the request, request type, offset, length, pointer to a buffer, and a bunch of "user-specific" flags and fields that can help implement various hacks. The important thing here is that ``bio``s are handled asynchronously. That means that, in most parts of the code, there is no analogue to userland's man:read[2] and man:write[2] calls that do not return until a request is done. Rather, a developer-supplied function is called as a notification when the request gets completed (or results in error). The asynchronous programming model (also called "event-driven") is somewhat harder than the much more used imperative one used in userland (at least it takes a while to get used to it). In some cases the helper routines `g_write_data`() and `g_read_data`() can be used, but __not always__. In particular, they cannot be used when a mutex is held; for example, the GEOM topology mutex or the internal mutex held during the `.start`() and `.stop`() functions. [[geom]] == On GEOM Programming [[geom-ggate]] === Ggate If maximum performance is not needed, a much simpler way of making a data transformation is to implement it in userland via the ggate (GEOM gate) facility. Unfortunately, there is no easy way to convert between, or even share code between the two approaches. [[geom-class]] === GEOM Class GEOM classes are transformations on the data. These transformations can be combined in a tree-like fashion. Instances of GEOM classes are called __geoms__. Each GEOM class has several "class methods" that get called when there is no geom instance available (or they are simply not bound to a single instance): * `.init` is called when GEOM becomes aware of a GEOM class (when the kernel module gets loaded.) * `.fini` gets called when GEOM abandons the class (when the module gets unloaded) * `.taste` is called next, once for each provider the system has available. If applicable, this function will usually create and start a geom instance. * `.destroy_geom` is called when the geom should be disbanded * `.ctlconf` is called when user requests reconfiguration of existing geom Also defined are the GEOM event functions, which will get copied to the geom instance. Field `.geom` in the `g_class` structure is a LIST of geoms instantiated from the class. These functions are called from the g_event kernel thread. [[geom-softc]] === Softc The name "softc" is a legacy term for "driver private data". The name most probably comes from the archaic term "software control block". In GEOM, it is a structure (more precise: pointer to a structure) that can be attached to a geom instance to hold whatever data is private to the geom instance. Most GEOM classes have the following members: * `struct g_provider *provider` : The "provider" this geom instantiates * `uint16_t n_disks` : Number of consumer this geom consumes * `struct g_consumer \**disks` : Array of `struct g_consumer*`. (It is not possible to use just single indirection because struct g_consumer* are created on our behalf by GEOM). The `softc` structure contains all the state of geom instance. Every geom instance has its own softc. [[geom-metadata]] === Metadata Format of metadata is more-or-less class-dependent, but MUST start with: * 16 byte buffer for null-terminated signature (usually the class name) * uint32 version ID It is assumed that geom classes know how to handle metadata with version ID's lower than theirs. Metadata is located in the last sector of the provider (and thus must fit in it). (All this is implementation-dependent but all existing code works like that, and it is supported by libraries.) [[geom-creating]] === Labeling/creating a GEOM The sequence of events is: * user calls man:geom[8] utility (or one of its hardlinked friends) * the utility figures out which geom class it is supposed to handle and searches for [.filename]#geom_CLASSNAME.so# library (usually in [.filename]#/lib/geom#). * it man:dlopen[3]-s the library, extracts the definitions of command-line parameters and helper functions. In the case of creating/labeling a new geom, this is what happens: * man:geom[8] looks in the command-line argument for the command (usually `label`), and calls a helper function. * The helper function checks parameters and gathers metadata, which it proceeds to write to all concerned providers. * This "spoils" existing geoms (if any) and initializes a new round of "tasting" of the providers. The intended geom class recognizes the metadata and brings the geom up. (The above sequence of events is implementation-dependent but all existing code works like that, and it is supported by libraries.) [[geom-command]] === GEOM Command Structure The helper [.filename]#geom_CLASSNAME.so# library exports `class_commands` structure, which is an array of `struct g_command` elements. Commands are of uniform format and look like: [.programlisting] .... verb [-options] geomname [other] .... Common verbs are: * label - to write metadata to devices so they can be recognized at tasting and brought up in geoms * destroy - to destroy metadata, so the geoms get destroyed Common options are: * `-v` : be verbose * `-f` : force Many actions, such as labeling and destroying metadata can be performed in userland. For this, `struct g_command` provides field `gc_func` that can be set to a function (in the same [.filename]#.so#) that will be called to process a verb. If `gc_func` is NULL, the command will be passed to kernel module, to `.ctlreq` function of the geom class. [[geom-geoms]] === Geoms Geoms are instances of GEOM classes. They have internal data (a softc structure) and some functions with which they respond to external events. The event functions are: * `.access` : calculates permissions (read/write/exclusive) * `.dumpconf` : returns XML-formatted information about the geom * `.orphan` : called when some underlying provider gets disconnected * `.spoiled` : called when some underlying provider gets written to * `.start` : handles I/O These functions are called from the `g_down` kernel thread and there can be no sleeping in this context, (see definition of sleeping elsewhere) which limits what can be done quite a bit, but forces the handling to be fast. Of these, the most important function for doing actual useful work is the `.start`() function, which is called when a BIO request arrives for a provider managed by a instance of geom class. [[geom-threads]] === GEOM Threads There are three kernel threads created and run by the GEOM framework: * `g_down` : Handles requests coming from high-level entities (such as a userland request) on the way to physical devices * `g_up` : Handles responses from device drivers to requests made by higher-level entities * `g_event` : Handles all other cases: creation of geom instances, access counting, "spoil" events, etc. When a user process issues "read data X at offset Y of a file" request, this is what happens: * The filesystem converts the request into a struct bio instance and passes it to the GEOM subsystem. It knows what geom instance should handle it because filesystems are hosted directly on a geom instance. * The request ends up as a call to the `.start`() function made on the g_down thread and reaches the top-level geom instance. * This top-level geom instance (for example the partition slicer) determines that the request should be routed to a lower-level instance (for example the disk driver). It makes a copy of the bio request (bio requests _ALWAYS_ need to be copied between instances, with `g_clone_bio`()!), modifies the data offset and target provider fields and executes the copy with `g_io_request`() * The disk driver gets the bio request also as a call to `.start`() on the `g_down` thread. It talks to hardware, gets the data back, and calls `g_io_deliver`() on the bio. * Now, the notification of bio completion "bubbles up" in the `g_up` thread. First the partition slicer gets `.done`() called in the `g_up` thread, it uses information stored in the bio to free the cloned `bio` structure (with `g_destroy_bio`()) and calls `g_io_deliver`() on the original request. * The filesystem gets the data and transfers it to userland. See man:g_bio[9] man page for information how the data is passed back and forth in the `bio` structure (note in particular the `bio_parent` and `bio_children` fields and how they are handled). One important feature is: __THERE CAN BE NO SLEEPING IN G_UP AND G_DOWN THREADS__. This means that none of the following things can be done in those threads (the list is of course not complete, but only informative): * Calls to `msleep`() and `tsleep`(), obviously. * Calls to `g_write_data`() and `g_read_data`(), because these sleep between passing the data to consumers and returning. * Waiting for I/O. * Calls to man:malloc[9] and `uma_zalloc`() with `M_WAITOK` flag set * sx and other sleepable locks This restriction is here to stop GEOM code clogging the I/O request path, since sleeping is usually not time-bound and there can be no guarantees on how long will it take (there are some other, more technical reasons also). It also means that there is not much that can be done in those threads; for example, almost any complex thing requires memory allocation. Fortunately, there is a way out: creating additional kernel threads. [[geom-kernelthreads]] === Kernel Threads for Use in GEOM Code Kernel threads are created with man:kthread_create[9] function, and they are sort of similar to userland threads in behavior, only they cannot return to caller to signify termination, but must call man:kthread_exit[9]. In GEOM code, the usual use of threads is to offload processing of requests from `g_down` thread (the `.start`() function). These threads look like "event handlers": they have a linked list of event associated with them (on which events can be posted by various functions in various threads so it must be protected by a mutex), take the events from the list one by one and process them in a big `switch`() statement. The main benefit of using a thread to handle I/O requests is that it can sleep when needed. Now, this sounds good, but should be carefully thought out. Sleeping is well and very convenient but can very effectively destroy performance of the geom transformation. Extremely performance-sensitive classes probably should do all the work in `.start`() function call, taking great care to handle out-of-memory and similar errors. The other benefit of having a event-handler thread like that is to serialize all the requests and responses coming from different geom threads into one thread. This is also very convenient but can be slow. In most cases, handling of `.done`() requests can be left to the `g_up` thread. Mutexes in FreeBSD kernel (see man:mutex[9]) have one distinction from their more common userland cousins - the code cannot sleep while holding a mutex). If the code needs to sleep a lot, man:sx[9] locks may be more appropriate. On the other hand, if you do almost everything in a single thread, you may get away with no mutexes at all. diff --git a/documentation/content/en/articles/gjournal-desktop/_index.adoc b/documentation/content/en/articles/gjournal-desktop/_index.adoc index be46c41b91..c39fa2d3a1 100644 --- a/documentation/content/en/articles/gjournal-desktop/_index.adoc +++ b/documentation/content/en/articles/gjournal-desktop/_index.adoc @@ -1,505 +1,519 @@ --- title: Implementing UFS Journaling on a Desktop PC authors: - author: Manolis Kiagias email: manolis@FreeBSD.org description: Implementing UFS Journaling on a Desktop PC trademarks: ["freebsd", "general"] tags: ["UFS", "Journaling" , "Desktop", "FreeBSD"] --- = Implementing UFS Journaling on a Desktop PC :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/gjournal-desktop/ endif::[] ifeval::["{backend}" == "pdf"] :imagesdir: ../../../../static/images/articles/gjournal-desktop/ endif::[] ifeval::["{backend}" == "epub3"] :imagesdir: ../../../../static/images/articles/gjournal-desktop/ endif::[] [.abstract-title] Abstract A journaling file system uses a log to record all transactions that take place in the file system, and preserves its integrity in the event of a system crash or power failure. Although it is still possible to lose unsaved changes to files, journaling almost completely eliminates the possibility of file system corruption caused by an unclean shutdown. It also shortens to a minimum the time required for after-failure file system checking. Although the UFS file system employed by FreeBSD does not implement journaling itself, the new journal class of the GEOM framework in FreeBSD 7._X_ can be used to provide file system independent journaling. This article explains how to implement UFS journaling on a typical desktop PC scenario. ''' toc::[] [[introduction]] == Introduction While professional servers are usually well protected from unforeseen shutdowns, the typical desktop is at the mercy of power failures, accidental resets, and other user related incidents that can lead to unclean shutdowns. Soft Updates usually protect the file system efficiently in such cases, although most of the times a lengthy background check is required. On rare occasions, file system corruption reaches a point where user intervention is required and data may be lost. The new journaling capability provided by GEOM can greatly assist in such scenarios, by virtually eliminating the time required for file system checking, and ensuring that the file system is quickly restored to a consistent state. This article describes a procedure for implementing UFS journaling on a typical desktop PC scenario (one hard disk used for both operating system and data). It should be followed during a fresh installation of FreeBSD. The steps are simple enough and do not require overly complex interaction with the command line. After reading this article, you will know: * How to reserve space for journaling during a new installation of FreeBSD. * How to load and enable the `geom_journal` module (or build support for it in your custom kernel). * How to convert your existing file systems to utilize journaling, and what options to use in [.filename]#/etc/fstab# to mount them. * How to implement journaling in new (empty) partitions. * How to troubleshoot common problems associated with journaling. Before reading this article, you should be able to: * Understand basic UNIX(R) and FreeBSD concepts. * Be familiar with the installation procedure of FreeBSD and the sysinstall utility. [WARNING] ==== The procedure described here is intended for preparing a new installation where no actual user data is stored on the disk yet. While it is possible to modify and extend this procedure for systems already in production, you should _backup_ all important data before doing so. Messing around with disks and partitions at a low level can lead to fatal mistakes and data loss. ==== [[understanding-journaling]] == Understanding Journaling in FreeBSD The journaling provided by GEOM in FreeBSD 7._X_ is not file system specific (unlike for example the ext3 file system in Linux(R)) but is functioning at the block level. Though this means it can be applied to different file systems, for FreeBSD 7.0-RELEASE, it can only be used on UFS2. This functionality is provided by loading the [.filename]#geom_journal.ko# module into the kernel (or building it into a custom kernel) and using the `gjournal` command to configure the file systems. In general, you would like to journal large file systems, like [.filename]#/usr#. You will need however (see the following section) to reserve some free disk space. When a file system is journaled, some disk space is needed to keep the journal itself. The disk space that holds the actual data is referred to as the __data provider__, while the one that holds the journal is referred to as the __journal provider__. The data and journal providers need to be on different partitions when journaling an existing (non-empty) partition. When journaling a new partition, you have the option to use a single provider for both data and journal. In any case, the `gjournal` command combines both providers to create the final journaled file system. For example: * You wish to journal your [.filename]#/usr# file system, stored in [.filename]#/dev/ad0s1f# (which already contains data). * You reserved some free disk space in a partition in [.filename]#/dev/ad0s1g#. * Using `gjournal`, a new [.filename]#/dev/ad0s1f.journal# device is created where [.filename]#/dev/ad0s1f# is the data provider, and [.filename]#/dev/ad0s1g# is the journal provider. This new device is then used for all subsequent file operations. The amount of disk space you need to reserve for the journal provider depends on the usage load of the file system and not on the size of the data provider. For example on a typical office desktop, a 1 GB journal provider for the [.filename]#/usr# file system will suffice, while a machine that deals with heavy disk I/O (i.e. video editing) may need more. A kernel panic will occur if the journal space is exhausted before it has a chance to be committed. [NOTE] ==== The journal sizes suggested here, are highly unlikely to cause problems in typical desktop use (such as web browsing, word processing and playback of media files). If your workload includes intense disk activity, use the following rule for maximum reliability: Your RAM size should fit in 30% of the journal provider's space. For example, if your system has 1 GB RAM, create an approximately 3.3 GB journal provider. (Multiply your RAM size with 3.3 to obtain the size of the journal). ==== For more information about journaling, please read the manual page of man:gjournal[8]. [[reserve-space]] == Steps During the Installation of FreeBSD === Reserving Space for Journaling A typical desktop machine usually has one hard disk that stores both the OS and user data. Arguably, the default partitioning scheme selected by sysinstall is more or less suitable: A desktop machine does not need a large [.filename]#/var# partition, while [.filename]#/usr# is allocated the bulk of the disk space, since user data and a lot of packages are installed into its subdirectories. The default partitioning (the one obtained by pressing kbd:[A] at the FreeBSD partition editor, called Disklabel) does not leave any unallocated space. Each partition that will be journaled, requires another partition for the journal. Since the [.filename]#/usr# partition is the largest, it makes sense to shrink this partition slightly, to obtain the space required for journaling. In our example, an 80 GB disk is used. The following screenshot shows the default partitions created by Disklabel during installation: image::disklabel1.png[] If this is more or less what you need, it is very easy to adjust for journaling. Simply use the arrow keys to move the highlight to the [.filename]#/usr# partition and press kbd:[D] to delete it. Now, move the highlight to the disk name at the top of the screen and press kbd:[C] to create a new partition for [.filename]#/usr#. This new partition should be smaller by 1 GB (if you intend to journal [.filename]#/usr# only), or 2 GB (if you intend to journal both [.filename]#/usr# and [.filename]#/var#). From the pop-up that appears, opt to create a file system, and type [.filename]#/usr# as the mount point. [NOTE] ==== Should you journal the [.filename]#/var# partition? Normally, journaling makes sense on quite large partitions. You may decide not to journal [.filename]#/var#, although doing so on a typical desktop will cause no harm. If the file system is lightly used (quite probable for a desktop) you may wish to allocate less disk space for its journal. In our example, we journal both [.filename]#/usr# and [.filename]#/var#. You may of course adjust the procedure to your own needs. ==== To keep things as easy going as possible, we are going to use sysinstall to create the partitions required for journaling. However, during installation, sysinstall insists on asking a mount point for each partition you create. At this point, you do not have any mount points for the partitions that will hold the journals, and in reality you __do not even need them__. These are not partitions that we are ever going to mount somewhere. To avoid these problems with sysinstall, we are going to create the journal partitions as swap space. Swap is never mounted, and sysinstall has no problem creating as many swap partitions as needed. After the first reboot, [.filename]#/etc/fstab# will have to be edited, and the extra swap space entries removed. To create the swap, again use the arrow keys to move the highlight to the top of Disklabel screen, so that the disk name itself is highlighted. Then press kbd:[N], enter the desired size (_1024M_), and select "swap space" from the pop-up menu that appears. Repeat for every journal you wish to create. In our example, we create two partitions to provide for the journals of [.filename]#/usr# and [.filename]#/var#. The final result is shown in the following screenshot: image::disklabel2.png[] When you have completed creating the partitions, we suggest you write down the partition names, and mount points, so you can easily refer to this information during the configuration phase. This will help alleviate mistakes that may damage your installation. The following table shows our notes for the sample configuration: .Partitions and Journals [cols="1,1,1", options="header"] |=== | Partition | Mount Point | Journal |ad0s1d |/var |ad0s1h |ad0s1f |/usr |ad0s1g |=== Continue the installation as you would normally do. We would however suggest you postpone installation of third party software (packages) until you have completely setup journaling. [[first-boot]] === Booting for the first time Your system will come up normally, but you will need to edit [.filename]#/etc/fstab# and remove the extra swap partitions you created for the journals. Normally, the swap partition you will actually use is the one with the "b" suffix (i.e. ad0s1b in our example). Remove all other swap space entries and reboot so that FreeBSD will stop using them. When the system comes up again, we will be ready to configure journaling. [[configure-journal]] == Setting Up Journaling [[running-gjournal]] === Executing `gjournal` Having prepared all the required partitions, it is quite easy to configure journaling. We will need to switch to single user mode, so login as `root` and type: [source,shell] .... # shutdown now .... Press kbd:[Enter] to get the default shell. We will need to unmount the partitions that will be journaled, in our example [.filename]#/usr# and [.filename]#/var#: [source,shell] .... # umount /usr /var .... Load the module required for journaling: [source,shell] .... # gjournal load .... Now, use your notes to determine which partition will be used for each journal. In our example, [.filename]#/usr# is [.filename]#ad0s1f# and its journal will be [.filename]#ad0s1g#, while [.filename]#/var# is [.filename]#ad0s1d# and will be journaled to [.filename]#ad0s1h#. The following commands are required: [source,shell] .... # gjournal label ad0s1f ad0s1g GEOM_JOURNAL: Journal 2948326772: ad0s1f contains data. GEOM_JOURNAL: Journal 2948326772: ad0s1g contains journal. # gjournal label ad0s1d ad0s1h GEOM_JOURNAL: Journal 3193218002: ad0s1d contains data. GEOM_JOURNAL: Journal 3193218002: ad0s1h contains journal. .... [NOTE] ==== If the last sector of either partition is used, `gjournal` will return an error. You will have to run the command using the `-f` flag to force an overwrite, i.e.: [source,shell] .... # gjournal label -f ad0s1d ad0s1h .... Since this is a new installation, it is highly unlikely that anything will be actually overwritten. ==== At this point, two new devices are created, namely [.filename]#ad0s1d.journal# and [.filename]#ad0s1f.journal#. These represent the [.filename]#/var# and [.filename]#/usr# partitions we have to mount. Before mounting, we must however set the journal flag on them and clear the Soft Updates flag: [source,shell] .... # tunefs -J enable -n disable ad0s1d.journal tunefs: gjournal set tunefs: soft updates cleared # tunefs -J enable -n disable ad0s1f.journal tunefs: gjournal set tunefs: soft updates cleared .... Now, mount the new devices manually at their respective places (note that we can now use the `async` mount option): [source,shell] .... # mount -o async /dev/ad0s1d.journal /var # mount -o async /dev/ad0s1f.journal /usr .... Edit [.filename]#/etc/fstab# and update the entries for [.filename]#/usr# and [.filename]#/var#: [.programlisting] .... /dev/ad0s1f.journal /usr ufs rw,async 2 2 /dev/ad0s1d.journal /var ufs rw,async 2 2 .... [WARNING] ==== Make sure the above entries are correct, or you will have trouble starting up normally after you reboot! ==== Finally, edit [.filename]#/boot/loader.conf# and add the following line so the man:gjournal[8] module is loaded at every boot: [.programlisting] .... geom_journal_load="YES" .... Congratulations! Your system is now set for journaling. You can either type `exit` to return to multi-user mode, or reboot to test your configuration (recommended). During the boot you will see messages like the following: [source,shell] .... ad0: 76293MB XEC XE800JD-00HBC0 08.02D08 at ata0-master SATA150 GEOM_JOURNAL: Journal 2948326772: ad0s1g contains journal. GEOM_JOURNAL: Journal 3193218002: ad0s1h contains journal. GEOM_JOURNAL: Journal 3193218002: ad0s1d contains data. GEOM_JOURNAL: Journal ad0s1d clean. GEOM_JOURNAL: Journal 2948326772: ad0s1f contains data. GEOM_JOURNAL: Journal ad0s1f clean. .... After an unclean shutdown, the messages will vary slightly, i.e.: [source,shell] .... GEOM_JOURNAL: Journal ad0s1d consistent. .... This usually means that man:gjournal[8] used the information in the journal provider to return the file system to a consistent state. [[gjournal-new]] === Journaling Newly Created Partitions While the above procedure is necessary for journaling partitions that already contain data, journaling an empty partition is somewhat easier, since both the data and the journal provider can be stored in the same partition. For example, assume a new disk was installed, and a new partition [.filename]#/dev/ad1s1d# was created. Creating the journal would be as simple as: [source,shell] .... # gjournal label ad1s1d .... The journal size will be 1 GB by default. You may adjust it by using the `-s` option. The value can be given in bytes, or appended by `K`, `M` or `G` to denote Kilobytes, Megabytes or Gigabytes respectively. Note that `gjournal` will not allow you to create unsuitably small journal sizes. For example, to create a 2 GB journal, you could use the following command: [source,shell] .... # gjournal label -s 2G ad1s1d .... You can then create a file system on your new partition, and enable journaling using the `-J` option: [source,shell] .... # newfs -J /dev/ad1s1d.journal .... [[configure-kernel]] === Building Journaling into Your Custom Kernel If you do not wish to load `geom_journal` as a module, you can build its functions right into your kernel. Edit your custom kernel configuration file, and make sure it includes these two lines: [.programlisting] .... options UFS_GJOURNAL # Note: This is already in GENERIC options GEOM_JOURNAL # You will have to add this one .... Rebuild and reinstall your kernel following the relevant link:{handbook}#kernelconfig[instructions in the FreeBSD Handbook.] Do not forget to remove the relevant "load" entry from [.filename]#/boot/loader.conf# if you have previously used it. [[troubleshooting-gjournal]] == Troubleshooting Journaling The following section covers frequently asked questions regarding problems related to journaling. === I am getting kernel panics during periods of high disk activity. How is this related to journaling? The journal probably fills up before it has a chance to get committed (flushed) to disk. Keep in mind the size of the journal depends on the usage load, and not the size of the data provider. If your disk activity is high, you need a larger partition for the journal. See the note in the <> section. === I made some mistake during configuration, and I cannot boot normally now. Can this be fixed some way? You either forgot (or misspelled) the entry in [.filename]#/boot/loader.conf#, or there are errors in your [.filename]#/etc/fstab# file. These are usually easy to fix. Press kbd:[Enter] to get to the default single user shell. Then locate the root of the problem: [source,shell] .... # cat /boot/loader.conf .... If the `geom_journal_load` entry is missing or misspelled, the journaled devices are never created. Load the module manually, mount all partitions, and continue with multi-user boot: [source,shell] .... # gjournal load GEOM_JOURNAL: Journal 2948326772: ad0s1g contains journal. GEOM_JOURNAL: Journal 3193218002: ad0s1h contains journal. GEOM_JOURNAL: Journal 3193218002: ad0s1d contains data. GEOM_JOURNAL: Journal ad0s1d clean. GEOM_JOURNAL: Journal 2948326772: ad0s1f contains data. GEOM_JOURNAL: Journal ad0s1f clean. # mount -a # exit (boot continues) .... If, on the other hand, this entry is correct, have a look at [.filename]#/etc/fstab#. You will probably find a misspelled or missing entry. In this case, mount all remaining partitions by hand and continue with the multi-user boot. === Can I remove journaling and return to my standard file system with Soft Updates? Sure. Use the following procedure, which reverses the changes. The partitions you created for the journal providers can then be used for other purposes, if you so wish. Login as `root` and switch to single user mode: [source,shell] .... # shutdown now .... Unmount the journaled partitions: [source,shell] .... # umount /usr /var .... Synchronize the journals: [source,shell] .... # gjournal sync .... Stop the journaling providers: [source,shell] .... # gjournal stop ad0s1d.journal # gjournal stop ad0s1f.journal .... Clear journaling metadata from all the devices used: [source,shell] .... # gjournal clear ad0s1d # gjournal clear ad0s1f # gjournal clear ad0s1g # gjournal clear ad0s1h .... Clear the file system journaling flag, and restore the Soft Updates flag: [source,shell] .... # tunefs -J disable -n enable ad0s1d tunefs: gjournal cleared tunefs: soft updates set # tunefs -J disable -n enable ad0s1f tunefs: gjournal cleared tunefs: soft updates set .... Remount the old devices by hand: [source,shell] .... # mount -o rw /dev/ad0s1d /var # mount -o rw /dev/ad0s1f /usr .... Edit [.filename]#/etc/fstab# and restore it to its original state: [.programlisting] .... /dev/ad0s1f /usr ufs rw 2 2 /dev/ad0s1d /var ufs rw 2 2 .... Finally, edit [.filename]#/boot/loader.conf#, remove the entry that loads the `geom_journal` module and reboot. [[further-reading]] == Further Reading Journaling is a fairly new feature of FreeBSD, and as such, it is not very well documented yet. You may however find the following additional references useful: * A link:{handbook}#geom-gjournal[new section on journaling] is now part of the FreeBSD Handbook. * https://lists.freebsd.org/pipermail/freebsd-current/2006-June/064043.html[This post] in {freebsd-current} by man:gjournal[8]'s developer, `{pjd}`. * https://lists.freebsd.org/pipermail/freebsd-questions/2008-April/173501.html[This post] in {freebsd-questions} by `{ivoras}`. * The manual pages of man:gjournal[8] and man:geom[8]. diff --git a/documentation/content/en/articles/hubs/_index.adoc b/documentation/content/en/articles/hubs/_index.adoc index e79fdf9ba2..fa28584e38 100644 --- a/documentation/content/en/articles/hubs/_index.adoc +++ b/documentation/content/en/articles/hubs/_index.adoc @@ -1,380 +1,394 @@ --- title: Mirroring FreeBSD authors: - author: Jun Kuriyama email: kuriyama@FreeBSD.org - author: Valentino Vaschetto email: logo@FreeBSD.org - author: Daniel Lang email: dl@leo.org - author: Ken Smith email: kensmith@FreeBSD.org description: The all in one guide for mirroring the FreeBSD website, FTP servers, and more trademarks: ["freebsd", "general"] tags: ["Mirroring", "FreeBSD", "Hub"] --- = Mirroring FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] include::shared/releases.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +include::../../../../shared/releases.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +include::../../../../shared/releases.adoc[] +endif::[] [.abstract-title] Abstract An in-progress article on how to mirror FreeBSD, aimed at hub administrators. ''' toc::[] [NOTE] ==== We are not accepting new mirrors at this time. ==== [[mirror-contact]] == Contact Information The Mirror System Coordinators can be reached through email at mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org]. There is also a {freebsd-hubs}. [[mirror-requirements]] == Requirements for FreeBSD Mirrors [[mirror-diskspace]] === Disk Space Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that _official_ mirrors are probably required to be complete. The web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at {rel120-current}-RELEASE/{rel113-current}-RELEASE). Further development and releases will only increase the required amount. Also make sure to keep some (ca. 10-20%) extra space around just to be sure. Here are some approximate figures: * Full FTP Distribution: 1.4 TB * CTM deltas: 10 GB * Web pages: 1GB The current disk usage of FTP Distribution can be found at link:ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes[ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes]. [[mirror-bandwidth]] === Network Connection/Bandwidth Of course, you need to be connected to the Internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available. If you intend to become an official mirror, the bandwidth required will be even higher. We can only give rough estimates here: * Local site, no public access: basically no minimum, but < 2 Mbps could make syncing too slow. * Unofficial public site: 34 Mbps is probably a good start. * Official site: > 100 Mbps is recommended, and your host should be connected as close as possible to your border router. [[mirror-system]] === System Requirements, CPU, RAM One thing this depends on the expected number of clients, which is determined by the server's policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out if you provide rsync. This can have a huge impact on CPU and memory requirements as it is considered a memory hog. The following are just examples to give you a very rough hint. For a moderately visited site that offers rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an _official_ site. For a frequently used site you definitely need more RAM (consider 2GB as a good start) and possibly more CPU, which could also mean that you need to go for a SMP system. You also want to consider a fast disk subsystem. Operations on the SVN repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk. [[mirror-services]] === Services to Offer Every mirror site is required to have a set of core services available. In addition to these required services, there are a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them. [[mirror-serv-ftp]] ==== FTP (required for FTP Fileset) This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and _must_ never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path [.filename]#/pub/FreeBSD#. There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order). * `/usr/libexec/ftpd`: FreeBSD's own ftpd can be used. Be sure to read man:ftpd[8]. * package:ftp/ncftpd[]: A commercial package, free for educational use. * package:ftp/oftpd[]: An ftpd designed with security as a main focus. * package:ftp/proftpd[]: A modular and very flexible ftpd. * package:ftp/pure-ftpd[]: Another ftpd developed with security in mind. * package:ftp/twoftpd[]: As above. * package:ftp/vsftpd[]: The "very secure" ftpd. FreeBSD's `ftpd`, `proftpd` and maybe `ncftpd` are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed. [[mirror-serv-rsync]] ==== Rsync (optional for FTP Fileset) Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. Rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. Rsync can use `rsh` and `ssh` (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available: * package:net/rsync[] [[mirror-serv-http]] ==== HTTP (required for Web Pages, Optional for FTP Fileset) If you want to offer the FreeBSD web pages, you will need to install a web server. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are: * package:www/apache24[]: Apache is still one of the most widely deployed web servers on the Internet. It is used extensively by the FreeBSD Project. * package:www/boa[]: Boa is a single-tasking HTTP server. Unlike traditional web servers, it does not fork for each incoming connection, nor does it fork many copies of itself to handle multiple connections. Although, it should provide considerably great performance for purely static content. * package:www/cherokee[]: Cherokee is a very fast, flexible and easy to configure web server. It supports the widespread technologies nowadays: FastCGI, SCGI, PHP, CGI, SSL/TLS encrypted connections, vhosts, users authentication, on the fly encoding and load balancing. It also generates Apache compatible log files. * package:www/lighttpd[]: lighttpd is a secure, fast, compliant and very flexible web server which has been optimized for high-performance environments. It has a very low memory footprint compared to other web servers and takes care of cpu-load. * package:www/nginx[]: nginx is a high performance edge web server with a low memory footprint and key features to build a modern and efficient web infrastructure. Features include a HTTP server, HTTP and mail reverse proxy, caching, load balancing, compression, request throttling, connection multiplexing and reuse, SSL offload and HTTP media streaming. * package:www/thttpd[]: If you are going to be serving a large amount of static content you may find that using an application such as thttpd is more efficient than others. It is also optimized for excellent performance on FreeBSD. [[mirror-howto]] == How to Mirror FreeBSD Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from. [[mirror-ftp-rsync]] === Mirroring the FTP Site The FTP area is the largest amount of data that needs to be mirrored. It includes the _distribution sets_ required for network installation, the _branches_ which are actually snapshots of checked-out source trees, the _ISO Images_ to write CD-ROMs with the installation distribution, a live file system, and a snapshot of the ports tree. All of course for various FreeBSD versions, and various architectures. The best way to mirror the FTP area is rsync. You can install the port package:net/rsync[] and then use rsync to sync with your upstream host. rsync is already mentioned in <>. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access. [NOTE] ==== Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror). ==== A command line to mirror FreeBSD might look like: [source,shell] .... % rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/ .... Consult the documentation for rsync, which is also available at http://rsync.samba.org/[http://rsync.samba.org/], about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via man:cron[8]. [[mirror-www]] === Mirroring the WWW Pages The FreeBSD website should only be mirrored via rsync. A command line to mirror the FreeBSD web site might look like: [source,shell] .... % rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/ .... [[mirror-pkgs]] === Mirroring Packages Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the man:pkg[8] process. Alternatively specific packages and their dependencies can be fetched by running something like the following: [source,shell] .... % pkg fetch -d -o /usr/local/mirror vim .... Once those packages have been fetched, the repository metadata must be generated by running: [source,shell] .... % pkg repo /usr/local/mirror .... Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP. For additional information see the man pages for man:pkg[8], specifically the man:pkg-repo[8] page. [[mirror-how-often]] === How Often Should I Mirror? Every mirror should be updated at a minimum of once per day. Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from man:cron[8]. Since nearly every admin does this in their own way, specific instructions cannot be provided. It could work something like this: [.procedure] ==== . Put the command to run your mirroring application in a script. Use of a plain `/bin/sh` script is recommended. . Add some output redirections so diagnostic messages are logged to a file. . Test if your script works. Check the logs. . Use man:crontab[1] to add the script to the appropriate user's man:crontab[5]. This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files cannot be accessed by anonymous FTP. This is used to "stage" releases - making sure all of the official mirror sites have all of the necessary release files on release day. ==== Here are some recommended schedules: * FTP fileset: daily * WWW pages: daily [[mirror-where]] == Where to Mirror From This is an important issue. So this section will spend some effort to explain the backgrounds. We will say this several times: under no circumstances should you mirror from `ftp.FreeBSD.org`. [[mirror-where-organization]] === A few Words About the Organization Mirrors are organized by country. All official mirrors have a DNS entry of the form `ftpN.CC.FreeBSD.org`. _CC_ (i.e., country code) is the _top level domain_ (TLD) of the country where this mirror is located. _N_ is a number, telling that the host would be the _Nth_ mirror in that country. (Same applies to `wwwN.CC.FreeBSD.org`, etc.) There are mirrors with no _CC_ part. These are the mirror sites that are very well connected and allow a large number of concurrent users. `ftp.FreeBSD.org` is actually two machines, one currently located in Denmark and the other in the United States. It is _NOT_ a master site and should never be used to mirror from. Lots of online documentation leads "interactive"users to `ftp.FreeBSD.org` so automated mirroring systems should find a different machine to mirror from. Additionally there exists a hierarchy of mirrors, which is described in terms of __tiers__. The master sites are not referred to but can be described as __Tier-0__. Mirrors that mirror from these sites can be considered __Tier-1__, mirrors of __Tier-1__-mirrors, are __Tier-2__, etc. Official sites are encouraged to be of a low __tier__, but the lower the tier the higher the requirements in terms as described in <>. Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted. The __tier__-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites. However, official mirrors with low numbers like 1-4, are usually _Tier-1_ (this is just a rough hint, and there is no rule). [[mirror-where-where]] === Ok, but Where Should I get the Stuff Now? Under no circumstances should you mirror from `ftp.FreeBSD.org`. The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access. [[mirror-where-simple]] ==== I Just Want to Mirror from Somewhere! If you have no special intentions or requirements, the statement in <> applies. This means: [.procedure] ==== . Check for those which provide fastest access (number of hops, round-trip-times) and offer the services you intend to use (like rsync). . Contact the administrators of your chosen site stating your request, and asking about their terms and policies. . Set up your mirror as described above. ==== [[mirror-where-official]] ==== I am an Official Mirror, What is the Right Rite for Me? In general the description in <> still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about _official_ mirrors that are described in <>. [[mirror-where-master]] ==== I Want to Access the Master Sites! If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an _official_ mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the "zone administrator" (mailto:hostmaster@CC.FreeBSD.org[hostmaster@CC.FreeBSD.org]) or {freebsd-hubs} first. Whoever helped you become, an _official_ should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org] to request help with that. There is one master site for the FTP fileset. [[mirror-where-master-ftp]] ===== ftp-master.FreeBSD.org This is the master site for the FTP fileset. `ftp-master.FreeBSD.org` provides rsync access, in addition to FTP. Refer to <>. Mirrors are also encouraged to allow rsync access for the FTP contents, since they are __Tier-1__-mirrors. [[mirror-official]] == Official Mirrors Official mirrors are mirrors that * a) have a `FreeBSD.org` DNS entry (usually a CNAME). * b) are listed as an official mirror in the FreeBSD documentation (like handbook). So far to distinguish official mirrors. Official mirrors are not necessarily __Tier-1__-mirrors. However you probably will not find a __Tier-1__-mirror, that is not also official. [[mirror-official-requirements]] === Special Requirements for Official (tier-1) Mirrors It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here. It is more easy to say, what _official tier-1 mirrors_ are required to. All other official mirrors can consider this a big __should__. Tier-1 mirrors are required to: * carry the complete fileset * allow access to other mirror sites * provide FTP and rsync access Furthermore, admins should be subscribed to the {freebsd-hubs}. See link:{handbook}#eresources-mail[this link] for details, how to subscribe. [IMPORTANT] ==== It is _very_ important for a hub administrator, especially Tier-1 hub admins, to check the https://www.FreeBSD.org/releng/[release schedule] for the next FreeBSD release. This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it. It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors). If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral... Keep your mirrors up to date! ==== [[mirror-official-become]] === How to Become Official Then? We are not accepting any new mirrors at this time. [[mirror-statpages]] == Some Statistics from Mirror Sites Here are links to the stat pages of your favorite mirrors (aka the only ones who feel like providing stats). [[mirror-statpagesftp]] === FTP Site Statistics * ftp.is.FreeBSD.org - mailto:hostmaster@is.FreeBSD.org[hostmaster@is.FreeBSD.org] - http://www.rhnet.is/status/draupnir/draupnir.html[ (Bandwidth)] http://www.rhnet.is/status/ftp/ftp-notendur.html[(FTP processes)] http://www.rhnet.is/status/ftp/http-notendur.html[(HTTP processes)] * ftp2.ru.FreeBSD.org - mailto:mirror@macomnet.ru[mirror@macomnet.ru] - http://mirror.macomnet.net/mrtg/mirror.macomnet.net_195.128.64.25.html[(Bandwidth)] http://mirror.macomnet.net/mrtg/mirror.macomnet.net_proc.html[(HTTP and FTP users)] diff --git a/documentation/content/en/articles/ipsec-must/_index.adoc b/documentation/content/en/articles/ipsec-must/_index.adoc index 816eccce4f..7e2cbef7db 100644 --- a/documentation/content/en/articles/ipsec-must/_index.adoc +++ b/documentation/content/en/articles/ipsec-must/_index.adoc @@ -1,275 +1,285 @@ --- title: Independent Verification of IPsec Functionality in FreeBSD authors: - author: David Honig email: honig@sprynet.com description: Independent Verification of IPsec Functionality in FreeBSD trademarks: ["freebsd", "opengroup", "general"] tags: ["IPsec", "verification", "FreeBSD"] --- = Independent Verification of IPsec Functionality in FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract You installed IPsec and it seems to be working. How do you know? I describe a method for experimentally verifying that IPsec is working. ''' toc::[] [[problem]] == The Problem First, lets assume you have <>. How do you know it is <>? Sure, your connection will not work if it is misconfigured, and it will work when you finally get it right. man:netstat[1] will list it. But can you independently confirm it? [[solution]] == The Solution First, some crypto-relevant info theory: . Encrypted data is uniformly distributed, i.e., has maximal entropy per symbol; . Raw, uncompressed data is typically redundant, i.e., has sub-maximal entropy. Suppose you could measure the entropy of the data to- and from- your network interface. Then you could see the difference between unencrypted data and encrypted data. This would be true even if some of the data in "encrypted mode" was not encrypted---as the outermost IP header must be if the packet is to be routable. [[MUST]] === MUST Ueli Maurer's "Universal Statistical Test for Random Bit Generators"(https://web.archive.org/web/20011115002319/http://www.geocities.com/SiliconValley/Code/4704/universal.pdf[MUST]) quickly measures the entropy of a sample. It uses a compression-like algorithm. <> for a variant which measures successive (~quarter megabyte) chunks of a file. [[tcpdump]] === Tcpdump We also need a way to capture the raw network data. A program called man:tcpdump[1] lets you do this, if you have enabled the _Berkeley Packet Filter_ interface in your <>. The command: [source,shell] .... tcpdump -c 4000 -s 10000 -w dumpfile.bin .... will capture 4000 raw packets to _dumpfile.bin_. Up to 10,000 bytes per packet will be captured in this example. [[experiment]] == The Experiment Here is the experiment: [.procedure] ==== . Open a window to an IPsec host and another window to an insecure host. . Now start <>. . In the "secure" window, run the UNIX(R) command man:yes[1], which will stream the `y` character. After a while, stop this. Switch to the insecure window, and repeat. After a while, stop. . Now run <> on the captured packets. You should see something like the following. The important thing to note is that the secure connection has 93% (6.7) of the expected value (7.18), and the "normal" connection has 29% (2.1) of the expected value. + [source,shell] .... % tcpdump -c 4000 -s 10000 -w ipsecdemo.bin % uliscan ipsecdemo.bin Uliscan 21 Dec 98 L=8 256 258560 Measuring file ipsecdemo.bin Init done Expected value for L=8 is 7.1836656 6.9396 -------------------------------------------------------- 6.6177 ----------------------------------------------------- 6.4100 --------------------------------------------------- 2.1101 ----------------- 2.0838 ----------------- 2.0983 ----------------- .... ==== [[caveat]] == Caveat This experiment shows that IPsec _does_ seem to be distributing the payload data __uniformly__, as encryption should. However, the experiment described here _cannot_ detect many possible flaws in a system (none of which do I have any evidence for). These include poor key generation or exchange, data or keys being visible to others, use of weak algorithms, kernel subversion, etc. Study the source; know the code. [[IPsec]] == IPsec---Definition Internet Protocol security extensions to IPv4; required for IPv6. A protocol for negotiating encryption and authentication at the IP (host-to-host) level. SSL secures only one application socket; SSH secures only a login; PGP secures only a specified file or message. IPsec encrypts everything between two hosts. [[ipsec-install]] == Installing IPsec Most of the modern versions of FreeBSD have IPsec support in their base source. So you will need to include the `IPSEC` option in your kernel config and, after kernel rebuild and reinstall, configure IPsec connections using man:setkey[8] command. A comprehensive guide on running IPsec on FreeBSD is provided in link:{handbook}#ipsec[FreeBSD Handbook]. [[kernel]] == src/sys/i386/conf/KERNELNAME This needs to be present in the kernel config file in order to capture network data with man:tcpdump[1]. Be sure to run man:config[8] after adding this, and rebuild and reinstall. [.programlisting] .... device bpf .... [[code]] == Maurer's Universal Statistical Test (for block size=8 bits) You can find the same code at https://web.archive.org/web/20031204230654/http://www.geocities.com:80/SiliconValley/Code/4704/uliscanc.txt[this link]. [.programlisting] .... /* ULISCAN.c ---blocksize of 8 1 Oct 98 1 Dec 98 21 Dec 98 uliscan.c derived from ueli8.c This version has // comments removed for Sun cc This implements Ueli M Maurer's "Universal Statistical Test for Random Bit Generators" using L=8 Accepts a filename on the command line; writes its results, with other info, to stdout. Handles input file exhaustion gracefully. Ref: J. Cryptology v 5 no 2, 1992 pp 89-105 also on the web somewhere, which is where I found it. -David Honig honig@sprynet.com Usage: ULISCAN filename outputs to stdout */ #define L 8 #define V (1< #include int main(argc, argv) int argc; char **argv; { FILE *fptr; int i,j; int b, c; int table[V]; double sum = 0.0; int iproduct = 1; int run; extern double log(/* double x */); printf("Uliscan 21 Dec 98 \nL=%d %d %d \n", L, V, MAXSAMP); if (argc < 2) { printf("Usage: Uliscan filename\n"); exit(-1); } else { printf("Measuring file %s\n", argv[1]); } fptr = fopen(argv[1],"rb"); if (fptr == NULL) { printf("Can't find %s\n", argv[1]); exit(-1); } for (i = 0; i < V; i++) { table[i] = 0; } for (i = 0; i < Q; i++) { b = fgetc(fptr); table[b] = i; } printf("Init done\n"); printf("Expected value for L=8 is 7.1836656\n"); run = 1; while (run) { sum = 0.0; iproduct = 1; if (run) for (i = Q; run && i < Q + K; i++) { j = i; b = fgetc(fptr); if (b < 0) run = 0; if (run) { if (table[b] > j) j += K; sum += log((double)(j-table[b])); table[b] = i; } } if (!run) printf("Premature end of file; read %d blocks.\n", i - Q); sum = (sum/((double)(i - Q))) / log(2.0); printf("%4.4f ", sum); for (i = 0; i < (int)(sum*8.0 + 0.50); i++) printf("-"); printf("\n"); /* refill initial table */ if (0) { for (i = 0; i < Q; i++) { b = fgetc(fptr); if (b < 0) { run = 0; } else { table[b] = i; } } } } } .... diff --git a/documentation/content/en/articles/leap-seconds/_index.adoc b/documentation/content/en/articles/leap-seconds/_index.adoc index b2909b28cc..1a2ac552bb 100644 --- a/documentation/content/en/articles/leap-seconds/_index.adoc +++ b/documentation/content/en/articles/leap-seconds/_index.adoc @@ -1,85 +1,95 @@ --- title: FreeBSD Support for Leap Seconds description: A short description of how leap seconds are handled on FreeBSD tags: ["Leap Seconds", "Support", "Verification", "FreeBSD"] --- = FreeBSD Support for Leap Seconds :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] ''' toc::[] [[leapseconds-definition]] == Introduction A _leap second_ is an one second adjustment made at specific times of year to UTC to synchronize atomic time scales with variations in the rotation of the Earth. This article describes how FreeBSD interacts with leap seconds. As of this writing, the next leap second will occur at 2015-Jun-30 23:59:60 UTC. This leap second will occur during a business day for North and South America and the Asia/Pacific region. Leap seconds are announced by http://datacenter.iers.org/[IERS] on http://datacenter.iers.org/web/guest/bulletins/-/somos/5Rgv/product/16[Bulletin C]. Standard leap second behavior is described in https://tools.ietf.org/html/rfc7164#section-3[RFC 7164]. Also see man:time2posix[3]. [[leapseconds-posix]] == Default Leap Second Handling on FreeBSD The easiest way to handle leap seconds is with the POSIX time rules FreeBSD uses by default, combined with link:{handbook}#network-ntp[NTP]. When man:ntpd[8] is running and the time is synchronized with upstream NTP servers that handle leap seconds correctly, the leap second will cause the system time to automatically repeat the last second of the day. No other adjustments are necessary. If the upstream NTP servers do not handle leap seconds correctly, man:ntpd[8] will step the time by one second after the errant upstream server has noticed and stepped itself. If NTP is not being used, manual adjustment of the system clock will be required after the leap second has passed. [[leapseconds-cautions]] == Cautions Leap seconds are inserted at the same instant all over the world: UTC midnight. In Japan that is mid-morning, in the Pacific mid-day, in the Americas late afternoon, and in Europe at night. We believe and expect that FreeBSD, if provided correct and stable NTP service, will work as designed during this leap second, as it did during the previous ones. However, we caution that practically no applications have ever asked the kernel about leap seconds. Our experience is that, as designed, leap seconds are essentially a replay of the second before the leap second, and this is a surprise to most application programmers. Other operating systems and other computers may or may not handle the leap-second the same way as FreeBSD, and systems without correct and stable NTP service will not know anything about leap seconds at all. It is not unheard of for computers to crash because of leap seconds, and experience has shown that a large fraction of all public NTP servers might handle and announce the leap second incorrectly. Please try to make sure nothing horrible happens because of the leap second. [[leapseconds-testing]] == Testing It is possible to test whether a leap second will be used. Due to the nature of NTP, the test might work up to 24 hours before the leap second. Some major reference clock sources only announce leap seconds one hour ahead of the event. Query the NTP daemon: [source,shell] .... % ntpq -c 'rv 0 leap' .... Output that includes `leap_add_sec` indicates proper support of the leap second. Before the 24 hours leading up to the leap second, or after the leap second has passed, `leap_none` will be shown. [[leapseconds-conclusion]] == Conclusion In practice, leap seconds are usually not a problem on FreeBSD. We hope that this overview helps clarify what to expect and how to make the leap second event proceed more smoothly. diff --git a/documentation/content/en/articles/linux-emulation/_index.adoc b/documentation/content/en/articles/linux-emulation/_index.adoc index 449654c254..6512d55b80 100644 --- a/documentation/content/en/articles/linux-emulation/_index.adoc +++ b/documentation/content/en/articles/linux-emulation/_index.adoc @@ -1,1417 +1,1427 @@ --- title: Linux® emulation in FreeBSD authors: - author: Roman Divacky email: rdivacky@FreeBSD.org description: A technical description about the internals of the Linux emulation layer in FreeBSD trademarks: ["freebsd", "ibm", "adobe", "netbsd", "realnetworks", "oracle", "linux", "sun", "general"] tags: ["Emulation", "Linuxulator", "kernel", "FreeBSD"] --- = Linux(R) emulation in FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +endif::[] [.abstract-title] Abstract This masters thesis deals with updating the Linux(R) emulation layer (the so called _Linuxulator_). The task was to update the layer to match the functionality of Linux(R) 2.6. As a reference implementation, the Linux(R) 2.6.16 kernel was chosen. The concept is loosely based on the NetBSD implementation. Most of the work was done in the summer of 2006 as a part of the Google Summer of Code students program. The focus was on bringing the _NPTL_ (new POSIX(R) thread library) support into the emulation layer, including _TLS_ (thread local storage), _futexes_ (fast user space mutexes), _PID mangling_, and some other minor things. Many small problems were identified and fixed in the process. My work was integrated into the main FreeBSD source repository and will be shipped in the upcoming 7.0R release. We, the emulation development team, are working on making the Linux(R) 2.6 emulation the default emulation layer in FreeBSD. ''' toc::[] [[intro]] == Introduction In the last few years the open source UNIX(R) based operating systems started to be widely deployed on server and client machines. Among these operating systems I would like to point out two: FreeBSD, for its BSD heritage, time proven code base and many interesting features and Linux(R) for its wide user base, enthusiastic open developer community and support from large companies. FreeBSD tends to be used on server class machines serving heavy duty networking tasks with less usage on desktop class machines for ordinary users. While Linux(R) has the same usage on servers, but it is used much more by home based users. This leads to a situation where there are many binary only programs available for Linux(R) that lack support for FreeBSD. Naturally, a need for the ability to run Linux(R) binaries on a FreeBSD system arises and this is what this thesis deals with: the emulation of the Linux(R) kernel in the FreeBSD operating system. During the Summer of 2006 Google Inc. sponsored a project which focused on extending the Linux(R) emulation layer (the so called Linuxulator) in FreeBSD to include Linux(R) 2.6 facilities. This thesis is written as a part of this project. [[inside]] == A look inside... In this section we are going to describe every operating system in question. How they deal with syscalls, trapframes etc., all the low-level stuff. We also describe the way they understand common UNIX(R) primitives like what a PID is, what a thread is, etc. In the third subsection we talk about how UNIX(R) on UNIX(R) emulation could be done in general. [[what-is-unix]] === What is UNIX(R) UNIX(R) is an operating system with a long history that has influenced almost every other operating system currently in use. Starting in the 1960s, its development continues to this day (although in different projects). UNIX(R) development soon forked into two main ways: the BSDs and System III/V families. They mutually influenced themselves by growing a common UNIX(R) standard. Among the contributions originated in BSD we can name virtual memory, TCP/IP networking, FFS, and many others. The System V branch contributed to SysV interprocess communication primitives, copy-on-write, etc. UNIX(R) itself does not exist any more but its ideas have been used by many other operating systems world wide thus forming the so called UNIX(R)-like operating systems. These days the most influential ones are Linux(R), Solaris, and possibly (to some extent) FreeBSD. There are in-company UNIX(R) derivatives (AIX, HP-UX etc.), but these have been more and more migrated to the aforementioned systems. Let us summarize typical UNIX(R) characteristics. [[tech-details]] === Technical details Every running program constitutes a process that represents a state of the computation. Running process is divided between kernel-space and user-space. Some operations can be done only from kernel space (dealing with hardware etc.), but the process should spend most of its lifetime in the user space. The kernel is where the management of the processes, hardware, and low-level details take place. The kernel provides a standard unified UNIX(R) API to the user space. The most important ones are covered below. [[kern-proc-comm]] ==== Communication between kernel and user space process Common UNIX(R) API defines a syscall as a way to issue commands from a user space process to the kernel. The most common implementation is either by using an interrupt or specialized instruction (think of `SYSENTER`/`SYSCALL` instructions for ia32). Syscalls are defined by a number. For example in FreeBSD, the syscall number 85 is the man:swapon[2] syscall and the syscall number 132 is man:mkfifo[2]. Some syscalls need parameters, which are passed from the user-space to the kernel-space in various ways (implementation dependant). Syscalls are synchronous. Another possible way to communicate is by using a _trap_. Traps occur asynchronously after some event occurs (division by zero, page fault etc.). A trap can be transparent for a process (page fault) or can result in a reaction like sending a _signal_ (division by zero). [[proc-proc-comm]] ==== Communication between processes There are other APIs (System V IPC, shared memory etc.) but the single most important API is signal. Signals are sent by processes or by the kernel and received by processes. Some signals can be ignored or handled by a user supplied routine, some result in a predefined action that cannot be altered or ignored. [[proc-mgmt]] ==== Process management Kernel instances are processed first in the system (so called init). Every running process can create its identical copy using the man:fork[2] syscall. Some slightly modified versions of this syscall were introduced but the basic semantic is the same. Every running process can morph into some other process using the man:exec[3] syscall. Some modifications of this syscall were introduced but all serve the same basic purpose. Processes end their lives by calling the man:exit[2] syscall. Every process is identified by a unique number called PID. Every process has a defined parent (identified by its PID). [[thread-mgmt]] ==== Thread management Traditional UNIX(R) does not define any API nor implementation for threading, while POSIX(R) defines its threading API but the implementation is undefined. Traditionally there were two ways of implementing threads. Handling them as separate processes (1:1 threading) or envelope the whole thread group in one process and managing the threading in userspace (1:N threading). Comparing main features of each approach: 1:1 threading * - heavyweight threads * - the scheduling cannot be altered by the user (slightly mitigated by the POSIX(R) API) * + no syscall wrapping necessary * + can utilize multiple CPUs 1:N threading * + lightweight threads * + scheduling can be easily altered by the user * - syscalls must be wrapped * - cannot utilize more than one CPU [[what-is-freebsd]] === What is FreeBSD? The FreeBSD project is one of the oldest open source operating systems currently available for daily use. It is a direct descendant of the genuine UNIX(R) so it could be claimed that it is a true UNIX(R) although licensing issues do not permit that. The start of the project dates back to the early 1990's when a crew of fellow BSD users patched the 386BSD operating system. Based on this patchkit a new operating system arose named FreeBSD for its liberal license. Another group created the NetBSD operating system with different goals in mind. We will focus on FreeBSD. FreeBSD is a modern UNIX(R)-based operating system with all the features of UNIX(R). Preemptive multitasking, multiuser facilities, TCP/IP networking, memory protection, symmetric multiprocessing support, virtual memory with merged VM and buffer cache, they are all there. One of the interesting and extremely useful features is the ability to emulate other UNIX(R)-like operating systems. As of December 2006 and 7-CURRENT development, the following emulation functionalities are supported: * FreeBSD/i386 emulation on FreeBSD/amd64 * FreeBSD/i386 emulation on FreeBSD/ia64 * Linux(R)-emulation of Linux(R) operating system on FreeBSD * NDIS-emulation of Windows networking drivers interface * NetBSD-emulation of NetBSD operating system * PECoff-support for PECoff FreeBSD executables * SVR4-emulation of System V revision 4 UNIX(R) Actively developed emulations are the Linux(R) layer and various FreeBSD-on-FreeBSD layers. Others are not supposed to work properly nor be usable these days. [[freebsd-tech-details]] ==== Technical details FreeBSD is traditional flavor of UNIX(R) in the sense of dividing the run of processes into two halves: kernel space and user space run. There are two types of process entry to the kernel: a syscall and a trap. There is only one way to return. In the subsequent sections we will describe the three gates to/from the kernel. The whole description applies to the i386 architecture as the Linuxulator only exists there but the concept is similar on other architectures. The information was taken from [1] and the source code. [[freebsd-sys-entries]] ===== System entries FreeBSD has an abstraction called an execution class loader, which is a wedge into the man:execve[2] syscall. This employs a structure `sysentvec`, which describes an executable ABI. It contains things like errno translation table, signal translation table, various functions to serve syscall needs (stack fixup, coredumping, etc.). Every ABI the FreeBSD kernel wants to support must define this structure, as it is used later in the syscall processing code and at some other places. System entries are handled by trap handlers, where we can access both the kernel-space and the user-space at once. [[freebsd-syscalls]] ===== Syscalls Syscalls on FreeBSD are issued by executing interrupt `0x80` with register `%eax` set to a desired syscall number with arguments passed on the stack. When a process issues an interrupt `0x80`, the `int0x80` syscall trap handler is issued (defined in [.filename]#sys/i386/i386/exception.s#), which prepares arguments (i.e. copies them on to the stack) for a call to a C function man:syscall[2] (defined in [.filename]#sys/i386/i386/trap.c#), which processes the passed in trapframe. The processing consists of preparing the syscall (depending on the `sysvec` entry), determining if the syscall is 32-bit or 64-bit one (changes size of the parameters), then the parameters are copied, including the syscall. Next, the actual syscall function is executed with processing of the return code (special cases for `ERESTART` and `EJUSTRETURN` errors). Finally an `userret()` is scheduled, switching the process back to the users-pace. The parameters to the actual syscall handler are passed in the form of `struct thread *td`, `struct syscall args *` arguments where the second parameter is a pointer to the copied in structure of parameters. [[freebsd-traps]] ===== Traps Handling of traps in FreeBSD is similar to the handling of syscalls. Whenever a trap occurs, an assembler handler is called. It is chosen between alltraps, alltraps with regs pushed or calltrap depending on the type of the trap. This handler prepares arguments for a call to a C function `trap()` (defined in [.filename]#sys/i386/i386/trap.c#), which then processes the occurred trap. After the processing it might send a signal to the process and/or exit to userland using `userret()`. [[freebsd-exits]] ===== Exits Exits from kernel to userspace happen using the assembler routine `doreti` regardless of whether the kernel was entered via a trap or via a syscall. This restores the program status from the stack and returns to the userspace. [[freebsd-unix-primitives]] ===== UNIX(R) primitives FreeBSD operating system adheres to the traditional UNIX(R) scheme, where every process has a unique identification number, the so called _PID_ (Process ID). PID numbers are allocated either linearly or randomly ranging from `0` to `PID_MAX`. The allocation of PID numbers is done using linear searching of PID space. Every thread in a process receives the same PID number as result of the man:getpid[2] call. There are currently two ways to implement threading in FreeBSD. The first way is M:N threading followed by the 1:1 threading model. The default library used is M:N threading (`libpthread`) and you can switch at runtime to 1:1 threading (`libthr`). The plan is to switch to 1:1 library by default soon. Although those two libraries use the same kernel primitives, they are accessed through different API(es). The M:N library uses the `kse_*` family of syscalls while the 1:1 library uses the `thr_*` family of syscalls. Due to this, there is no general concept of thread ID shared between kernel and userspace. Of course, both threading libraries implement the pthread thread ID API. Every kernel thread (as described by `struct thread`) has td tid identifier but this is not directly accessible from userland and solely serves the kernel's needs. It is also used for 1:1 threading library as pthread's thread ID but handling of this is internal to the library and cannot be relied on. As stated previously there are two implementations of threading in FreeBSD. The M:N library divides the work between kernel space and userspace. Thread is an entity that gets scheduled in the kernel but it can represent various number of userspace threads. M userspace threads get mapped to N kernel threads thus saving resources while keeping the ability to exploit multiprocessor parallelism. Further information about the implementation can be obtained from the man page or [1]. The 1:1 library directly maps a userland thread to a kernel thread thus greatly simplifying the scheme. None of these designs implement a fairness mechanism (such a mechanism was implemented but it was removed recently because it caused serious slowdown and made the code more difficult to deal with). [[what-is-linux]] === What is Linux(R) Linux(R) is a UNIX(R)-like kernel originally developed by Linus Torvalds, and now being contributed to by a massive crowd of programmers all around the world. From its mere beginnings to today, with wide support from companies such as IBM or Google, Linux(R) is being associated with its fast development pace, full hardware support and benevolent dictator model of organization. Linux(R) development started in 1991 as a hobbyist project at University of Helsinki in Finland. Since then it has obtained all the features of a modern UNIX(R)-like OS: multiprocessing, multiuser support, virtual memory, networking, basically everything is there. There are also highly advanced features like virtualization etc. As of 2006 Linux(R) seems to be the most widely used open source operating system with support from independent software vendors like Oracle, RealNetworks, Adobe, etc. Most of the commercial software distributed for Linux(R) can only be obtained in a binary form so recompilation for other operating systems is impossible. Most of the Linux(R) development happens in a Git version control system. Git is a distributed system so there is no central source of the Linux(R) code, but some branches are considered prominent and official. The version number scheme implemented by Linux(R) consists of four numbers A.B.C.D. Currently development happens in 2.6.C.D, where C represents major version, where new features are added or changed while D is a minor version for bugfixes only. More information can be obtained from [3]. [[linux-tech-details]] ==== Technical details Linux(R) follows the traditional UNIX(R) scheme of dividing the run of a process in two halves: the kernel and user space. The kernel can be entered in two ways: via a trap or via a syscall. The return is handled only in one way. The further description applies to Linux(R) 2.6 on the i386(TM) architecture. This information was taken from [2]. [[linux-syscalls]] ===== Syscalls Syscalls in Linux(R) are performed (in userspace) using `syscallX` macros where X substitutes a number representing the number of parameters of the given syscall. This macro translates to a code that loads `%eax` register with a number of the syscall and executes interrupt `0x80`. After this syscall return is called, which translates negative return values to positive `errno` values and sets `res` to `-1` in case of an error. Whenever the interrupt `0x80` is called the process enters the kernel in system call trap handler. This routine saves all registers on the stack and calls the selected syscall entry. Note that the Linux(R) calling convention expects parameters to the syscall to be passed via registers as shown here: . parameter -> `%ebx` . parameter -> `%ecx` . parameter -> `%edx` . parameter -> `%esi` . parameter -> `%edi` . parameter -> `%ebp` There are some exceptions to this, where Linux(R) uses different calling convention (most notably the `clone` syscall). [[linux-traps]] ===== Traps The trap handlers are introduced in [.filename]#arch/i386/kernel/traps.c# and most of these handlers live in [.filename]#arch/i386/kernel/entry.S#, where handling of the traps happens. [[linux-exits]] ===== Exits Return from the syscall is managed by syscall man:exit[3], which checks for the process having unfinished work, then checks whether we used user-supplied selectors. If this happens stack fixing is applied and finally the registers are restored from the stack and the process returns to the userspace. [[linux-unix-primitives]] ===== UNIX(R) primitives In the 2.6 version, the Linux(R) operating system redefined some of the traditional UNIX(R) primitives, notably PID, TID and thread. PID is defined not to be unique for every process, so for some processes (threads) man:getppid[2] returns the same value. Unique identification of process is provided by TID. This is because _NPTL_ (New POSIX(R) Thread Library) defines threads to be normal processes (so called 1:1 threading). Spawning a new process in Linux(R) 2.6 happens using the `clone` syscall (fork variants are reimplemented using it). This clone syscall defines a set of flags that affect behavior of the cloning process regarding thread implementation. The semantic is a bit fuzzy as there is no single flag telling the syscall to create a thread. Implemented clone flags are: * `CLONE_VM` - processes share their memory space * `CLONE_FS` - share umask, cwd and namespace * `CLONE_FILES` - share open files * `CLONE_SIGHAND` - share signal handlers and blocked signals * `CLONE_PARENT` - share parent * `CLONE_THREAD` - be thread (further explanation below) * `CLONE_NEWNS` - new namespace * `CLONE_SYSVSEM` - share SysV undo structures * `CLONE_SETTLS` - setup TLS at supplied address * `CLONE_PARENT_SETTID` - set TID in the parent * `CLONE_CHILD_CLEARTID` - clear TID in the child * `CLONE_CHILD_SETTID` - set TID in the child `CLONE_PARENT` sets the real parent to the parent of the caller. This is useful for threads because if thread A creates thread B we want thread B to be parented to the parent of the whole thread group. `CLONE_THREAD` does exactly the same thing as `CLONE_PARENT`, `CLONE_VM` and `CLONE_SIGHAND`, rewrites PID to be the same as PID of the caller, sets exit signal to be none and enters the thread group. `CLONE_SETTLS` sets up GDT entries for TLS handling. The `CLONE_*_*TID` set of flags sets/clears user supplied address to TID or 0. As you can see the `CLONE_THREAD` does most of the work and does not seem to fit the scheme very well. The original intention is unclear (even for authors, according to comments in the code) but I think originally there was one threading flag, which was then parcelled among many other flags but this separation was never fully finished. It is also unclear what this partition is good for as glibc does not use that so only hand-written use of the clone permits a programmer to access this features. For non-threaded programs the PID and TID are the same. For threaded programs the first thread PID and TID are the same and every created thread shares the same PID and gets assigned a unique TID (because `CLONE_THREAD` is passed in) also parent is shared for all processes forming this threaded program. The code that implements man:pthread_create[3] in NPTL defines the clone flags like this: [.programlisting] .... int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGNAL | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID | CLONE_SYSVSEM #if __ASSUME_NO_CLONE_DETACHED == 0 | CLONE_DETACHED #endif | 0); .... The `CLONE_SIGNAL` is defined like [.programlisting] .... #define CLONE_SIGNAL (CLONE_SIGHAND | CLONE_THREAD) .... the last 0 means no signal is sent when any of the threads exits. [[what-is-emu]] === What is emulation According to a dictionary definition, emulation is the ability of a program or device to imitate another program or device. This is achieved by providing the same reaction to a given stimulus as the emulated object. In practice, the software world mostly sees three types of emulation - a program used to emulate a machine (QEMU, various game console emulators etc.), software emulation of a hardware facility (OpenGL emulators, floating point units emulation etc.) and operating system emulation (either in kernel of the operating system or as a userspace program). Emulation is usually used in a place, where using the original component is not feasible nor possible at all. For example someone might want to use a program developed for a different operating system than they use. Then emulation comes in handy. Sometimes there is no other way but to use emulation - e.g. when the hardware device you try to use does not exist (yet/anymore) then there is no other way but emulation. This happens often when porting an operating system to a new (non-existent) platform. Sometimes it is just cheaper to emulate. Looking from an implementation point of view, there are two main approaches to the implementation of emulation. You can either emulate the whole thing - accepting possible inputs of the original object, maintaining inner state and emitting correct output based on the state and/or input. This kind of emulation does not require any special conditions and basically can be implemented anywhere for any device/program. The drawback is that implementing such emulation is quite difficult, time-consuming and error-prone. In some cases we can use a simpler approach. Imagine you want to emulate a printer that prints from left to right on a printer that prints from right to left. It is obvious that there is no need for a complex emulation layer but simply reversing of the printed text is sufficient. Sometimes the emulating environment is very similar to the emulated one so just a thin layer of some translation is necessary to provide fully working emulation! As you can see this is much less demanding to implement, so less time-consuming and error-prone than the previous approach. But the necessary condition is that the two environments must be similar enough. The third approach combines the two previous. Most of the time the objects do not provide the same capabilities so in a case of emulating the more powerful one on the less powerful we have to emulate the missing features with full emulation described above. This master thesis deals with emulation of UNIX(R) on UNIX(R), which is exactly the case, where only a thin layer of translation is sufficient to provide full emulation. The UNIX(R) API consists of a set of syscalls, which are usually self contained and do not affect some global kernel state. There are a few syscalls that affect inner state but this can be dealt with by providing some structures that maintain the extra state. No emulation is perfect and emulations tend to lack some parts but this usually does not cause any serious drawbacks. Imagine a game console emulator that emulates everything but music output. No doubt that the games are playable and one can use the emulator. It might not be that comfortable as the original game console but its an acceptable compromise between price and comfort. The same goes with the UNIX(R) API. Most programs can live with a very limited set of syscalls working. Those syscalls tend to be the oldest ones (man:read[2]/man:write[2], man:fork[2] family, man:signal[3] handling, man:exit[3], man:socket[2] API) hence it is easy to emulate because their semantics is shared among all UNIX(R)es, which exist todays. [[freebsd-emulation]] == Emulation === How emulation works in FreeBSD As stated earlier, FreeBSD supports running binaries from several other UNIX(R)es. This works because FreeBSD has an abstraction called the execution class loader. This wedges into the man:execve[2] syscall, so when man:execve[2] is about to execute a binary it examines its type. There are basically two types of binaries in FreeBSD. Shell-like text scripts which are identified by `#!` as their first two characters and normal (typically _ELF_) binaries, which are a representation of a compiled executable object. The vast majority (one could say all of them) of binaries in FreeBSD are from type ELF. ELF files contain a header, which specifies the OS ABI for this ELF file. By reading this information, the operating system can accurately determine what type of binary the given file is. Every OS ABI must be registered in the FreeBSD kernel. This applies to the FreeBSD native OS ABI, as well. So when man:execve[2] executes a binary it iterates through the list of registered APIs and when it finds the right one it starts to use the information contained in the OS ABI description (its syscall table, `errno` translation table, etc.). So every time the process calls a syscall, it uses its own set of syscalls instead of some global one. This effectively provides a very elegant and easy way of supporting execution of various binary formats. The nature of emulation of different OSes (and also some other subsystems) led developers to invite a handler event mechanism. There are various places in the kernel, where a list of event handlers are called. Every subsystem can register an event handler and they are called accordingly. For example, when a process exits there is a handler called that possibly cleans up whatever the subsystem needs to be cleaned. Those simple facilities provide basically everything that is needed for the emulation infrastructure and in fact these are basically the only things necessary to implement the Linux(R) emulation layer. [[freebsd-common-primitives]] === Common primitives in the FreeBSD kernel Emulation layers need some support from the operating system. I am going to describe some of the supported primitives in the FreeBSD operating system. [[freebsd-locking-primitives]] ==== Locking primitives Contributed by: `{attilio}` The FreeBSD synchronization primitive set is based on the idea to supply a rather huge number of different primitives in a way that the better one can be used for every particular, appropriate situation. To a high level point of view you can consider three kinds of synchronization primitives in the FreeBSD kernel: * atomic operations and memory barriers * locks * scheduling barriers Below there are descriptions for the 3 families. For every lock, you should really check the linked manpage (where possible) for more detailed explanations. [[freebsd-atomic-op]] ===== Atomic operations and memory barriers Atomic operations are implemented through a set of functions performing simple arithmetics on memory operands in an atomic way with respect to external events (interrupts, preemption, etc.). Atomic operations can guarantee atomicity just on small data types (in the magnitude order of the `.long.` architecture C data type), so should be rarely used directly in the end-level code, if not only for very simple operations (like flag setting in a bitmap, for example). In fact, it is rather simple and common to write down a wrong semantic based on just atomic operations (usually referred as lock-less). The FreeBSD kernel offers a way to perform atomic operations in conjunction with a memory barrier. The memory barriers will guarantee that an atomic operation will happen following some specified ordering with respect to other memory accesses. For example, if we need that an atomic operation happen just after all other pending writes (in terms of instructions reordering buffers activities) are completed, we need to explicitly use a memory barrier in conjunction to this atomic operation. So it is simple to understand why memory barriers play a key role for higher-level locks building (just as refcounts, mutexes, etc.). For a detailed explanatory on atomic operations, please refer to man:atomic[9]. It is far, however, noting that atomic operations (and memory barriers as well) should ideally only be used for building front-ending locks (as mutexes). [[freebsd-refcounts]] ===== Refcounts Refcounts are interfaces for handling reference counters. They are implemented through atomic operations and are intended to be used just for cases, where the reference counter is the only one thing to be protected, so even something like a spin-mutex is deprecated. Using the refcount interface for structures, where a mutex is already used is often wrong since we should probably close the reference counter in some already protected paths. A manpage discussing refcount does not exist currently, just check [.filename]#sys/refcount.h# for an overview of the existing API. [[freebsd-locks]] ===== Locks FreeBSD kernel has huge classes of locks. Every lock is defined by some peculiar properties, but probably the most important is the event linked to contesting holders (or in other terms, the behavior of threads unable to acquire the lock). FreeBSD's locking scheme presents three different behaviors for contenders: . spinning . blocking . sleeping [NOTE] ==== numbers are not casual ==== [[freebsd-spinlocks]] ===== Spinning locks Spin locks let waiters to spin until they cannot acquire the lock. An important matter do deal with is when a thread contests on a spin lock if it is not descheduled. Since the FreeBSD kernel is preemptive, this exposes spin lock at the risk of deadlocks that can be solved just disabling interrupts while they are acquired. For this and other reasons (like lack of priority propagation support, poorness in load balancing schemes between CPUs, etc.), spin locks are intended to protect very small paths of code, or ideally not to be used at all if not explicitly requested (explained later). [[freebsd-blocking]] ===== Blocking Block locks let waiters to be descheduled and blocked until the lock owner does not drop it and wakes up one or more contenders. In order to avoid starvation issues, blocking locks do priority propagation from the waiters to the owner. Block locks must be implemented through the turnstile interface and are intended to be the most used kind of locks in the kernel, if no particular conditions are met. [[freebsd-sleeping]] ===== Sleeping Sleep locks let waiters to be descheduled and fall asleep until the lock holder does not drop it and wakes up one or more waiters. Since sleep locks are intended to protect large paths of code and to cater asynchronous events, they do not do any form of priority propagation. They must be implemented through the man:sleepqueue[9] interface. The order used to acquire locks is very important, not only for the possibility to deadlock due at lock order reversals, but even because lock acquisition should follow specific rules linked to locks natures. If you give a look at the table above, the practical rule is that if a thread holds a lock of level n (where the level is the number listed close to the kind of lock) it is not allowed to acquire a lock of superior levels, since this would break the specified semantic for a path. For example, if a thread holds a block lock (level 2), it is allowed to acquire a spin lock (level 1) but not a sleep lock (level 3), since block locks are intended to protect smaller paths than sleep lock (these rules are not about atomic operations or scheduling barriers, however). This is a list of lock with their respective behaviors: * spin mutex - spinning - man:mutex[9] * sleep mutex - blocking - man:mutex[9] * pool mutex - blocking - man:mtx[pool] * sleep family - sleeping - man:sleep[9] pause tsleep msleep msleep spin msleep rw msleep sx * condvar - sleeping - man:condvar[9] * rwlock - blocking - man:rwlock[9] * sxlock - sleeping - man:sx[9] * lockmgr - sleeping - man:lockmgr[9] * semaphores - sleeping - man:sema[9] Among these locks only mutexes, sxlocks, rwlocks and lockmgrs are intended to handle recursion, but currently recursion is only supported by mutexes and lockmgrs. [[freebsd-scheduling]] ===== Scheduling barriers Scheduling barriers are intended to be used in order to drive scheduling of threading. They consist mainly of three different stubs: * critical sections (and preemption) * sched_bind * sched_pin Generally, these should be used only in a particular context and even if they can often replace locks, they should be avoided because they do not let the diagnose of simple eventual problems with locking debugging tools (as man:witness[4]). [[freebsd-critical]] ===== Critical sections The FreeBSD kernel has been made preemptive basically to deal with interrupt threads. In fact, in order to avoid high interrupt latency, time-sharing priority threads can be preempted by interrupt threads (in this way, they do not need to wait to be scheduled as the normal path previews). Preemption, however, introduces new racing points that need to be handled, as well. Often, in order to deal with preemption, the simplest thing to do is to completely disable it. A critical section defines a piece of code (borderlined by the pair of functions man:critical_enter[9] and man:critical_exit[9], where preemption is guaranteed to not happen (until the protected code is fully executed). This can often replace a lock effectively but should be used carefully in order to not lose the whole advantage that preemption brings. [[freebsd-schedpin]] ===== sched_pin/sched_unpin Another way to deal with preemption is the `sched_pin()` interface. If a piece of code is closed in the `sched_pin()` and `sched_unpin()` pair of functions it is guaranteed that the respective thread, even if it can be preempted, it will always be executed on the same CPU. Pinning is very effective in the particular case when we have to access at per-cpu datas and we assume other threads will not change those data. The latter condition will determine a critical section as a too strong condition for our code. [[freebsd-schedbind]] ===== sched_bind/sched_unbind `sched_bind` is an API used in order to bind a thread to a particular CPU for all the time it executes the code, until a `sched_unbind` function call does not unbind it. This feature has a key role in situations where you cannot trust the current state of CPUs (for example, at very early stages of boot), as you want to avoid your thread to migrate on inactive CPUs. Since `sched_bind` and `sched_unbind` manipulate internal scheduler structures, they need to be enclosed in `sched_lock` acquisition/releasing when used. [[freebsd-proc]] ==== Proc structure Various emulation layers sometimes require some additional per-process data. It can manage separate structures (a list, a tree etc.) containing these data for every process but this tends to be slow and memory consuming. To solve this problem the FreeBSD `proc` structure contains `p_emuldata`, which is a void pointer to some emulation layer specific data. This `proc` entry is protected by the proc mutex. The FreeBSD `proc` structure contains a `p_sysent` entry that identifies, which ABI this process is running. In fact, it is a pointer to the `sysentvec` described above. So by comparing this pointer to the address where the `sysentvec` structure for the given ABI is stored we can effectively determine whether the process belongs to our emulation layer. The code typically looks like: [.programlisting] .... if (__predict_true(p->p_sysent != &elf_Linux(R)_sysvec)) return; .... As you can see, we effectively use the `__predict_true` modifier to collapse the most common case (FreeBSD process) to a simple return operation thus preserving high performance. This code should be turned into a macro because currently it is not very flexible, i.e. we do not support Linux(R)64 emulation nor A.OUT Linux(R) processes on i386. [[freebsd-vfs]] ==== VFS The FreeBSD VFS subsystem is very complex but the Linux(R) emulation layer uses just a small subset via a well defined API. It can either operate on vnodes or file handlers. Vnode represents a virtual vnode, i.e. representation of a node in VFS. Another representation is a file handler, which represents an opened file from the perspective of a process. A file handler can represent a socket or an ordinary file. A file handler contains a pointer to its vnode. More then one file handler can point to the same vnode. [[freebsd-namei]] ===== namei The man:namei[9] routine is a central entry point to pathname lookup and translation. It traverses the path point by point from the starting point to the end point using lookup function, which is internal to VFS. The man:namei[9] syscall can cope with symlinks, absolute and relative paths. When a path is looked up using man:namei[9] it is inputed to the name cache. This behavior can be suppressed. This routine is used all over the kernel and its performance is very critical. [[freebsd-vn]] ===== vn_fullpath The man:vn_fullpath[9] function takes the best effort to traverse VFS name cache and returns a path for a given (locked) vnode. This process is unreliable but works just fine for the most common cases. The unreliability is because it relies on VFS cache (it does not traverse the on medium structures), it does not work with hardlinks, etc. This routine is used in several places in the Linuxulator. [[freebsd-vnode]] ===== Vnode operations * `fgetvp` - given a thread and a file descriptor number it returns the associated vnode * man:vn_lock[9] - locks a vnode * `vn_unlock` - unlocks a vnode * man:VOP_READDIR[9] - reads a directory referenced by a vnode * man:VOP_GETATTR[9] - gets attributes of a file or a directory referenced by a vnode * man:VOP_LOOKUP[9] - looks up a path to a given directory * man:VOP_OPEN[9] - opens a file referenced by a vnode * man:VOP_CLOSE[9] - closes a file referenced by a vnode * man:vput[9] - decrements the use count for a vnode and unlocks it * man:vrele[9] - decrements the use count for a vnode * man:vref[9] - increments the use count for a vnode [[freebsd-file-handler]] ===== File handler operations * `fget` - given a thread and a file descriptor number it returns associated file handler and references it * `fdrop` - drops a reference to a file handler * `fhold` - references a file handler [[md]] == Linux(R) emulation layer -MD part This section deals with implementation of Linux(R) emulation layer in FreeBSD operating system. It first describes the machine dependent part talking about how and where interaction between userland and kernel is implemented. It talks about syscalls, signals, ptrace, traps, stack fixup. This part discusses i386 but it is written generally so other architectures should not differ very much. The next part is the machine independent part of the Linuxulator. This section only covers i386 and ELF handling. A.OUT is obsolete and untested. [[syscall-handling]] === Syscall handling Syscall handling is mostly written in [.filename]#linux_sysvec.c#, which covers most of the routines pointed out in the `sysentvec` structure. When a Linux(R) process running on FreeBSD issues a syscall, the general syscall routine calls linux prepsyscall routine for the Linux(R) ABI. [[linux-prepsyscall]] ==== Linux(R) prepsyscall Linux(R) passes arguments to syscalls via registers (that is why it is limited to 6 parameters on i386) while FreeBSD uses the stack. The Linux(R) prepsyscall routine must copy parameters from registers to the stack. The order of the registers is: `%ebx`, `%ecx`, `%edx`, `%esi`, `%edi`, `%ebp`. The catch is that this is true for only _most_ of the syscalls. Some (most notably `clone`) uses a different order but it is luckily easy to fix by inserting a dummy parameter in the `linux_clone` prototype. [[syscall-writing]] ==== Syscall writing Every syscall implemented in the Linuxulator must have its prototype with various flags in [.filename]#syscalls.master#. The form of the file is: [.programlisting] .... ... AUE_FORK STD { int linux_fork(void); } ... AUE_CLOSE NOPROTO { int close(int fd); } ... .... The first column represents the syscall number. The second column is for auditing support. The third column represents the syscall type. It is either `STD`, `OBSOL`, `NOPROTO` and `UNIMPL`. `STD` is a standard syscall with full prototype and implementation. `OBSOL` is obsolete and defines just the prototype. `NOPROTO` means that the syscall is implemented elsewhere so do not prepend ABI prefix, etc. `UNIMPL` means that the syscall will be substituted with the `nosys` syscall (a syscall just printing out a message about the syscall not being implemented and returning `ENOSYS`). From [.filename]#syscalls.master# a script generates three files: [.filename]#linux_syscall.h#, [.filename]#linux_proto.h# and [.filename]#linux_sysent.c#. The [.filename]#linux_syscall.h# contains definitions of syscall names and their numerical value, e.g.: [.programlisting] .... ... #define LINUX_SYS_linux_fork 2 ... #define LINUX_SYS_close 6 ... .... The [.filename]#linux_proto.h# contains structure definitions of arguments to every syscall, e.g.: [.programlisting] .... struct linux_fork_args { register_t dummy; }; .... And finally, [.filename]#linux_sysent.c# contains structure describing the system entry table, used to actually dispatch a syscall, e.g.: [.programlisting] .... { 0, (sy_call_t *)linux_fork, AUE_FORK, NULL, 0, 0 }, /* 2 = linux_fork */ { AS(close_args), (sy_call_t *)close, AUE_CLOSE, NULL, 0, 0 }, /* 6 = close */ .... As you can see `linux_fork` is implemented in Linuxulator itself so the definition is of `STD` type and has no argument, which is exhibited by the dummy argument structure. On the other hand `close` is just an alias for real FreeBSD man:close[2] so it has no linux arguments structure associated and in the system entry table it is not prefixed with linux as it calls the real man:close[2] in the kernel. [[dummy-syscalls]] ==== Dummy syscalls The Linux(R) emulation layer is not complete, as some syscalls are not implemented properly and some are not implemented at all. The emulation layer employs a facility to mark unimplemented syscalls with the `DUMMY` macro. These dummy definitions reside in [.filename]#linux_dummy.c# in a form of `DUMMY(syscall);`, which is then translated to various syscall auxiliary files and the implementation consists of printing a message saying that this syscall is not implemented. The `UNIMPL` prototype is not used because we want to be able to identify the name of the syscall that was called in order to know what syscalls are more important to implement. [[signal-handling]] === Signal handling Signal handling is done generally in the FreeBSD kernel for all binary compatibilities with a call to a compat-dependent layer. Linux(R) compatibility layer defines `linux_sendsig` routine for this purpose. [[linux-sendsig]] ==== Linux(R) sendsig This routine first checks whether the signal has been installed with a `SA_SIGINFO` in which case it calls `linux_rt_sendsig` routine instead. Furthermore, it allocates (or reuses an already existing) signal handle context, then it builds a list of arguments for the signal handler. It translates the signal number based on the signal translation table, assigns a handler, translates sigset. Then it saves context for the `sigreturn` routine (various registers, translated trap number and signal mask). Finally, it copies out the signal context to the userspace and prepares context for the actual signal handler to run. [[linux-rt-sendsig]] ==== linux_rt_sendsig This routine is similar to `linux_sendsig` just the signal context preparation is different. It adds `siginfo`, `ucontext`, and some POSIX(R) parts. It might be worth considering whether those two functions could not be merged with a benefit of less code duplication and possibly even faster execution. [[linux-sigreturn]] ==== linux_sigreturn This syscall is used for return from the signal handler. It does some security checks and restores the original process context. It also unmasks the signal in process signal mask. [[ptrace]] === Ptrace Many UNIX(R) derivates implement the man:ptrace[2] syscall in order to allow various tracking and debugging features. This facility enables the tracing process to obtain various information about the traced process, like register dumps, any memory from the process address space, etc. and also to trace the process like in stepping an instruction or between system entries (syscalls and traps). man:ptrace[2] also lets you set various information in the traced process (registers etc.). man:ptrace[2] is a UNIX(R)-wide standard implemented in most UNIX(R)es around the world. Linux(R) emulation in FreeBSD implements the man:ptrace[2] facility in [.filename]#linux_ptrace.c#. The routines for converting registers between Linux(R) and FreeBSD and the actual man:ptrace[2] syscall emulation syscall. The syscall is a long switch block that implements its counterpart in FreeBSD for every man:ptrace[2] command. The man:ptrace[2] commands are mostly equal between Linux(R) and FreeBSD so usually just a small modification is needed. For example, `PT_GETREGS` in Linux(R) operates on direct data while FreeBSD uses a pointer to the data so after performing a (native) man:ptrace[2] syscall, a copyout must be done to preserve Linux(R) semantics. The man:ptrace[2] implementation in Linuxulator has some known weaknesses. There have been panics seen when using `strace` (which is a man:ptrace[2] consumer) in the Linuxulator environment. Also `PT_SYSCALL` is not implemented. [[traps]] === Traps Whenever a Linux(R) process running in the emulation layer traps the trap itself is handled transparently with the only exception of the trap translation. Linux(R) and FreeBSD differs in opinion on what a trap is so this is dealt with here. The code is actually very short: [.programlisting] .... static int translate_traps(int signal, int trap_code) { if (signal != SIGBUS) return signal; switch (trap_code) { case T_PROTFLT: case T_TSSFLT: case T_DOUBLEFLT: case T_PAGEFLT: return SIGSEGV; default: return signal; } } .... [[stack-fixup]] === Stack fixup The RTLD run-time link-editor expects so called AUX tags on stack during an `execve` so a fixup must be done to ensure this. Of course, every RTLD system is different so the emulation layer must provide its own stack fixup routine to do this. So does Linuxulator. The `elf_linux_fixup` simply copies out AUX tags to the stack and adjusts the stack of the user space process to point right after those tags. So RTLD works in a smart way. [[aout-support]] === A.OUT support The Linux(R) emulation layer on i386 also supports Linux(R) A.OUT binaries. Pretty much everything described in the previous sections must be implemented for A.OUT support (beside traps translation and signals sending). The support for A.OUT binaries is no longer maintained, especially the 2.6 emulation does not work with it but this does not cause any problem, as the linux-base in ports probably do not support A.OUT binaries at all. This support will probably be removed in future. Most of the stuff necessary for loading Linux(R) A.OUT binaries is in [.filename]#imgact_linux.c# file. [[mi]] == Linux(R) emulation layer -MI part This section talks about machine independent part of the Linuxulator. It covers the emulation infrastructure needed for Linux(R) 2.6 emulation, the thread local storage (TLS) implementation (on i386) and futexes. Then we talk briefly about some syscalls. [[nptl-desc]] === Description of NPTL One of the major areas of progress in development of Linux(R) 2.6 was threading. Prior to 2.6, the Linux(R) threading support was implemented in the linuxthreads library. The library was a partial implementation of POSIX(R) threading. The threading was implemented using separate processes for each thread using the `clone` syscall to let them share the address space (and other things). The main weaknesses of this approach was that every thread had a different PID, signal handling was broken (from the pthreads perspective), etc. Also the performance was not very good (use of `SIGUSR` signals for threads synchronization, kernel resource consumption, etc.) so to overcome these problems a new threading system was developed and named NPTL. The NPTL library focused on two things but a third thing came along so it is usually considered a part of NPTL. Those two things were embedding of threads into a process structure and futexes. The additional third thing was TLS, which is not directly required by NPTL but the whole NPTL userland library depends on it. Those improvements yielded in much improved performance and standards conformance. NPTL is a standard threading library in Linux(R) systems these days. The FreeBSD Linuxulator implementation approaches the NPTL in three main areas. The TLS, futexes and PID mangling, which is meant to simulate the Linux(R) threads. Further sections describe each of these areas. [[linux26-emu]] === Linux(R) 2.6 emulation infrastructure These sections deal with the way Linux(R) threads are managed and how we simulate that in FreeBSD. [[linux26-runtime]] ==== Runtime determining of 2.6 emulation The Linux(R) emulation layer in FreeBSD supports runtime setting of the emulated version. This is done via man:sysctl[8], namely `compat.linux.osrelease`. Setting this man:sysctl[8] affects runtime behavior of the emulation layer. When set to 2.6.x it sets the value of `linux_use_linux26` while setting to something else keeps it unset. This variable (plus per-prison variables of the very same kind) determines whether 2.6 infrastructure (mainly PID mangling) is used in the code or not. The version setting is done system-wide and this affects all Linux(R) processes. The man:sysctl[8] should not be changed when running any Linux(R) binary as it might harm things. [[linux-proc-thread]] ==== Linux(R) processes and thread identifiers The semantics of Linux(R) threading are a little confusing and uses entirely different nomenclature to FreeBSD. A process in Linux(R) consists of a `struct task` embedding two identifier fields - PID and TGID. PID is _not_ a process ID but it is a thread ID. The TGID identifies a thread group in other words a process. For single-threaded process the PID equals the TGID. The thread in NPTL is just an ordinary process that happens to have TGID not equal to PID and have a group leader not equal to itself (and shared VM etc. of course). Everything else happens in the same way as to an ordinary process. There is no separation of a shared status to some external structure like in FreeBSD. This creates some duplication of information and possible data inconsistency. The Linux(R) kernel seems to use task -> group information in some places and task information elsewhere and it is really not very consistent and looks error-prone. Every NPTL thread is created by a call to the `clone` syscall with a specific set of flags (more in the next subsection). The NPTL implements strict 1:1 threading. In FreeBSD we emulate NPTL threads with ordinary FreeBSD processes that share VM space, etc. and the PID gymnastic is just mimicked in the emulation specific structure attached to the process. The structure attached to the process looks like: [.programlisting] .... struct linux_emuldata { pid_t pid; int *child_set_tid; /* in clone(): Child.s TID to set on clone */ int *child_clear_tid;/* in clone(): Child.s TID to clear on exit */ struct linux_emuldata_shared *shared; int pdeath_signal; /* parent death signal */ LIST_ENTRY(linux_emuldata) threads; /* list of linux threads */ }; .... The PID is used to identify the FreeBSD process that attaches this structure. The `child_se_tid` and `child_clear_tid` are used for TID address copyout when a process exits and is created. The `shared` pointer points to a structure shared among threads. The `pdeath_signal` variable identifies the parent death signal and the `threads` pointer is used to link this structure to the list of threads. The `linux_emuldata_shared` structure looks like: [.programlisting] .... struct linux_emuldata_shared { int refs; pid_t group_pid; LIST_HEAD(, linux_emuldata) threads; /* head of list of linux threads */ }; .... The `refs` is a reference counter being used to determine when we can free the structure to avoid memory leaks. The `group_pid` is to identify PID ( = TGID) of the whole process ( = thread group). The `threads` pointer is the head of the list of threads in the process. The `linux_emuldata` structure can be obtained from the process using `em_find`. The prototype of the function is: [.programlisting] .... struct linux_emuldata *em_find(struct proc *, int locked); .... Here, `proc` is the process we want the emuldata structure from and the locked parameter determines whether we want to lock or not. The accepted values are `EMUL_DOLOCK` and `EMUL_DOUNLOCK`. More about locking later. [[pid-mangling]] ==== PID mangling As there is a difference in view as what to the idea of a process ID and thread ID is between FreeBSD and Linux(R) we have to translate the view somehow. We do it by PID mangling. This means that we fake what a PID (=TGID) and TID (=PID) is between kernel and userland. The rule of thumb is that in kernel (in Linuxulator) PID = PID and TGID = shared -> group pid and to userland we present `PID = shared -> group_pid` and `TID = proc -> p_pid`. The PID member of `linux_emuldata structure` is a FreeBSD PID. The above affects mainly getpid, getppid, gettid syscalls. Where we use PID/TGID respectively. In copyout of TIDs in `child_clear_tid` and `child_set_tid` we copy out FreeBSD PID. [[clone-syscall]] ==== Clone syscall The `clone` syscall is the way threads are created in Linux(R). The syscall prototype looks like this: [.programlisting] .... int linux_clone(l_int flags, void *stack, void *parent_tidptr, int dummy, void * child_tidptr); .... The `flags` parameter tells the syscall how exactly the processes should be cloned. As described above, Linux(R) can create processes sharing various things independently, for example two processes can share file descriptors but not VM, etc. Last byte of the `flags` parameter is the exit signal of the newly created process. The `stack` parameter if non-`NULL` tells, where the thread stack is and if it is `NULL` we are supposed to copy-on-write the calling process stack (i.e. do what normal man:fork[2] routine does). The `parent_tidptr` parameter is used as an address for copying out process PID (i.e. thread id) once the process is sufficiently instantiated but is not runnable yet. The `dummy` parameter is here because of the very strange calling convention of this syscall on i386. It uses the registers directly and does not let the compiler do it what results in the need of a dummy syscall. The `child_tidptr` parameter is used as an address for copying out PID once the process has finished forking and when the process exits. The syscall itself proceeds by setting corresponding flags depending on the flags passed in. For example, `CLONE_VM` maps to RFMEM (sharing of VM), etc. The only nit here is `CLONE_FS` and `CLONE_FILES` because FreeBSD does not allow setting this separately so we fake it by not setting RFFDG (copying of fd table and other fs information) if either of these is defined. This does not cause any problems, because those flags are always set together. After setting the flags the process is forked using the internal `fork1` routine, the process is instrumented not to be put on a run queue, i.e. not to be set runnable. After the forking is done we possibly reparent the newly created process to emulate `CLONE_PARENT` semantics. Next part is creating the emulation data. Threads in Linux(R) does not signal their parents so we set exit signal to be 0 to disable this. After that setting of `child_set_tid` and `child_clear_tid` is performed enabling the functionality later in the code. At this point we copy out the PID to the address specified by `parent_tidptr`. The setting of process stack is done by simply rewriting thread frame `%esp` register (`%rsp` on amd64). Next part is setting up TLS for the newly created process. After this man:vfork[2] semantics might be emulated and finally the newly created process is put on a run queue and copying out its PID to the parent process via `clone` return value is done. The `clone` syscall is able and in fact is used for emulating classic man:fork[2] and man:vfork[2] syscalls. Newer glibc in a case of 2.6 kernel uses `clone` to implement man:fork[2] and man:vfork[2] syscalls. [[locking]] ==== Locking The locking is implemented to be per-subsystem because we do not expect a lot of contention on these. There are two locks: `emul_lock` used to protect manipulating of `linux_emuldata` and `emul_shared_lock` used to manipulate `linux_emuldata_shared`. The `emul_lock` is a nonsleepable blocking mutex while `emul_shared_lock` is a sleepable blocking `sx_lock`. Due to of the per-subsystem locking we can coalesce some locks and that is why the em find offers the non-locking access. [[tls]] === TLS This section deals with TLS also known as thread local storage. [[trheading-intro]] ==== Introduction to threading Threads in computer science are entities within a process that can be scheduled independently from each other. The threads in the process share process wide data (file descriptors, etc.) but also have their own stack for their own data. Sometimes there is a need for process-wide data specific to a given thread. Imagine a name of the thread in execution or something like that. The traditional UNIX(R) threading API, pthreads provides a way to do it via man:pthread_key_create[3], man:pthread_setspecific[3] and man:pthread_getspecific[3] where a thread can create a key to the thread local data and using man:pthread_getspecific[3] or man:pthread_getspecific[3] to manipulate those data. You can easily see that this is not the most comfortable way this could be accomplished. So various producers of C/C++ compilers introduced a better way. They defined a new modifier keyword thread that specifies that a variable is thread specific. A new method of accessing such variables was developed as well (at least on i386). The pthreads method tends to be implemented in userspace as a trivial lookup table. The performance of such a solution is not very good. So the new method uses (on i386) segment registers to address a segment, where TLS area is stored so the actual accessing of a thread variable is just appending the segment register to the address thus addressing via it. The segment registers are usually `%gs` and `%fs` acting like segment selectors. Every thread has its own area where the thread local data are stored and the segment must be loaded on every context switch. This method is very fast and used almost exclusively in the whole i386 UNIX(R) world. Both FreeBSD and Linux(R) implement this approach and it yields very good results. The only drawback is the need to reload the segment on every context switch which can slowdown context switches. FreeBSD tries to avoid this overhead by using only 1 segment descriptor for this while Linux(R) uses 3. Interesting thing is that almost nothing uses more than 1 descriptor (only Wine seems to use 2) so Linux(R) pays this unnecessary price for context switches. [[i386-segs]] ==== Segments on i386 The i386 architecture implements the so called segments. A segment is a description of an area of memory. The base address (bottom) of the memory area, the end of it (ceiling), type, protection, etc. The memory described by a segment can be accessed using segment selector registers (`%cs`, `%ds`, `%ss`, `%es`, `%fs`, `%gs`). For example let us suppose we have a segment which base address is 0x1234 and length and this code: [.programlisting] .... mov %edx,%gs:0x10 .... This will load the content of the `%edx` register into memory location 0x1244. Some segment registers have a special use, for example `%cs` is used for code segment and `%ss` is used for stack segment but `%fs` and `%gs` are generally unused. Segments are either stored in a global GDT table or in a local LDT table. LDT is accessed via an entry in the GDT. The LDT can store more types of segments. LDT can be per process. Both tables define up to 8191 entries. [[linux-i386]] ==== Implementation on Linux(R) i386 There are two main ways of setting up TLS in Linux(R). It can be set when cloning a process using the `clone` syscall or it can call `set_thread_area`. When a process passes `CLONE_SETTLS` flag to `clone`, the kernel expects the memory pointed to by the `%esi` register a Linux(R) user space representation of a segment, which gets translated to the machine representation of a segment and loaded into a GDT slot. The GDT slot can be specified with a number or -1 can be used meaning that the system itself should choose the first free slot. In practice, the vast majority of programs use only one TLS entry and does not care about the number of the entry. We exploit this in the emulation and in fact depend on it. [[tls-emu]] ==== Emulation of Linux(R) TLS [[tls-i386]] ===== i386 Loading of TLS for the current thread happens by calling `set_thread_area` while loading TLS for a second process in `clone` is done in the separate block in `clone`. Those two functions are very similar. The only difference being the actual loading of the GDT segment, which happens on the next context switch for the newly created process while `set_thread_area` must load this directly. The code basically does this. It copies the Linux(R) form segment descriptor from the userland. The code checks for the number of the descriptor but because this differs between FreeBSD and Linux(R) we fake it a little. We only support indexes of 6, 3 and -1. The 6 is genuine Linux(R) number, 3 is genuine FreeBSD one and -1 means autoselection. Then we set the descriptor number to constant 3 and copy out this to the userspace. We rely on the userspace process using the number from the descriptor but this works most of the time (have never seen a case where this did not work) as the userspace process typically passes in 1. Then we convert the descriptor from the Linux(R) form to a machine dependant form (i.e. operating system independent form) and copy this to the FreeBSD defined segment descriptor. Finally we can load it. We assign the descriptor to threads PCB (process control block) and load the `%gs` segment using `load_gs`. This loading must be done in a critical section so that nothing can interrupt us. The `CLONE_SETTLS` case works exactly like this just the loading using `load_gs` is not performed. The segment used for this (segment number 3) is shared for this use between FreeBSD processes and Linux(R) processes so the Linux(R) emulation layer does not add any overhead over plain FreeBSD. [[tls-amd64]] ===== amd64 The amd64 implementation is similar to the i386 one but there was initially no 32bit segment descriptor used for this purpose (hence not even native 32bit TLS users worked) so we had to add such a segment and implement its loading on every context switch (when a flag signaling use of 32bit is set). Apart from this the TLS loading is exactly the same just the segment numbers are different and the descriptor format and the loading differs slightly. [[futexes]] === Futexes [[sync-intro]] ==== Introduction to synchronization Threads need some kind of synchronization and POSIX(R) provides some of them: mutexes for mutual exclusion, read-write locks for mutual exclusion with biased ratio of reads and writes and condition variables for signaling a status change. It is interesting to note that POSIX(R) threading API lacks support for semaphores. Those synchronization routines implementations are heavily dependant on the type threading support we have. In pure 1:M (userspace) model the implementation can be solely done in userspace and thus be very fast (the condition variables will probably end up being implemented using signals, i.e. not fast) and simple. In 1:1 model, the situation is also quite clear - the threads must be synchronized using kernel facilities (which is very slow because a syscall must be performed). The mixed M:N scenario just combines the first and second approach or rely solely on kernel. Threads synchronization is a vital part of thread-enabled programming and its performance can affect resulting program a lot. Recent benchmarks on FreeBSD operating system showed that an improved sx_lock implementation yielded 40% speedup in _ZFS_ (a heavy sx user), this is in-kernel stuff but it shows clearly how important the performance of synchronization primitives is. Threaded programs should be written with as little contention on locks as possible. Otherwise, instead of doing useful work the thread just waits on a lock. As a result of this, the most well written threaded programs show little locks contention. [[futex-intro]] ==== Futexes introduction Linux(R) implements 1:1 threading, i.e. it has to use in-kernel synchronization primitives. As stated earlier, well written threaded programs have little lock contention. So a typical sequence could be performed as two atomic increase/decrease mutex reference counter, which is very fast, as presented by the following example: [.programlisting] .... pthread_mutex_lock(&mutex); ... pthread_mutex_unlock(&mutex); .... 1:1 threading forces us to perform two syscalls for those mutex calls, which is very slow. The solution Linux(R) 2.6 implements is called futexes. Futexes implement the check for contention in userspace and call kernel primitives only in a case of contention. Thus the typical case takes place without any kernel intervention. This yields reasonably fast and flexible synchronization primitives implementation. [[futex-api]] ==== Futex API The futex syscall looks like this: [.programlisting] .... int futex(void *uaddr, int op, int val, struct timespec *timeout, void *uaddr2, int val3); .... In this example `uaddr` is an address of the mutex in userspace, `op` is an operation we are about to perform and the other parameters have per-operation meaning. Futexes implement the following operations: * `FUTEX_WAIT` * `FUTEX_WAKE` * `FUTEX_FD` * `FUTEX_REQUEUE` * `FUTEX_CMP_REQUEUE` * `FUTEX_WAKE_OP` [[futex-wait]] ===== FUTEX_WAIT This operation verifies that on address `uaddr` the value `val` is written. If not, `EWOULDBLOCK` is returned, otherwise the thread is queued on the futex and gets suspended. If the argument `timeout` is non-zero it specifies the maximum time for the sleeping, otherwise the sleeping is infinite. [[futex-wake]] ===== FUTEX_WAKE This operation takes a futex at `uaddr` and wakes up `val` first futexes queued on this futex. [[futex-fd]] ===== FUTEX_FD This operations associates a file descriptor with a given futex. [[futex-requeue]] ===== FUTEX_REQUEUE This operation takes `val` threads queued on futex at `uaddr`, wakes them up, and takes `val2` next threads and requeues them on futex at `uaddr2`. [[futex-cmp-requeue]] ===== FUTEX_CMP_REQUEUE This operation does the same as `FUTEX_REQUEUE` but it checks that `val3` equals to `val` first. [[futex-wake-op]] ===== FUTEX_WAKE_OP This operation performs an atomic operation on `val3` (which contains coded some other value) and `uaddr`. Then it wakes up `val` threads on futex at `uaddr` and if the atomic operation returned a positive number it wakes up `val2` threads on futex at `uaddr2`. The operations implemented in `FUTEX_WAKE_OP`: * `FUTEX_OP_SET` * `FUTEX_OP_ADD` * `FUTEX_OP_OR` * `FUTEX_OP_AND` * `FUTEX_OP_XOR` [NOTE] ==== There is no `val2` parameter in the futex prototype. The `val2` is taken from the `struct timespec *timeout` parameter for operations `FUTEX_REQUEUE`, `FUTEX_CMP_REQUEUE` and `FUTEX_WAKE_OP`. ==== [[futex-emu]] ==== Futex emulation in FreeBSD The futex emulation in FreeBSD is taken from NetBSD and further extended by us. It is placed in `linux_futex.c` and [.filename]#linux_futex.h# files. The `futex` structure looks like: [.programlisting] .... struct futex { void *f_uaddr; int f_refcount; LIST_ENTRY(futex) f_list; TAILQ_HEAD(lf_waiting_paroc, waiting_proc) f_waiting_proc; }; .... And the structure `waiting_proc` is: [.programlisting] .... struct waiting_proc { struct thread *wp_t; struct futex *wp_new_futex; TAILQ_ENTRY(waiting_proc) wp_list; }; .... [[futex-get]] ===== futex_get / futex_put A futex is obtained using the `futex_get` function, which searches a linear list of futexes and returns the found one or creates a new futex. When releasing a futex from the use we call the `futex_put` function, which decreases a reference counter of the futex and if the refcount reaches zero it is released. [[futex-sleep]] ===== futex_sleep When a futex queues a thread for sleeping it creates a `working_proc` structure and puts this structure to the list inside the futex structure then it just performs a man:tsleep[9] to suspend the thread. The sleep can be timed out. After man:tsleep[9] returns (the thread was woken up or it timed out) the `working_proc` structure is removed from the list and is destroyed. All this is done in the `futex_sleep` function. If we got woken up from `futex_wake` we have `wp_new_futex` set so we sleep on it. This way the actual requeueing is done in this function. [[futex-wake-2]] ===== futex_wake Waking up a thread sleeping on a futex is performed in the `futex_wake` function. First in this function we mimic the strange Linux(R) behavior, where it wakes up N threads for all operations, the only exception is that the REQUEUE operations are performed on N+1 threads. But this usually does not make any difference as we are waking up all threads. Next in the function in the loop we wake up n threads, after this we check if there is a new futex for requeueing. If so, we requeue up to n2 threads on the new futex. This cooperates with `futex_sleep`. [[futex-wake-op-2]] ===== futex_wake_op The `FUTEX_WAKE_OP` operation is quite complicated. First we obtain two futexes at addresses `uaddr` and `uaddr2` then we perform the atomic operation using `val3` and `uaddr2`. Then `val` waiters on the first futex is woken up and if the atomic operation condition holds we wake up `val2` (i.e. `timeout`) waiter on the second futex. [[futex-atomic-op]] ===== futex atomic operation The atomic operation takes two parameters `encoded_op` and `uaddr`. The encoded operation encodes the operation itself, comparing value, operation argument, and comparing argument. The pseudocode for the operation is like this one: [.programlisting] .... oldval = *uaddr2 *uaddr2 = oldval OP oparg .... And this is done atomically. First a copying in of the number at `uaddr` is performed and the operation is done. The code handles page faults and if no page fault occurs `oldval` is compared to `cmparg` argument with cmp comparator. [[futex-locking]] ===== Futex locking Futex implementation uses two lock lists protecting `sx_lock` and global locks (either Giant or another `sx_lock`). Every operation is performed locked from the start to the very end. [[syscall-impl]] === Various syscalls implementation In this section I am going to describe some smaller syscalls that are worth mentioning because their implementation is not obvious or those syscalls are interesting from other point of view. [[syscall-at]] ==== *at family of syscalls During development of Linux(R) 2.6.16 kernel, the *at syscalls were added. Those syscalls (`openat` for example) work exactly like their at-less counterparts with the slight exception of the `dirfd` parameter. This parameter changes where the given file, on which the syscall is to be performed, is. When the `filename` parameter is absolute `dirfd` is ignored but when the path to the file is relative, it comes to the play. The `dirfd` parameter is a directory relative to which the relative pathname is checked. The `dirfd` parameter is a file descriptor of some directory or `AT_FDCWD`. So for example the `openat` syscall can be like this: [.programlisting] .... file descriptor 123 = /tmp/foo/, current working directory = /tmp/ openat(123, /tmp/bah\, flags, mode) /* opens /tmp/bah */ openat(123, bah\, flags, mode) /* opens /tmp/foo/bah */ openat(AT_FDWCWD, bah\, flags, mode) /* opens /tmp/bah */ openat(stdio, bah\, flags, mode) /* returns error because stdio is not a directory */ .... This infrastructure is necessary to avoid races when opening files outside the working directory. Imagine that a process consists of two threads, thread A and thread B. Thread A issues `open(./tmp/foo/bah., flags, mode)` and before returning it gets preempted and thread B runs. Thread B does not care about the needs of thread A and renames or removes [.filename]#/tmp/foo/#. We got a race. To avoid this we can open [.filename]#/tmp/foo# and use it as `dirfd` for `openat` syscall. This also enables user to implement per-thread working directories. Linux(R) family of *at syscalls contains: `linux_openat`, `linux_mkdirat`, `linux_mknodat`, `linux_fchownat`, `linux_futimesat`, `linux_fstatat64`, `linux_unlinkat`, `linux_renameat`, `linux_linkat`, `linux_symlinkat`, `linux_readlinkat`, `linux_fchmodat` and `linux_faccessat`. All these are implemented using the modified man:namei[9] routine and simple wrapping layer. [[implementation]] ===== Implementation The implementation is done by altering the man:namei[9] routine (described above) to take additional parameter `dirfd` in its `nameidata` structure, which specifies the starting point of the pathname lookup instead of using the current working directory every time. The resolution of `dirfd` from file descriptor number to a vnode is done in native *at syscalls. When `dirfd` is `AT_FDCWD` the `dvp` entry in `nameidata` structure is `NULL` but when `dirfd` is a different number we obtain a file for this file descriptor, check whether this file is valid and if there is vnode attached to it then we get a vnode. Then we check this vnode for being a directory. In the actual man:namei[9] routine we simply substitute the `dvp` vnode for `dp` variable in the man:namei[9] function, which determines the starting point. The man:namei[9] is not used directly but via a trace of different functions on various levels. For example the `openat` goes like this: [.programlisting] .... openat() --> kern_openat() --> vn_open() -> namei() .... For this reason `kern_open` and `vn_open` must be altered to incorporate the additional `dirfd` parameter. No compat layer is created for those because there are not many users of this and the users can be easily converted. This general implementation enables FreeBSD to implement their own *at syscalls. This is being discussed right now. [[ioctl]] ==== Ioctl The ioctl interface is quite fragile due to its generality. We have to bear in mind that devices differ between Linux(R) and FreeBSD so some care must be applied to do ioctl emulation work right. The ioctl handling is implemented in [.filename]#linux_ioctl.c#, where `linux_ioctl` function is defined. This function simply iterates over sets of ioctl handlers to find a handler that implements a given command. The ioctl syscall has three parameters, the file descriptor, command and an argument. The command is a 16-bit number, which in theory is divided into high 8 bits determining class of the ioctl command and low 8 bits, which are the actual command within the given set. The emulation takes advantage of this division. We implement handlers for each set, like `sound_handler` or `disk_handler`. Each handler has a maximum command and a minimum command defined, which is used for determining what handler is used. There are slight problems with this approach because Linux(R) does not use the set division consistently so sometimes ioctls for a different set are inside a set they should not belong to (SCSI generic ioctls inside cdrom set, etc.). FreeBSD currently does not implement many Linux(R) ioctls (compared to NetBSD, for example) but the plan is to port those from NetBSD. The trend is to use Linux(R) ioctls even in the native FreeBSD drivers because of the easy porting of applications. [[debugging]] ==== Debugging Every syscall should be debuggable. For this purpose we introduce a small infrastructure. We have the ldebug facility, which tells whether a given syscall should be debugged (settable via a sysctl). For printing we have LMSG and ARGS macros. Those are used for altering a printable string for uniform debugging messages. [[conclusion]] == Conclusion [[results]] === Results As of April 2007 the Linux(R) emulation layer is capable of emulating the Linux(R) 2.6.16 kernel quite well. The remaining problems concern futexes, unfinished *at family of syscalls, problematic signals delivery, missing `epoll` and `inotify` and probably some bugs we have not discovered yet. Despite this we are capable of running basically all the Linux(R) programs included in FreeBSD Ports Collection with Fedora Core 4 at 2.6.16 and there are some rudimentary reports of success with Fedora Core 6 at 2.6.16. The Fedora Core 6 linux_base was recently committed enabling some further testing of the emulation layer and giving us some more hints where we should put our effort in implementing missing stuff. We are able to run the most used applications like package:www/linux-firefox[], package:net-im/skype[] and some games from the Ports Collection. Some of the programs exhibit bad behavior under 2.6 emulation but this is currently under investigation and hopefully will be fixed soon. The only big application that is known not to work is the Linux(R) Java(TM) Development Kit and this is because of the requirement of `epoll` facility which is not directly related to the Linux(R) kernel 2.6. We hope to enable 2.6.16 emulation by default some time after FreeBSD 7.0 is released at least to expose the 2.6 emulation parts for some wider testing. Once this is done we can switch to Fedora Core 6 linux_base, which is the ultimate plan. [[future-work]] === Future work Future work should focus on fixing the remaining issues with futexes, implement the rest of the *at family of syscalls, fix the signal delivery and possibly implement the `epoll` and `inotify` facilities. We hope to be able to run the most important programs flawlessly soon, so we will be able to switch to the 2.6 emulation by default and make the Fedora Core 6 the default linux_base because our currently used Fedora Core 4 is not supported any more. The other possible goal is to share our code with NetBSD and DragonflyBSD. NetBSD has some support for 2.6 emulation but its far from finished and not really tested. DragonflyBSD has expressed some interest in porting the 2.6 improvements. Generally, as Linux(R) develops we would like to keep up with their development, implementing newly added syscalls. Splice comes to mind first. Some already implemented syscalls are also heavily crippled, for example `mremap` and others. Some performance improvements can also be made, finer grained locking and others. [[team]] === Team I cooperated on this project with (in alphabetical order): * `{jhb}` * `{kib}` * Emmanuel Dreyfus * Scot Hetzel * `{jkim}` * `{netchild}` * `{ssouhlal}` * Li Xiao * `{davidxu}` I would like to thank all those people for their advice, code reviews and general support. [[literatures]] == Literatures . Marshall Kirk McKusick - George V. Nevile-Neil. Design and Implementation of the FreeBSD operating system. Addison-Wesley, 2005. . https://tldp.org[https://tldp.org] . https://www.kernel.org[https://www.kernel.org] diff --git a/documentation/content/en/articles/linux-users/_index.adoc b/documentation/content/en/articles/linux-users/_index.adoc index 7bfa62206f..aa2e76682a 100644 --- a/documentation/content/en/articles/linux-users/_index.adoc +++ b/documentation/content/en/articles/linux-users/_index.adoc @@ -1,358 +1,368 @@ --- title: FreeBSD Quickstart Guide for Linux® Users authors: - author: John Ferrell copyright: 2008 The FreeBSD Documentation Project description: This document is intended to quickly familiarize intermediate to advanced Linux® users with the basics of FreeBSD. trademarks: ["freebsd", "intel", "redhat", "linux", "unix", "general"] tags: ["Quickstart", "guide", "Linux", "FreeBSD"] --- = FreeBSD Quickstart Guide for Linux(R) Users :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This document is intended to quickly familiarize intermediate to advanced Linux(R) users with the basics of FreeBSD. ''' toc::[] [[intro]] == Introduction This document highlights some of the technical differences between FreeBSD and Linux(R) so that intermediate to advanced Linux(R) users can quickly familiarize themselves with the basics of FreeBSD. This document assumes that FreeBSD is already installed. Refer to the link:{handbook}#bsdinstall[Installing FreeBSD] chapter of the FreeBSD Handbook for help with the installation process. [[shells]] == Default Shell Linux(R) users are often surprised to find that Bash is not the default shell in FreeBSD. In fact, Bash is not included in the default installation. Instead, FreeBSD uses man:tcsh[1] as the default root shell, and the Bourne shell-compatible man:sh[1] as the default user shell. man:sh[1] is very similar to Bash but with a much smaller feature-set. Generally shell scripts written for man:sh[1] will run in Bash, but the reverse is not always true. However, Bash and other shells are available for installation using the FreeBSD link:{handbook}#ports[Packages and Ports Collection]. After installing another shell, use man:chsh[1] to change a user's default shell. It is recommended that the `root` user's default shell remain unchanged since shells which are not included in the base distribution are installed to [.filename]#/usr/local/bin#. In the event of a problem, the file system where [.filename]#/usr/local/bin# is located may not be mounted. In this case, `root` would not have access to its default shell, preventing `root` from logging in and fixing the problem. [[software]] == Packages and Ports: Adding Software in FreeBSD FreeBSD provides two methods for installing applications: binary packages and compiled ports. Each method has its own benefits: .Binary Packages * Faster installation as compared to compiling large applications. * Does not require an understanding of how to compile software. * No need to install a compiler. .Ports * Ability to customize installation options. * Custom patches can be applied. If an application installation does not require any customization, installing the package is sufficient. Compile the port instead whenever an application requires customization of the default options. If needed, a custom package can be compiled from ports using `make package`. A complete list of all available ports and packages can be found https://www.freebsd.org/ports/[here]. [[packages]] === Packages Packages are pre-compiled applications, the FreeBSD equivalents of [.filename]#.deb# files on Debian/Ubuntu based systems and [.filename]#.rpm# files on Red Hat/Fedora based systems. Packages are installed using `pkg`. For example, the following command installs Apache 2.4: [source,shell] .... # pkg install apache24 .... For more information on packages refer to section 5.4 of the FreeBSD Handbook: link:{handbook}#pkgng-intro[Using pkgng for Binary Package Management]. [[ports]] === Ports The FreeBSD Ports Collection is a framework of [.filename]#Makefiles# and patches specifically customized for installing applications from source on FreeBSD. When installing a port, the system will fetch the source code, apply any required patches, compile the code, and install the application and any required dependencies. The Ports Collection, sometimes referred to as the ports tree, can be installed to [.filename]#/usr/ports# using man:portsnap[8]. Detailed instructions for installing the Ports Collection can be found in link:{handbook}#ports-using[section 5.5] of the FreeBSD Handbook. To compile a port, change to the port's directory and start the build process. The following example installs Apache 2.4 from the Ports Collection: [source,shell] .... # cd /usr/ports/www/apache24 # make install clean .... A benefit of using ports to install software is the ability to customize the installation options. This example specifies that the mod_ldap module should also be installed: [source,shell] .... # cd /usr/ports/www/apache24 # make WITH_LDAP="YES" install clean .... Refer to link:{handbook}#ports-using[Using the Ports Collection] for more information. [[startup]] == System Startup Many Linux(R) distributions use the SysV init system, whereas FreeBSD uses the traditional BSD-style man:init[8]. Under the BSD-style man:init[8], there are no run-levels and [.filename]#/etc/inittab# does not exist. Instead, startup is controlled by man:rc[8] scripts. At system boot, [.filename]#/etc/rc# reads [.filename]#/etc/rc.conf# and [.filename]#/etc/defaults/rc.conf# to determine which services are to be started. The specified services are then started by running the corresponding service initialization scripts located in [.filename]#/etc/rc.d/# and [.filename]#/usr/local/etc/rc.d/#. These scripts are similar to the scripts located in [.filename]#/etc/init.d/# on Linux(R) systems. The scripts found in [.filename]#/etc/rc.d/# are for applications that are part of the "base" system, such as man:cron[8], man:sshd[8], and man:syslog[3]. The scripts in [.filename]#/usr/local/etc/rc.d/# are for user-installed applications such as Apache and Squid. Since FreeBSD is developed as a complete operating system, user-installed applications are not considered to be part of the "base" system. User-installed applications are generally installed using link:{handbook}#ports-using[Packages or Ports]. In order to keep them separate from the base system, user-installed applications are installed under [.filename]#/usr/local/#. Therefore, user-installed binaries reside in [.filename]#/usr/local/bin/#, configuration files are in [.filename]#/usr/local/etc/#, and so on. Services are enabled by adding an entry for the service in [.filename]#/etc/rc.conf#. The system defaults are found in [.filename]#/etc/defaults/rc.conf# and these default settings are overridden by settings in [.filename]#/etc/rc.conf#. Refer to man:rc.conf[5] for more information about the available entries. When installing additional applications, review the application's install message to determine how to enable any associated services. The following entries in [.filename]#/etc/rc.conf# enable man:sshd[8], enable Apache 2.4, and specify that Apache should be started with SSL. [.programlisting] .... # enable SSHD sshd_enable="YES" # enable Apache with SSL apache24_enable="YES" apache24_flags="-DSSL" .... Once a service has been enabled in [.filename]#/etc/rc.conf#, it can be started without rebooting the system: [source,shell] .... # service sshd start # service apache24 start .... If a service has not been enabled, it can be started from the command line using `onestart`: [source,shell] .... # service sshd onestart .... [[network]] == Network Configuration Instead of a generic _ethX_ identifier that Linux(R) uses to identify a network interface, FreeBSD uses the driver name followed by a number. The following output from man:ifconfig[8] shows two Intel(R) Pro 1000 network interfaces ([.filename]#em0# and [.filename]#em1#): [source,shell] .... % ifconfig em0: flags=8843 mtu 1500 options=b inet 10.10.10.100 netmask 0xffffff00 broadcast 10.10.10.255 ether 00:50:56:a7:70:b2 media: Ethernet autoselect (1000baseTX ) status: active em1: flags=8843 mtu 1500 options=b inet 192.168.10.222 netmask 0xffffff00 broadcast 192.168.10.255 ether 00:50:56:a7:03:2b media: Ethernet autoselect (1000baseTX ) status: active .... An IP address can be assigned to an interface using man:ifconfig[8]. To remain persistent across reboots, the IP configuration must be included in [.filename]#/etc/rc.conf#. The following [.filename]#/etc/rc.conf# entries specify the hostname, IP address, and default gateway: [.programlisting] .... hostname="server1.example.com" ifconfig_em0="inet 10.10.10.100 netmask 255.255.255.0" defaultrouter="10.10.10.1" .... Use the following entries to instead configure an interface for DHCP: [.programlisting] .... hostname="server1.example.com" ifconfig_em0="DHCP" .... [[firewall]] == Firewall FreeBSD does not use Linux(R) IPTABLES for its firewall. Instead, FreeBSD offers a choice of three kernel level firewalls: * link:{handbook}#firewalls-pf[PF] * link:{handbook}#firewalls-ipf[IPFILTER] * link:{handbook}#firewalls-ipfw[IPFW] PF is developed by the OpenBSD project and ported to FreeBSD. PF was created as a replacement for IPFILTER and its syntax is similar to that of IPFILTER. PF can be paired with man:altq[4] to provide QoS features. This sample PF entry allows inbound SSH: [.programlisting] .... pass in on $ext_if inet proto tcp from any to ($ext_if) port 22 .... IPFILTER is the firewall application developed by Darren Reed. It is not specific to FreeBSD and has been ported to several operating systems including NetBSD, OpenBSD, SunOS, HP/UX, and Solaris. The IPFILTER syntax to allow inbound SSH is: [.programlisting] .... pass in on $ext_if proto tcp from any to any port = 22 .... IPFW is the firewall developed and maintained by FreeBSD. It can be paired with man:dummynet[4] to provide traffic shaping capabilities and simulate different types of network connections. The IPFW syntax to allow inbound SSH would be: [.programlisting] .... ipfw add allow tcp from any to me 22 in via $ext_if .... [[updates]] == Updating FreeBSD There are two methods for updating a FreeBSD system: from source or binary updates. Updating from source is the most involved update method, but offers the greatest amount of flexibility. The process involves synchronizing a local copy of the FreeBSD source code with the FreeBSD Subversion servers. Once the local source code is up-to-date, a new version of the kernel and userland can be compiled. Binary updates are similar to using `yum` or `apt-get` to update a Linux(R) system. In FreeBSD, man:freebsd-update[8] can be used fetch new binary updates and install them. These updates can be scheduled using man:cron[8]. [NOTE] ==== When using man:cron[8] to schedule updates, use `freebsd-update cron` in the man:crontab[1] to reduce the possibility of a large number of machines all pulling updates at the same time: [.programlisting] .... 0 3 * * * root /usr/sbin/freebsd-update cron .... ==== For more information on source and binary updates, refer to link:{handbook}#updating-upgrading[the chapter on updating] in the FreeBSD Handbook. [[procfs]] == procfs: Gone But Not Forgotten In some Linux(R) distributions, one could look at [.filename]#/proc/sys/net/ipv4/ip_forward# to determine if IP forwarding is enabled. In FreeBSD, man:sysctl[8] is instead used to view this and other system settings. For example, use the following to determine if IP forwarding is enabled on a FreeBSD system: [source,shell] .... % sysctl net.inet.ip.forwarding net.inet.ip.forwarding: 0 .... Use `-a` to list all the system settings: [source,shell] .... % sysctl -a | more .... If an application requires procfs, add the following entry to [.filename]#/etc/fstab#: [source,shell] .... proc /proc procfs rw,noauto 0 0 .... Including `noauto` will prevent [.filename]#/proc# from being automatically mounted at boot. To mount the file system without rebooting: [source,shell] .... # mount /proc .... [[commands]] == Common Commands Some common command equivalents are as follows: [.informaltable] [cols="1,1,1", frame="none", options="header"] |=== | Linux(R) command (Red Hat/Debian) | FreeBSD equivalent | Purpose |`yum install _package_` / `apt-get install _package_` |`pkg install _package_` |Install package from remote repository |`rpm -ivh _package_` / `dpkg -i _package_` |`pkg add _package_` |Install local package |`rpm -qa` / `dpkg -l` |`pkg info` |List installed packages |`lspci` |`pciconf` |List PCI devices |`lsmod` |`kldstat` |List loaded kernel modules |`modprobe` |`kldload` / `kldunload` |Load/Unload kernel modules |`strace` |`truss` |Trace system calls |=== [[conclusion]] == Conclusion This document has provided an overview of FreeBSD. Refer to the link:{handbook}[FreeBSD Handbook] for more in-depth coverage of these topics as well as the many topics not covered by this document. diff --git a/documentation/content/en/articles/mailing-list-faq/_index.adoc b/documentation/content/en/articles/mailing-list-faq/_index.adoc index 23f86fcd23..5f3322f29f 100644 --- a/documentation/content/en/articles/mailing-list-faq/_index.adoc +++ b/documentation/content/en/articles/mailing-list-faq/_index.adoc @@ -1,192 +1,206 @@ --- title: Frequently Asked Questions About The FreeBSD Mailing Lists authors: - author: The FreeBSD Documentation Project copyright: 2004-2021 The FreeBSD Documentation Project description: How to best use the mailing lists, such as how to help avoid frequently-repeated discussions tags: ["FAQ", "Mailing Lists", "FreeBSD"] --- = Frequently Asked Questions About The FreeBSD Mailing Lists :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This is the FAQ for the FreeBSD mailing lists. If you are interested in helping with this project, send email to the {freebsd-doc}. The latest version of this document is always available from the link:.[FreeBSD World Wide Web server]. It may also be downloaded as one large link:.[HTML] file with HTTP or as plain text, PostScript, PDF, etc. from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server]. You may also want to link:https://www.FreeBSD.org/search/[Search the FAQ]. ''' toc::[] [[introduction]] == Introduction As is usual with FAQs, this document aims to cover the most frequently asked questions concerning the FreeBSD mailing lists (and of course answer them!). Although originally intended to reduce bandwidth and avoid the same old questions being asked over and over again, FAQs have become recognized as valuable information resources. This document attempts to represent a community consensus, and as such it can never really be __authoritative__. However, if you find technical errors within this document, or have suggestions about items that should be added, please either submit a PR, or email the {freebsd-doc}. Thanks. === What is the purpose of the FreeBSD mailing lists? The FreeBSD mailing lists serve as the primary communication channels for the FreeBSD community, covering many different topic areas and communities of interest. === Who is the audience for the FreeBSD mailing lists? This depends on charter of each individual list. Some lists are more oriented to developers; some are more oriented towards the FreeBSD community as a whole. Please see http://lists.FreeBSD.org/mailman/listinfo[this list] for the current summary. === Are the FreeBSD mailing lists open for anyone to participate? Again, this depends on charter of each individual list. Please read the charter of a mailing list before you post to it, and respect it when you post. This will help everyone to have a better experience with the lists. If after reading the above lists, you still do not know which mailing list to post a question to, you will probably want to post to freebsd-questions (but see below, first). Also note that the mailing lists have traditionally been open to postings from non-subscribers. This has been a deliberate choice, to help make joining the FreeBSD community an easier process, and to encourage open sharing of ideas. However, due to past abuse by some individuals, certain lists now have a policy where postings from non-subscribers must be manually screened to ensure that they are appropriate. === How can I subscribe? You can use http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface] to subscribe to any of the public lists. === How can I unsubscribe? You can use the same interface as above; or, you can follow the instructions that are at the bottom of every mailing list message that is sent. Please do not send unsubscribe messages directly to the public lists themselves. First, this will not accomplish your goal, and second, it will irritate the existing subscribers, and you will probably get flamed. This is a classical mistake when using mailing lists; please try to avoid it. === Are archives available? Yes. Threaded archives are available http://docs.FreeBSD.org/mail/[here]. === Are mailing lists available in a digest format? Yes. See http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface]. [[etiquette]] == Mailing List Etiquette Participation in the mailing lists, like participation in any community, requires a common basis for communication. Please make only appropriate postings, and follow common rules of etiquette. === What should I do before I post? You have already taken the most important step by reading this document. However, if you are new to FreeBSD, you may first need to familiarize yourself with the software, and all the social history around it, by reading the numerous link:https://www.FreeBSD.org/docs/books/[books and articles] that are available. Items of particular interest include the link:{faq}[FreeBSD Frequently Asked Questions (FAQ)] document, the link:{handbook}[FreeBSD Handbook], and the articles link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list], link:{explaining-bsd}[Explaining BSD], and link:{new-users}[FreeBSD First Steps]. It is always considered bad form to ask a question that is already answered in the above documents. This is not because the volunteers who work on this project are particularly mean people, but after a certain number of times answering the same questions over and over again, frustration begins to set in. This is particularly true if there is an existing answer to the question that is already available. Always keep in mind that almost all of the work done on FreeBSD is done by volunteers, and that we are only human. === What constitutes an inappropriate posting? * Postings must be in accordance with the charter of the mailing list. * Personal attacks are discouraged. As good net-citizens, we should try to hold ourselves to high standards of behavior. * Spam is not allowed, ever. The mailing lists are actively processed to ban offenders to this rule. === What is considered proper etiquette when posting to the mailing lists? * Please wrap lines at 75 characters, since not everyone uses fancy GUI mail reading programs. * Please respect the fact that bandwidth is not infinite. Not everyone reads email through high-speed connections, so if your posting involves something like the content of [.filename]#config.log# or an extensive stack trace, please consider putting that information up on a website somewhere and just provide a URL to it. Remember, too, that these postings will be archived indefinitely, so huge postings will simply inflate the size of the archives long after their purpose has expired. * Format your message so that it is legible, and PLEASE DO NOT SHOUT!!!!!. Do not underestimate the effect that a poorly formatted mail message has, and not just on the FreeBSD mailing lists. Your mail message is all that people see of you, and if it is poorly formatted, badly spelled, full of errors, and/or has lots of exclamation points, it will give people a poor impression of you. * Please use an appropriate human language for a particular mailing list. Many non-English mailing lists are link:https://www.FreeBSD.org/community/mailinglists/[available]. + For the ones that are not, we do appreciate that many people do not speak English as their first language, and we try to make allowances for that. It is considered particularly poor form to criticize non-native speakers for spelling or grammatical errors. FreeBSD has an excellent track record in this regard; please, help us to uphold that tradition. * Please use a standards-compliant Mail User Agent (MUA). A lot of badly formatted messages come from http://www.lemis.com/grog/email/email.php[bad mailers or badly configured mailers]. The following mailers are known to send out badly formatted messages without you finding out about them: ** exmh ** Microsoft(R) Exchange ** Microsoft(R) Outlook(R) + Try not to use MIME: a lot of people use mailers which do not get on very well with MIME. * Make sure your time and time zone are set correctly. This may seem a little silly, since your message still gets there, but many of the people on these mailing lists get several hundred messages a day. They frequently sort the incoming messages by subject and by date, and if your message does not come before the first answer, they may assume that they missed it and not bother to look. * A lot of the information you need to supply is the output of programs, such as man:dmesg[8], or console messages, which usually appear in [.filename]#/var/log/messages#. Do not try to copy this information by typing it in again; not only it is a real pain, but you are bound to make a mistake. To send log file contents, either make a copy of the file and use an editor to trim the information to what is relevant, or cut and paste into your message. For the output of programs like `dmesg`, redirect the output to a file and include that. For example, + [source,shell] .... % dmesg > /tmp/dmesg.out .... + This redirects the information to the file [.filename]#/tmp/dmesg.out#. * When using cut-and-paste, please be aware that some such operations badly mangle their messages. This is of particular concern when posting contents of [.filename]#Makefiles#, where `tab` is a significant character. This is a very common, and very annoying, problem with submissions to the link:https://www.FreeBSD.org/support/[Problem Reports database]. [.filename]#Makefiles# with tabs changed to either spaces, or the annoying `=3B` escape sequence, create a great deal of aggravation for committers. === What are the special etiquette consideration when replying to an existing posting on the mailing lists? * Please include relevant text from the original message. Trim it to the minimum, but do not overdo it. It should still be possible for somebody who did not read the original message to understand what you are talking about. + This is especially important for postings of the type "yes, I see this too", where the initial posting was dozens or hundreds of lines. * Use some technique to identify which text came from the original message, and which text you add. A common convention is to prepend "`>`" to the original message. Leaving white space after the "`>`" and leaving empty lines between your text and the original text both make the result more readable. * Please ensure that the attributions of the text you are quoting is correct. People can become offended if you attribute words to them that they themselves did not write. * Please do not `top post`. By this, we mean that if you are replying to a message, please put your replies after the text that you copy in your reply. + ** A: Because it reverses the logical flow of conversation. ** Q: Why is top posting frowned upon? + (Thanks to Randy Bush for the joke.) [[recurring]] == Recurring Topics On The Mailing Lists Participation in the mailing lists, like participation in any community, requires a common basis for communication. Many of the mailing lists presuppose a knowledge of the Project's history. In particular, there are certain topics that seem to regularly occur to newcomers to the community. It is the responsibility of each poster to ensure that their postings do not fall into one of these categories. By doing so, you will help the mailing lists to stay on-topic, and probably save yourself being flamed in the process. The best method to avoid this is to familiarize yourself with the http://docs.FreeBSD.org/mail/[mailing list archives], to help yourself understand the background of what has gone before. In this, the https://www.FreeBSD.org/search/#mailinglists[mailing list search interface] is invaluable. (If that method does not yield useful results, please supplement it with a search with your favorite major search engine). By familiarizing yourself with the archives, not only will you learn what topics have been discussed before, but also how discussion tends to proceed on that list, who the participants are, and who the target audience is. These are always good things to know before you post to any mailing list, not just a FreeBSD mailing list. There is no doubt that the archives are quite extensive, and some questions recur more often than others, sometimes as followups where the subject line no longer accurately reflects the new content. Nevertheless, the burden is on you, the poster, to do your homework to help avoid these recurring topics. [[bikeshed]] == What Is A "Bikeshed"? Literally, a `bikeshed` is a small outdoor shelter into which one may store one's two-wheeled form of transportation. However, in FreeBSD parlance, the term refers to topics that are simple enough that (nearly) anyone can offer an opinion about, and often (nearly) everyone does. The genesis of this term is explained in more detail link:{faq}#bikeshed-painting[in this document]. You simply must have a working knowledge of this concept before posting to any FreeBSD mailing list. More generally, a bikeshed is a topic that will tend to generate immediate meta-discussions and flames if you have not read up on their past history. Please help us to keep the mailing lists as useful for as many people as possible by avoiding bikesheds whenever you can. Thanks. [[acknowledgments]] == Acknowledgments `{grog}`:: Original author of most of the material on mailing list etiquette, taken from the article on link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list]. `{linimon}`:: Creation of the rough draft of this FAQ. diff --git a/documentation/content/en/articles/nanobsd/_index.adoc b/documentation/content/en/articles/nanobsd/_index.adoc index f7ca27afa6..b819e0ee10 100644 --- a/documentation/content/en/articles/nanobsd/_index.adoc +++ b/documentation/content/en/articles/nanobsd/_index.adoc @@ -1,434 +1,446 @@ --- title: Introduction to NanoBSD authors: - author: Daniel Gerzo copyright: 2006 The FreeBSD Documentation Project description: This document provides information about the NanoBSD tools, which can be used to create FreeBSD system images for embedded applications, suitable for use on a USB key, memory card or other mass storage media. trademarks: ["freebsd", "general"] tags: ["nanobsd", "guide", "embedded", "FreeBSD"] --- = Introduction to NanoBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This document provides information about the NanoBSD tools, which can be used to create FreeBSD system images for embedded applications, suitable for use on a USB key, memory card or other mass storage media. ''' toc::[] [[intro]] == Introduction to NanoBSD NanoBSD is a tool developed by {phk} and now maintained by {imp}. It creates a FreeBSD system image for embedded applications, suitable for use on a USB key, memory card or other mass storage media. It can be used to build specialized install images, designed for easy installation and maintenance of systems commonly called "computer appliances". Computer appliances have their hardware and software bundled in the product, which means all applications are pre-installed. The appliance is plugged into an existing network and can begin working (almost) immediately. The features of NanoBSD include: * Ports and packages work as in FreeBSD - Every single application can be installed and used in a NanoBSD image, the same way as in FreeBSD. * No missing functionality - If it is possible to do something with FreeBSD, it is possible to do the same thing with NanoBSD, unless the specific feature or features were explicitly removed from the NanoBSD image when it was created. * Everything is read-only at run-time - It is safe to pull the power-plug. There is no necessity to run man:fsck[8] after a non-graceful shutdown of the system. * Easy to build and customize - Making use of just one shell script and one configuration file it is possible to build reduced and customized images satisfying any arbitrary set of requirements. [[howto]] == NanoBSD Howto [[design]] === The Design of NanoBSD Once the image is present on the medium, it is possible to boot NanoBSD. The mass storage medium is divided into three parts by default: * Two image partitions: `code#1` and `code#2`. * The configuration file partition, which can be mounted under the [.filename]#/cfg# directory at run time. These partitions are normally mounted read-only. The [.filename]#/etc# and [.filename]#/var# directories are man:md[4] (malloc) disks. The configuration file partition persists under the [.filename]#/cfg# directory. It contains files for [.filename]#/etc# directory and is briefly mounted read-only right after the system boot, therefore it is required to copy modified files from [.filename]#/etc# back to the [.filename]#/cfg# directory if changes are expected to persist after the system restarts. .Making Persistent Changes to [.filename]#/etc/resolv.conf# [example] ==== [source,shell] .... # vi /etc/resolv.conf [...] # mount /cfg # cp /etc/resolv.conf /cfg # umount /cfg .... ==== [NOTE] ==== The partition containing [.filename]#/cfg# should be mounted only at boot time and while overriding the configuration files. Keeping [.filename]#/cfg# mounted at all times is not a good idea, especially if the NanoBSD system runs off a mass storage medium that may be adversely affected by a large number of writes to the partition (like when the filesystem syncer flushes data to the system disks). ==== === Building a NanoBSD Image A NanoBSD image is built using a simple [.filename]#nanobsd.sh# shell script, which can be found in the [.filename]#/usr/src/tools/tools/nanobsd# directory. This script creates an image, which can be copied on the storage medium using the man:dd[1] utility. The necessary commands to build a NanoBSD image are: [source,shell] .... # cd /usr/src/tools/tools/nanobsd <.> # sh nanobsd.sh <.> # cd /usr/obj/nanobsd.full <.> # dd if=_.disk.full of=/dev/da0 bs=64k <.> .... <.> Change the current directory to the base directory of the NanoBSD build script. <.> Start the build process. <.> Change the current directory to the place where the built images are located. <.> Install NanoBSD onto the storage medium. ==== Options When Building a NanoBSD Image When building a NanoBSD image, several build options can be passed to [.filename]#nanobsd.sh# on the command line. These options can have a significant impact on the build process. Some options are for verbosity purposes: * `-h`: prints the help summary page. * `-q`: makes output quieter. * `-v`: makes output more verbose Some other options can be used to restrict the building process. Sometimes it is not necessary to rebuild everything from sources, especially if an image has already been built, and only little change is made. * `-k`: do not build the kernel * `-w`: do not build world * `-b`: do not build either kernel and world * `-i`: do not build a disk image at all. As a file will not be created, it will not be possible to man:dd[1] it to a storage media. * `-f`: do not build a disk image of the first partition (which is useful for upgrade purposes) * `-n`: add `-DNO_CLEAN` to `buildworld`, `buildkernel`. Also, all the files that have already been built in a previous run are kept. A configuration file can be used to tweak as many elements as desired. Load it with `-c` The last options are: * `-K`: do not install a kernel. A disk image without a kernel will not be able to achieve a normal boot sequence. ==== The Complete Image Building Process The complete image building process is going through a lot of steps. The exact steps taken will depend on the chosen options when starting the script. Assuming the script is run with no particular options, this is what will happen. . `run_early_customize`: commands that are defined in a supplied configuration file. . `clean_build`: Just cleans the build environment by deleting the previously built files. . `make_conf_build`: Assemble make.conffrom the `CONF_WORLD` and `CONF_BUILD` variables. . `build_world`: Build world. . `build_kernel`: Build the kernel files. . `clean_world`: Clean the destination directory. . `make_conf_install`: Assemble make.conf from the `CONF_WORLD` and `CONF_INSTALL` variables. . `install_world`: Install all files built during `buildworld`. . `install_etc`: Install the necessary files in the [.filename]#/etc# directory, based on the `make distribution` command. . `setup_nanobsd_etc`: the first configuration specific to NanoBSD takes place at this stage. The [.filename]#/etc/diskless# is created and the root filesystem is defined as read-only. . `install_kernel`: the kernel and modules files are installed. . `run_customize`: all the customizing routines defined by the user will be called. . `setup_nanobsd`: a special configuration directory layout is setup. The [.filename]#/usr/local/etc# gets moved to [.filename]#/etc/local# and a symbolic link is created back from [.filename]#/etc/local# to [.filename]#/usr/local/etc#. . `prune_usr`: the empty directories from [.filename]#/usr# are removed. . `run_late_customize`: the very last custom scripts can be run at this point. . `fixup_before_diskimage`: List all installed files in a metalog . `create_diskimage`: creates the actual disk image, based on the disk geometries provides parameters. . `last_orders`: does nothing for now. === Customizing a NanoBSD Image This is probably the most important and most interesting feature of NanoBSD. This is also where you will be spending most of the time when developing with NanoBSD. Invocation of the following command will force the [.filename]#nanobsd.sh# to read its configuration from [.filename]#myconf.nano# located in the current directory: [source,shell] .... # sh nanobsd.sh -c myconf.nano .... Customization is done in two ways: * Configuration options * Custom functions ==== Configuration Options With configuration settings, it is possible to configure options passed to both the `buildworld` and `installworld` stages of the NanoBSD build process, as well as internal options passed to the main build process of NanoBSD. Through these options it is possible to cut the system down, so it will fit on as little as 64MB. You can use the configuration options to trim down FreeBSD even more, until it will consists of just the kernel and two or three files in the userland. The configuration file consists of configuration options, which override the default values. The most important directives are: * `NANO_NAME` - Name of build (used to construct the workdir names). * `NANO_SRC` - Path to the source tree used to build the image. * `NANO_KERNEL` - Name of kernel configuration file used to build kernel. * `CONF_BUILD` - Options passed to the `buildworld` stage of the build. * `CONF_INSTALL` - Options passed to the `installworld` stage of the build. * `CONF_WORLD` - Options passed to both the `buildworld` and the `installworld` stage of the build. * `FlashDevice` - Defines what type of media to use. Check [.filename]#FlashDevice.sub# for more details. There are many more configuration options that could be relevant depending upon the kind of NanoBSD that is desired. ===== General Customization There are three stages, by design, at which it is possible to make changes that affect the building process, just by setting up a variable in the provided configuration file: * `run_early_customize`: before anything else happens. * `run_customize`: after all the standard files have been laid out * `run_late_customize`: at the very end of the process, just before the actual NanoBSD image is built. To customize a NanoBSD image, at any of these steps, it is best to add a specific value to one of the corresponding variables. The `NANO_EARLY_CUSTOMIZE` variable is used at the first step of the building process. At this point, there is no example as to what can be done using that variable, but it may change in the future. The `NANO_CUSTOMIZE` variable is used after the kernel, world and etc configuration files have been installed, and the etc files have been setup as to be a NanoBSD installation. So it is the correct step in the building process to tweak configuration options and add packages, like in the cust_nobeastie example. The `NANO_LATE_CUSTOMIZE` variable is used just before the disk image is created, so it is the very last moment to change anything. Remember that the `setup_nanobsd` routine already executed and that the [.filename]#etc#, [.filename]#conf# and [.filename]#cfg# directories and subdirectories are already modified, so it is not time to change them at this point. Rather, it is possible to add or remove specific files. ===== Booting Options There are also variables that can change the way the NanoBSD image boots. Two options are passed to man:boot0cfg[8] to initialize the boot sector of the disk image: * `NANO_BOOT0CFG` * `NANO_BOOTLOADER` With `NANO_BOOTLOADER` a bootloader file can be chosen. The most common possible options are between [.filename]#boot0sio# and [.filename]#boot0# depending on whether the appliance has a serial port or not. It is best to avoid supplying a different bootloader, but it is possible. To do so, it is best to have checked the link:{handbook}boot/[FreeBSD Handbook] chapter on the boot process. With `NANO_BOOT0CFG`, the booting process can be tweaked, like selecting on which partition the NanoBSD image will actually boot. It is best to check the man:boot0cfg[8] page before changing the default value of this variable. One option that could be interesting to change is the timeout of the booting procedure. To do so, the `NANO_BOOT0CFG` variable can be changed to `"-o packet -s 1 -m 3 -t 36"`. That way the booting process would start after approximately 2 seconds; because it is rare that waiting 10 seconds before actually booting is desired. Good to know: the `NANO_BOOT2CFG` variable is only used in the `cust_comconsole` routine that can be called at the `NANO_CUSTOMIZE` step if the appliance has a serial port and all console input and output has to take place through it. Be sure to check the relevant parameters of the serial port, as setting a bad parameter value can make it useless. ===== Disk Image Creation In the end of the boot process is the disk image creation. With this step, the NanoBSD script provides a file that can simply be copied onto a disk for the appliance, and that will make it boot and start. There are many variable that need to be set just right for the script to produce a usable disk image. * The `NANO_DRIVE` variable must be set to the drive name of the media at runtime. Usually, the default value `ada0`, which represents the first `IDE`/`ATA`/`SATA` device on the appliance is expected to be the correct one, but a different type of storage could also be used - like a USB key, in which case, it would rather be da0. * The `NANO_MEDIASIZE` variable must be set to the size (in 512 bytes sectors) of the storage media that will be used. If you set it wrong, it is possible that the NanoBSD image will not boot at all, and a message at boot time will be warning about incorrect disk geometry. * The [.filename]#/etc#, [.filename]#/var#, and [.filename]#/tmp# directories are allocated as man:md[4] (malloc) disks at boot time; so their sizes can be tailored to suit the appliance needs. The `NANO_RAM_ETCSIZE` variable sets the size of the [.filename]#/etc#; and the `NANO_RAM_TMPVARSIZE` variable sets the size of both the [.filename]#/var# and [.filename]#/tmp# directory, as [.filename]#/tmp# is symbolically linked to [.filename]#/var/tmp#. By default, both malloc disks sizes are set at 20MB each. They can always be changed, but usually the [.filename]#/etc# does not grow too much in size, so 20MB is a good starting point, whereas the [.filename]#/var# and especially [.filename]#/tmp# can grow much larger if not careful about it. For memory constrained systems, smaller filesystem sizes may be chosen. * As NanoBSD is mainly designed to build a system image for an appliance, it is assumed that the storage media used will be relatively small. For that reason, the filesystem that is laid out is configured to have a small block size (4Kb) and a small fragment size (512b). The configuration options of the filesystem can be modified through the `NANO_NEWFS` variable, but the syntax must respect the man:newfs[8] command format. Also, by default, the filesystem has Soft Updates enabled. The link:{handbook}[FreeBSD Handbook] can be checked about this. * The different partition sizes can be set through the use of `NANO_CODESIZE`, `NANO_CONFSIZE`, and `NANO_DATASIZE` as a multiple of 512 bytes sectors. `NANO_CODESIZE` defines the size of the first two image partitions: `code#1` and `code#2`. They have to be big enough to hold all the files that will be produced as a result of the `buildworld` and `buildkernel` processes. `NANO_CONFSIZE` defines the size of the configuration file partition, so it does not need to be very big; but do not make it so small that it will not hold all configuration files. Finally, `NANO_DATASIZE` defines the size of an optional partition, that can be used on the appliance. The last partition can be used, for example, to keep files created on the fly on disk. ==== Custom Functions It is possible to fine-tune NanoBSD using shell functions in the configuration file. The following example illustrates the basic model of custom functions: [.programlisting] .... cust_foo () ( echo "bar=baz" > \ ${NANO_WORLDDIR}/etc/foo ) customize_cmd cust_foo .... A more useful example of a customization function is the following, which changes the default size of the [.filename]#/etc# directory from 5MB to 30MB: [.programlisting] .... cust_etc_size () ( cd ${NANO_WORLDDIR}/conf echo 30000 > default/etc/md_size ) customize_cmd cust_etc_size .... There are a few default pre-defined customization functions ready for use: * `cust_comconsole` - Disables man:getty[8] on the VGA devices (the [.filename]#/dev/ttyv*# device nodes) and enables the use of the COM1 serial port as the system console. * `cust_allow_ssh_root` - Allow `root` to login via man:sshd[8]. * `cust_install_files` - Installs files from the [.filename]#nanobsd/Files# directory, which contains some useful scripts for system administration. ==== Adding Packages Packages can be added to a NanoBSD image, to provide specific functionalities on the appliance. To do so, either: * Add the `cust_pkgng` to the `NANO_CUSTOMIZE` variable, or * Add a `'customize_cmd cust_pkgng'` command in a customized configuration file. Both methods achieve the same result: launching the `cust_pkgng` routine. This routine will go through `NANO_PACKAGE_DIR` directory to find either all packages or just the list of packages in the `NANO_PACKAGE_LIST` variable. It is common, when installing applications through pkg on a standard FreeBSD environment, that the install process puts configuration files, in the [.filename]#usr/local/etc# directory, and startup scripts in the [.filename]#/usr/local/etc/rc.d# directory. So, after the required packages have been installed, they need to be configured in order for them to start right out of the box. To do so, the necessary configuration files have to be installed in the correct directories. This can be achieved by writing dedicated routines or the generic `cust_install_files` routine can be used to lay out files properly from the [.filename]#/usr/src/tools/tools/nanobsd/Files# directory. Usually a statement, sometimes multiple statements, in the [.filename]#/etc/rc.conf# also needs to be added for each package. ==== Configuration File Example A complete example of a configuration file for building a custom NanoBSD image can be: [.programlisting] .... NANO_NAME=custom NANO_SRC=/usr/src NANO_KERNEL=MYKERNEL NANO_IMAGES=2 CONF_BUILD=' WITHOUT_KLDLOAD=YES WITHOUT_NETGRAPH=YES WITHOUT_PAM=YES ' CONF_INSTALL=' WITHOUT_ACPI=YES WITHOUT_BLUETOOTH=YES WITHOUT_FORTRAN=YES WITHOUT_HTML=YES WITHOUT_LPR=YES WITHOUT_MAN=YES WITHOUT_SENDMAIL=YES WITHOUT_SHAREDOCS=YES WITHOUT_EXAMPLES=YES WITHOUT_INSTALLLIB=YES WITHOUT_CALENDAR=YES WITHOUT_MISC=YES WITHOUT_SHARE=YES ' CONF_WORLD=' WITHOUT_BIND=YES WITHOUT_MODULES=YES WITHOUT_KERBEROS=YES WITHOUT_GAMES=YES WITHOUT_RESCUE=YES WITHOUT_LOCALES=YES WITHOUT_SYSCONS=YES WITHOUT_INFO=YES ' FlashDevice SanDisk 1G cust_nobeastie() ( touch ${NANO_WORLDDIR}/boot/loader.conf echo "beastie_disable=\"YES\"" >> ${NANO_WORLDDIR}/boot/loader.conf ) customize_cmd cust_comconsole customize_cmd cust_install_files customize_cmd cust_allow_ssh_root customize_cmd cust_nobeastie .... All the build and install compilation options can be found in the man:src.conf[5] man page, but not all options can or should be used when building a NanoBSD image. The build and install options should be defined according to the needs of the image being built. For example, the ftp client and server might not be needed. Adding `WITHOUT_FTP=TRUE` to a configuration file in the `CONF_BUILD` section will avoid having them built. Also, if the NanoBSD appliance will not be used to build programs then it is possible to add the `WITHOUT_BINUTILS=TRUE` in the `CONF_INSTALL` section; but not in the `CONF_BUILD` section as they will be used to build the NanoBSD image. Not building a particular set of programs - through a compilation option - shortens the overall building time and lowers the required size for the disk image, whereas not installing the same specific set of programs does not lower the overall building time. === Updating NanoBSD The update process of NanoBSD is relatively simple: [.procedure] ==== . Build a new NanoBSD image, as usual. . Upload the new image into an unused partition of a running NanoBSD appliance. + The most important difference of this step from the initial NanoBSD installation is that now instead of using [.filename]#\_.disk.full# (which contains an image of the entire disk), the [.filename]#_.disk.image# image is installed (which contains an image of a single system partition). . Reboot, and start the system from the newly installed partition. . If all goes well, the upgrade is finished. . If anything goes wrong, reboot back into the previous partition (which contains the old, working image), to restore system functionality as fast as possible. Fix any problems of the new build, and repeat the process. ==== To install new image onto the running NanoBSD system, it is possible to use either the [.filename]#updatep1# or [.filename]#updatep2# script located in the [.filename]#/root# directory, depending from which partition is running the current system. According to which services are available on host serving new NanoBSD image and what type of transfer is preferred, it is possible to examine one of these three ways: ==== Using man:ftp[1] If the transfer speed is in first place, use this example: [source,shell] .... # ftp myhost get _.disk.image "| sh updatep1" .... ==== Using man:ssh[1] If a secure transfer is preferred, consider using this example: [source,shell] .... # ssh myhost cat _.disk.image.gz | zcat | sh updatep1 .... ==== Using man:nc[1] Try this example if the remote host is not running neither man:ftpd[8] or man:sshd[8] service: [.procedure] ==== . At first, open a TCP listener on host serving the image and make it send the image to client: + [source,shell] .... myhost# nc -l 2222 < _.disk.image .... + [NOTE] ====== Make sure that the used port is not blocked to receive incoming connections from NanoBSD host by firewall. ====== . Connect to the host serving new image and execute [.filename]#updatep1# script: + [source,shell] .... # nc myhost 2222 | sh updatep1 .... ==== diff --git a/documentation/content/en/articles/pam/_index.adoc b/documentation/content/en/articles/pam/_index.adoc index e425d941a1..bbea5aa3d2 100644 --- a/documentation/content/en/articles/pam/_index.adoc +++ b/documentation/content/en/articles/pam/_index.adoc @@ -1,642 +1,672 @@ --- title: Pluggable Authentication Modules authors: - author: Dag-Erling Smørgrav copyright: 2001-2003 Networks Associates Technology, Inc. description: A guide to the PAM system and modules under FreeBSD trademarks: ["pam", "freebsd", "linux", "opengroup", "sun", "general"] tags: ["pam", "introduction", "authentication", "modules", "FreeBSD"] --- = Pluggable Authentication Modules :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: [.abstract-title] Abstract This article describes the underlying principles and mechanisms of the Pluggable Authentication Modules (PAM) library, and explains how to configure PAM, how to integrate PAM into applications, and how to write PAM modules. ''' toc::[] [[pam-intro]] == Introduction The Pluggable Authentication Modules (PAM) library is a generalized API for authentication-related services which allows a system administrator to add new authentication methods simply by installing new PAM modules, and to modify authentication policies by editing configuration files. PAM was defined and developed in 1995 by Vipin Samar and Charlie Lai of Sun Microsystems, and has not changed much since. In 1997, the Open Group published the X/Open Single Sign-on (XSSO) preliminary specification, which standardized the PAM API and added extensions for single (or rather integrated) sign-on. At the time of this writing, this specification has not yet been adopted as a standard. Although this article focuses primarily on FreeBSD 5.x, which uses OpenPAM, it should be equally applicable to FreeBSD 4.x, which uses Linux-PAM, and other operating systems such as Linux and Solaris(TM). [[pam-terms]] == Terms and Conventions [[pam-definitions]] === Definitions The terminology surrounding PAM is rather confused. Neither Samar and Lai's original paper nor the XSSO specification made any attempt at formally defining terms for the various actors and entities involved in PAM, and the terms that they do use (but do not define) are sometimes misleading and ambiguous. The first attempt at establishing a consistent and unambiguous terminology was a whitepaper written by Andrew G. Morgan (author of Linux-PAM) in 1999. While Morgan's choice of terminology was a huge leap forward, it is in this author's opinion by no means perfect. What follows is an attempt, heavily inspired by Morgan, to define precise and unambiguous terms for all actors and entities involved in PAM. account:: The set of credentials the applicant is requesting from the arbitrator. applicant:: The user or entity requesting authentication. arbitrator:: The user or entity who has the privileges necessary to verify the applicant's credentials and the authority to grant or deny the request. chain:: A sequence of modules that will be invoked in response to a PAM request. The chain includes information about the order in which to invoke the modules, what arguments to pass to them, and how to interpret the results. client:: The application responsible for initiating an authentication request on behalf of the applicant and for obtaining the necessary authentication information from him. facility:: One of the four basic groups of functionality provided by PAM: authentication, account management, session management and authentication token update. module:: A collection of one or more related functions implementing a particular authentication facility, gathered into a single (normally dynamically loadable) binary file and identified by a single name. policy:: The complete set of configuration statements describing how to handle PAM requests for a particular service. A policy normally consists of four chains, one for each facility, though some services do not use all four facilities. server:: The application acting on behalf of the arbitrator to converse with the client, retrieve authentication information, verify the applicant's credentials and grant or deny requests. service:: A class of servers providing similar or related functionality and requiring similar authentication. PAM policies are defined on a per-service basis, so all servers that claim the same service name will be subject to the same policy. session:: The context within which service is rendered to the applicant by the server. One of PAM's four facilities, session management, is concerned exclusively with setting up and tearing down this context. token:: A chunk of information associated with the account, such as a password or passphrase, which the applicant must provide to prove his identity. transaction:: A sequence of requests from the same applicant to the same instance of the same server, beginning with authentication and session set-up and ending with session tear-down. [[pam-usage-examples]] === Usage Examples This section aims to illustrate the meanings of some of the terms defined above by way of a handful of simple examples. ==== Client and Server Are One This simple example shows `alice` man:su[1]'ing to `root`. [source,shell] .... % whoami alice % ls -l `which su` -r-sr-xr-x 1 root wheel 10744 Dec 6 19:06 /usr/bin/su % su - Password: xi3kiune # whoami root .... * The applicant is `alice`. * The account is `root`. * The man:su[1] process is both client and server. * The authentication token is `xi3kiune`. * The arbitrator is `root`, which is why man:su[1] is setuid `root`. ==== Client and Server Are Separate The example below shows `eve` try to initiate an man:ssh[1] connection to `login.example.com`, ask to log in as `bob`, and succeed. Bob should have chosen a better password! [source,shell] .... % whoami eve % ssh bob@login.example.com bob@login.example.com's password: % god Last login: Thu Oct 11 09:52:57 2001 from 192.168.0.1 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.4-STABLE (LOGIN) 4: Tue Nov 27 18:10:34 PST 2001 Welcome to FreeBSD! % .... * The applicant is `eve`. * The client is Eve's man:ssh[1] process. * The server is the man:sshd[8] process on `login.example.com` * The account is `bob`. * The authentication token is `god`. * Although this is not shown in this example, the arbitrator is `root`. ==== Sample Policy The following is FreeBSD's default policy for `sshd`: [.programlisting] .... sshd auth required pam_nologin.so no_warn sshd auth required pam_unix.so no_warn try_first_pass sshd account required pam_login_access.so sshd account required pam_unix.so sshd session required pam_lastlog.so no_fail sshd password required pam_permit.so .... * This policy applies to the `sshd` service (which is not necessarily restricted to the man:sshd[8] server.) * `auth`, `account`, `session` and `password` are facilities. * [.filename]#pam_nologin.so#, [.filename]#pam_unix.so#, [.filename]#pam_login_access.so#, [.filename]#pam_lastlog.so# and [.filename]#pam_permit.so# are modules. It is clear from this example that [.filename]#pam_unix.so# provides at least two facilities (authentication and account management.) [[pam-essentials]] == PAM Essentials [[pam-facilities-primitives]] === Facilities and Primitives The PAM API offers six different authentication primitives grouped in four facilities, which are described below. `auth`:: _Authentication._ This facility concerns itself with authenticating the applicant and establishing the account credentials. It provides two primitives: ** man:pam_authenticate[3] authenticates the applicant, usually by requesting an authentication token and comparing it with a value stored in a database or obtained from an authentication server. ** man:pam_setcred[3] establishes account credentials such as user ID, group membership and resource limits. `account`:: _Account management._ This facility handles non-authentication-related issues of account availability, such as access restrictions based on the time of day or the server's work load. It provides a single primitive: ** man:pam_acct_mgmt[3] verifies that the requested account is available. `session`:: _Session management._ This facility handles tasks associated with session set-up and tear-down, such as login accounting. It provides two primitives: ** man:pam_open_session[3] performs tasks associated with session set-up: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, start an SSH agent, etc. ** man:pam_close_session[3] performs tasks associated with session tear-down: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, stop the SSH agent, etc. `password`:: _Password management._ This facility is used to change the authentication token associated with an account, either because it has expired or because the user wishes to change it. It provides a single primitive: ** man:pam_chauthtok[3] changes the authentication token, optionally verifying that it is sufficiently hard to guess, has not been used previously, etc. [[pam-modules]] === Modules Modules are a very central concept in PAM; after all, they are the "M" in "PAM". A PAM module is a self-contained piece of program code that implements the primitives in one or more facilities for one particular mechanism; possible mechanisms for the authentication facility, for instance, include the UNIX(R) password database, NIS, LDAP and Radius. [[pam-module-naming]] ==== Module Naming FreeBSD implements each mechanism in a single module, named `pam_mechanism.so` (for instance, `pam_unix.so` for the UNIX(R) mechanism.) Other implementations sometimes have separate modules for separate facilities, and include the facility name as well as the mechanism name in the module name. To name one example, Solaris(TM) has a `pam_dial_auth.so.1` module which is commonly used to authenticate dialup users. [[pam-module-versioning]] ==== Module Versioning FreeBSD's original PAM implementation, based on Linux-PAM, did not use version numbers for PAM modules. This would commonly cause problems with legacy applications, which might be linked against older versions of the system libraries, as there was no way to load a matching version of the required modules. OpenPAM, on the other hand, looks for modules that have the same version number as the PAM library (currently 2), and only falls back to an unversioned module if no versioned module could be loaded. Thus legacy modules can be provided for legacy applications, while allowing new (or newly built) applications to take advantage of the most recent modules. Although Solaris(TM) PAM modules commonly have a version number, they are not truly versioned, because the number is a part of the module name and must be included in the configuration. [[pam-chains-policies]] === Chains and Policies When a server initiates a PAM transaction, the PAM library tries to load a policy for the service specified in the man:pam_start[3] call. The policy specifies how authentication requests should be processed, and is defined in a configuration file. This is the other central concept in PAM: the possibility for the admin to tune the system security policy (in the wider sense of the word) simply by editing a text file. A policy consists of four chains, one for each of the four PAM facilities. Each chain is a sequence of configuration statements, each specifying a module to invoke, some (optional) parameters to pass to the module, and a control flag that describes how to interpret the return code from the module. Understanding the control flags is essential to understanding PAM configuration files. There are four different control flags: `binding`:: If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the rest of the chain is executed, but the request is ultimately denied. + This control flag was introduced by Sun in Solaris(TM) 9 (SunOS(TM) 5.9), and is also supported by OpenPAM. `required`:: If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the rest of the chain is also executed, but the request is ultimately denied. `requisite`:: If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the chain is immediately terminated and the request is denied. `sufficient`:: If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the module is ignored and the rest of the chain is executed. + As the semantics of this flag may be somewhat confusing, especially when it is used for the last module in a chain, it is recommended that the `binding` control flag be used instead if the implementation supports it. `optional`:: The module is executed, but its result is ignored. If all modules in a chain are marked `optional`, all requests will always be granted. When a server invokes one of the six PAM primitives, PAM retrieves the chain for the facility the primitive belongs to, and invokes each of the modules listed in the chain, in the order they are listed, until it reaches the end, or determines that no further processing is necessary (either because a `binding` or `sufficient` module succeeded, or because a `requisite` module failed.) The request is granted if and only if at least one module was invoked, and all non-optional modules succeeded. Note that it is possible, though not very common, to have the same module listed several times in the same chain. For instance, a module that looks up user names and passwords in a directory server could be invoked multiple times with different parameters specifying different directory servers to contact. PAM treat different occurrences of the same module in the same chain as different, unrelated modules. [[pam-transactions]] === Transactions The lifecycle of a typical PAM transaction is described below. Note that if any of these steps fails, the server should report a suitable error message to the client and abort the transaction. . If necessary, the server obtains arbitrator credentials through a mechanism independent of PAM-most commonly by virtue of having been started by `root`, or of being setuid `root`. . The server calls man:pam_start[3] to initialize the PAM library and specify its service name and the target account, and register a suitable conversation function. . The server obtains various information relating to the transaction (such as the applicant's user name and the name of the host the client runs on) and submits it to PAM using man:pam_set_item[3]. . The server calls man:pam_authenticate[3] to authenticate the applicant. . The server calls man:pam_acct_mgmt[3] to verify that the requested account is available and valid. If the password is correct but has expired, man:pam_acct_mgmt[3] will return `PAM_NEW_AUTHTOK_REQD` instead of `PAM_SUCCESS`. . If the previous step returned `PAM_NEW_AUTHTOK_REQD`, the server now calls man:pam_chauthtok[3] to force the client to change the authentication token for the requested account. . Now that the applicant has been properly authenticated, the server calls man:pam_setcred[3] to establish the credentials of the requested account. It is able to do this because it acts on behalf of the arbitrator, and holds the arbitrator's credentials. . Once the correct credentials have been established, the server calls man:pam_open_session[3] to set up the session. . The server now performs whatever service the client requested-for instance, provide the applicant with a shell. . Once the server is done serving the client, it calls man:pam_close_session[3] to tear down the session. . Finally, the server calls man:pam_end[3] to notify the PAM library that it is done and that it can release whatever resources it has allocated in the course of the transaction. [[pam-config]] == PAM Configuration [[pam-config-file]] === PAM Policy Files [[pam-config-pam.conf]] ==== The [.filename]#/etc/pam.conf# The traditional PAM policy file is [.filename]#/etc/pam.conf#. This file contains all the PAM policies for your system. Each line of the file describes one step in a chain, as shown below: [.programlisting] .... login auth required pam_nologin.so no_warn .... The fields are, in order: service name, facility name, control flag, module name, and module arguments. Any additional fields are interpreted as additional module arguments. A separate chain is constructed for each service / facility pair, so while the order in which lines for the same service and facility appear is significant, the order in which the individual services and facilities are listed is not. The examples in the original PAM paper grouped configuration lines by facility, and the Solaris(TM) stock [.filename]#pam.conf# still does that, but FreeBSD's stock configuration groups configuration lines by service. Either way is fine; either way makes equal sense. [[pam-config-pam.d]] ==== The [.filename]#/etc/pam.d# OpenPAM and Linux-PAM support an alternate configuration mechanism, which is the preferred mechanism in FreeBSD. In this scheme, each policy is contained in a separate file bearing the name of the service it applies to. These files are stored in [.filename]#/etc/pam.d/#. These per-service policy files have only four fields instead of [.filename]#pam.conf#'s five: the service name field is omitted. Thus, instead of the sample [.filename]#pam.conf# line from the previous section, one would have the following line in [.filename]#/etc/pam.d/login#: [.programlisting] .... auth required pam_nologin.so no_warn .... As a consequence of this simplified syntax, it is possible to use the same policy for multiple services by linking each service name to a same policy file. For instance, to use the same policy for the `su` and `sudo` services, one could do as follows: [source,shell] .... # cd /etc/pam.d # ln -s su sudo .... This works because the service name is determined from the file name rather than specified in the policy file, so the same file can be used for multiple differently-named services. Since each service's policy is stored in a separate file, the [.filename]#pam.d# mechanism also makes it very easy to install additional policies for third-party software packages. [[pam-config-file-order]] ==== The Policy Search Order As we have seen above, PAM policies can be found in a number of places. What happens if policies for the same service exist in multiple places? It is essential to understand that PAM's configuration system is centered on chains. [[pam-config-breakdown]] === Breakdown of a Configuration Line As explained in <>, each line in [.filename]#/etc/pam.conf# consists of four or more fields: the service name, the facility name, the control flag, the module name, and zero or more module arguments. The service name is generally (though not always) the name of the application the statement applies to. If you are unsure, refer to the individual application's documentation to determine what service name it uses. Note that if you use [.filename]#/etc/pam.d/# instead of [.filename]#/etc/pam.conf#, the service name is specified by the name of the policy file, and omitted from the actual configuration lines, which then start with the facility name. The facility is one of the four facility keywords described in <>. Likewise, the control flag is one of the four keywords described in <>, describing how to interpret the return code from the module. Linux-PAM supports an alternate syntax that lets you specify the action to associate with each possible return code, but this should be avoided as it is non-standard and closely tied in with the way Linux-PAM dispatches service calls (which differs greatly from the way Solaris(TM) and OpenPAM do it.) Unsurprisingly, OpenPAM does not support this syntax. [[pam-policies]] === Policies To configure PAM correctly, it is essential to understand how policies are interpreted. When an application calls man:pam_start[3], the PAM library loads the policy for the specified service and constructs four module chains (one for each facility.) If one or more of these chains are empty, the corresponding chains from the policy for the `other` service are substituted. When the application later calls one of the six PAM primitives, the PAM library retrieves the chain for the corresponding facility and calls the appropriate service function in each module listed in the chain, in the order in which they were listed in the configuration. After each call to a service function, the module type and the error code returned by the service function are used to determine what happens next. With a few exceptions, which we discuss below, the following table applies: .PAM Chain Execution Summary [cols="1,1,1,1", options="header"] |=== | | PAM_SUCCESS | PAM_IGNORE | other |binding |if (!fail) break; |- |fail = true; |required |- |- |fail = true; |requisite |- |- |fail = true; break; |sufficient |if (!fail) break; |- |- |optional |- |- |- |=== If `fail` is true at the end of a chain, or when a "break" is reached, the dispatcher returns the error code returned by the first module that failed. Otherwise, it returns `PAM_SUCCESS`. The first exception of note is that the error code `PAM_NEW_AUTHTOK_REQD` is treated like a success, except that if no module failed, and at least one module returned `PAM_NEW_AUTHTOK_REQD`, the dispatcher will return `PAM_NEW_AUTHTOK_REQD`. The second exception is that man:pam_setcred[3] treats `binding` and `sufficient` modules as if they were `required`. The third and final exception is that man:pam_chauthtok[3] runs the entire chain twice (once for preliminary checks and once to actually set the password), and in the preliminary phase it treats `binding` and `sufficient` modules as if they were `required`. [[pam-freebsd-modules]] == FreeBSD PAM Modules [[pam-modules-deny]] === man:pam_deny[8] The man:pam_deny[8] module is one of the simplest modules available; it responds to any request with `PAM_AUTH_ERR`. It is useful for quickly disabling a service (add it to the top of every chain), or for terminating chains of `sufficient` modules. [[pam-modules-echo]] === man:pam_echo[8] The man:pam_echo[8] module simply passes its arguments to the conversation function as a `PAM_TEXT_INFO` message. It is mostly useful for debugging, but can also serve to display messages such as "Unauthorized access will be prosecuted" before starting the authentication procedure. [[pam-modules-exec]] === man:pam_exec[8] The man:pam_exec[8] module takes its first argument to be the name of a program to execute, and the remaining arguments are passed to that program as command-line arguments. One possible application is to use it to run a program at login time which mounts the user's home directory. [[pam-modules-ftpusers]] === man:pam_ftpusers[8] The man:pam_ftpusers[8] module [[pam-modules-group]] === man:pam_group[8] The man:pam_group[8] module accepts or rejects applicants on the basis of their membership in a particular file group (normally `wheel` for man:su[1]). It is primarily intended for maintaining the traditional behavior of BSD man:su[1], but has many other uses, such as excluding certain groups of users from a particular service. [[pam-modules-guest]] === man:pam_guest[8] The man:pam_guest[8] module allows guest logins using fixed login names. Various requirements can be placed on the password, but the default behavior is to allow any password as long as the login name is that of a guest account. The man:pam_guest[8] module can easily be used to implement anonymous FTP logins. [[pam-modules-krb5]] === man:pam_krb5[8] The man:pam_krb5[8] module [[pam-modules-ksu]] === man:pam_ksu[8] The man:pam_ksu[8] module [[pam-modules-lastlog]] === man:pam_lastlog[8] The man:pam_lastlog[8] module [[pam-modules-login-access]] === man:pam_login_access[8] The man:pam_login_access[8] module provides an implementation of the account management primitive which enforces the login restrictions specified in the man:login.access[5] table. [[pam-modules-nologin]] === man:pam_nologin[8] The man:pam_nologin[8] module refuses non-root logins when [.filename]#/var/run/nologin# exists. This file is normally created by man:shutdown[8] when less than five minutes remain until the scheduled shutdown time. [[pam-modules-opie]] === man:pam_opie[8] The man:pam_opie[8] module implements the man:opie[4] authentication method. The man:opie[4] system is a challenge-response mechanism where the response to each challenge is a direct function of the challenge and a passphrase, so the response can be easily computed "just in time" by anyone possessing the passphrase, eliminating the need for password lists. Moreover, since man:opie[4] never reuses a challenge that has been correctly answered, it is not vulnerable to replay attacks. [[pam-modules-opieaccess]] === man:pam_opieaccess[8] The man:pam_opieaccess[8] module is a companion module to man:pam_opie[8]. Its purpose is to enforce the restrictions codified in man:opieaccess[5], which regulate the conditions under which a user who would normally authenticate herself using man:opie[4] is allowed to use alternate methods. This is most often used to prohibit the use of password authentication from untrusted hosts. In order to be effective, the man:pam_opieaccess[8] module must be listed as `requisite` immediately after a `sufficient` entry for man:pam_opie[8], and before any other modules, in the `auth` chain. [[pam-modules-passwdqc]] === man:pam_passwdqc[8] The man:pam_passwdqc[8] module [[pam-modules-permit]] === man:pam_permit[8] The man:pam_permit[8] module is one of the simplest modules available; it responds to any request with `PAM_SUCCESS`. It is useful as a placeholder for services where one or more chains would otherwise be empty. [[pam-modules-radius]] === man:pam_radius[8] The man:pam_radius[8] module [[pam-modules-rhosts]] === man:pam_rhosts[8] The man:pam_rhosts[8] module [[pam-modules-rootok]] === man:pam_rootok[8] The man:pam_rootok[8] module reports success if and only if the real user id of the process calling it (which is assumed to be run by the applicant) is 0. This is useful for non-networked services such as man:su[1] or man:passwd[1], to which the `root` should have automatic access. [[pam-modules-securetty]] === man:pam_securetty[8] The man:pam_securetty[8] module [[pam-modules-self]] === man:pam_self[8] The man:pam_self[8] module reports success if and only if the names of the applicant matches that of the target account. It is most useful for non-networked services such as man:su[1], where the identity of the applicant can be easily verified. [[pam-modules-ssh]] === man:pam_ssh[8] The man:pam_ssh[8] module provides both authentication and session services. The authentication service allows users who have passphrase-protected SSH secret keys in their [.filename]#~/.ssh# directory to authenticate themselves by typing their passphrase. The session service starts man:ssh-agent[1] and preloads it with the keys that were decrypted in the authentication phase. This feature is particularly useful for local logins, whether in X (using man:xdm[1] or another PAM-aware X login manager) or at the console. [[pam-modules-tacplus]] === man:pam_tacplus[8] The man:pam_tacplus[8] module [[pam-modules-unix]] === man:pam_unix[8] The man:pam_unix[8] module implements traditional UNIX(R) password authentication, using man:getpwnam[3] to obtain the target account's password and compare it with the one provided by the applicant. It also provides account management services (enforcing account and password expiration times) and password-changing services. This is probably the single most useful module, as the great majority of admins will want to maintain historical behavior for at least some services. [[pam-appl-prog]] == PAM Application Programming This section has not yet been written. [[pam-module-prog]] == PAM Module Programming This section has not yet been written. :sectnums!: [appendix] [[pam-sample-appl]] == Sample PAM Application The following is a minimal implementation of man:su[1] using PAM. Note that it uses the OpenPAM-specific man:openpam_ttyconv[3] conversation function, which is prototyped in [.filename]#security/openpam.h#. If you wish build this application on a system with a different PAM library, you will have to provide your own conversation function. A robust conversation function is surprisingly difficult to implement; the one presented in <> is a good starting point, but should not be used in real-world applications. [.programlisting] .... +ifeval::["{backend}" == "html5"] include::static/source/articles/pam/su.c[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../static/source/articles/pam/su.c[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../static/source/articles/pam/su.c[] +endif::[] .... :sectnums!: [appendix] [[pam-sample-module]] == Sample PAM Module The following is a minimal implementation of man:pam_unix[8], offering only authentication services. It should build and run with most PAM implementations, but takes advantage of OpenPAM extensions if available: note the use of man:pam_get_authtok[3], which enormously simplifies prompting the user for a password. [.programlisting] .... +ifeval::["{backend}" == "html5"] include::static/source/articles/pam/pam_unix.c[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../static/source/articles/pam/pam_unix.c[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../static/source/articles/pam/pam_unix.c[] +endif::[] .... :sectnums!: [appendix] [[pam-sample-conv]] == Sample PAM Conversation Function The conversation function presented below is a greatly simplified version of OpenPAM's man:openpam_ttyconv[3]. It is fully functional, and should give the reader a good idea of how a conversation function should behave, but it is far too simple for real-world use. Even if you are not using OpenPAM, feel free to download the source code and adapt man:openpam_ttyconv[3] to your uses; we believe it to be as robust as a tty-oriented conversation function can reasonably get. [.programlisting] .... +ifeval::["{backend}" == "html5"] include::static/source/articles/pam/converse.c[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../static/source/articles/pam/converse.c[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../static/source/articles/pam/converse.c[] +endif::[] .... :sectnums!: [[pam-further]] == Further Reading === Papers Making Login Services Independent of Authentication Technologies Vipin Samar. Charlie Lai. Sun Microsystems. _link:https://pubs.opengroup.org/onlinepubs/8329799/toc.htm[X/Open Single Sign-on Preliminary Specification]_. The Open Group. 1-85912-144-6. June 1997. _link:https://mirrors.kernel.org/pub/linux/libs/pam/pre/doc/draft-morgan-pam-07.txt[Pluggable Authentication Modules]_. Andrew G. Morgan. 1999-10-06. === User Manuals _link:https://docs.oracle.com/cd/E26505_01/html/E27224/pam-1.html[PAM Administration]_. Sun Microsystems. === Related Web Pages _link:https://www.openpam.org/[OpenPAM homepage]_ Dag-Erling Smørgrav. ThinkSec AS. _link:http://www.kernel.org/pub/linux/libs/pam/[Linux-PAM homepage]_ Andrew Morgan. _Solaris PAM homepage_. Sun Microsystems. diff --git a/documentation/content/en/articles/pgpkeys/_index.adoc b/documentation/content/en/articles/pgpkeys/_index.adoc index 4921806f61..d7d7698fb1 100644 --- a/documentation/content/en/articles/pgpkeys/_index.adoc +++ b/documentation/content/en/articles/pgpkeys/_index.adoc @@ -1,1795 +1,1807 @@ --- title: OpenPGP Keys description: List of OpenPGP keys that can be used to verify a signature or send encrypted email to FreeBSD.org officers or developers. tags: ["OpenPGP", "Developers", "Officers", "FreeBSD"] --- = OpenPGP Keys :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/teams.adoc[lines=16..-1] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/teams.adoc[lines=16..-1] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/teams.adoc[lines=16..-1] +endif::[] ''' toc::[] These OpenPGP keys can be used to verify a signature or send encrypted email to `FreeBSD.org` officers or developers. The complete keyring can be downloaded at link:https://www.FreeBSD.org/doc/pgpkeyring.txt[https://www.FreeBSD.org/doc/pgpkeyring.txt]. //// Do not edit this file except as instructed by the addkey.sh script. See the README file in /data/pgpkeys for instructions. This article contains all the keys. The officer keys are also shown in the Handbook PGP keys chapter. //// [[pgpkeys-officers]] == Officers === {security-officer-name} `<{security-officer-email}>` include::static/pgpkeys/security-officer.key[] === {secteam-secretary-name} `<{secteam-secretary-email}>` include::static/pgpkeys/secteam-secretary.key[] === {core-secretary-name} `<{core-secretary-email}>` include::static/pgpkeys/core-secretary.key[] === {portmgr-secretary-name} `<{portmgr-secretary-email}>` include::static/pgpkeys/portmgr-secretary.key[] === `{doceng-secretary-email}` include::static/pgpkeys/doceng-secretary.key[] [[pgpkeys-core]] == Core Team Members === `{bapt}` include::static/pgpkeys/bapt.key[] === `{emaste}` include::static/pgpkeys/emaste.key[] === `{gnn}` include::static/pgpkeys/gnn.key[] === `{hrs}` include::static/pgpkeys/hrs.key[] === `{imp}` include::static/pgpkeys/imp.key[] === `{kevans}` include::static/pgpkeys/kevans.key[] === `{markj}` include::static/pgpkeys/markj.key[] === `{scottl}` include::static/pgpkeys/scottl.key[] === `{seanc}` include::static/pgpkeys/seanc.key[] [[pgpkeys-developers]] == Developers === `{ariff}` include::static/pgpkeys/ariff.key[] === `{tabthorpe}` include::static/pgpkeys/tabthorpe.key[] === `{eadler}` include::static/pgpkeys/eadler.key[] === `{mahrens}` include::static/pgpkeys/mahrens.key[] === `{shaun}` include::static/pgpkeys/shaun.key[] === `{brix}` include::static/pgpkeys/brix.key[] === `{mandree}` include::static/pgpkeys/mandree.key[] === `{will}` include::static/pgpkeys/will.key[] === `{dim}` include::static/pgpkeys/dim.key[] === `{anholt}` include::static/pgpkeys/anholt.key[] === `{fernape}` include::static/pgpkeys/fernape.key[] === `{mva}` include::static/pgpkeys/mva.key[] === `{araujo}` include::static/pgpkeys/araujo.key[] === `{mat}` include::static/pgpkeys/mat.key[] === `{syuu}` include::static/pgpkeys/syuu.key[] === `{asami}` include::static/pgpkeys/asami.key[] === `{gavin}` include::static/pgpkeys/gavin.key[] === `{jsa}` include::static/pgpkeys/jsa.key[] === `{jadawin}` include::static/pgpkeys/jadawin.key[] === `{jwb}` include::static/pgpkeys/jwb.key[] === `{badger}` include::static/pgpkeys/badger.key[] === `{dbaio}` include::static/pgpkeys/dbaio.key[] === `{timur}` include::static/pgpkeys/timur.key[] === `{jhb}` include::static/pgpkeys/jhb.key[] === `{gjb}` include::static/pgpkeys/gjb.key[] === `{snb}` include::static/pgpkeys/snb.key[] === `{barner}` include::static/pgpkeys/barner.key[] === `{lbartoletti}` include::static/pgpkeys/lbartoletti.key[] === `{jbeich}` include::static/pgpkeys/jbeich.key[] === `{art}` include::static/pgpkeys/art.key[] === `{tobez}` include::static/pgpkeys/tobez.key[] === `{damien}` include::static/pgpkeys/damien.key[] === `{bdragon}` include::static/pgpkeys/bdragon.key[] === `{tcberner}` include::static/pgpkeys/tcberner.key[] === `{tdb}` include::static/pgpkeys/tdb.key[] === `{gblach}` include::static/pgpkeys/gblach.key[] === `{mbr}` include::static/pgpkeys/mbr.key[] === `{wblock}` include::static/pgpkeys/wblock.key[] === `{bvs}` include::static/pgpkeys/bvs.key[] === `{zbb}` include::static/pgpkeys/zbb.key[] === `{novel}` include::static/pgpkeys/novel.key[] === `{garga}` include::static/pgpkeys/garga.key[] === `{kbowling}` include::static/pgpkeys/kbowling.key[] === `{alexbl}` include::static/pgpkeys/alexbl.key[] === `{sbz}` include::static/pgpkeys/sbz.key[] === `{ebrandi}` include::static/pgpkeys/ebrandi.key[] === `{dab}` include::static/pgpkeys/dab.key[] === `{harti}` include::static/pgpkeys/harti.key[] === `{obraun}` include::static/pgpkeys/obraun.key[] === `{makc}` include::static/pgpkeys/makc.key[] === `{jmb}` include::static/pgpkeys/jmb.key[] === `{antoine}` include::static/pgpkeys/antoine.key[] === `{db}` include::static/pgpkeys/db.key[] === `{brueffer}` include::static/pgpkeys/brueffer.key[] === `{markus}` include::static/pgpkeys/markus.key[] === `{sbruno}` include::static/pgpkeys/sbruno.key[] === `{br}` include::static/pgpkeys/br.key[] === `{oleg}` include::static/pgpkeys/oleg.key[] === `{bushman}` include::static/pgpkeys/bushman.key[] === `{adrian}` include::static/pgpkeys/adrian.key[] === `{jch}` include::static/pgpkeys/jch.key[] === `{jchandra}` include::static/pgpkeys/jchandra.key[] === `{jcamou}` include::static/pgpkeys/jcamou.key[] === `{acm}` include::static/pgpkeys/acm.key[] === `{gahr}` include::static/pgpkeys/gahr.key[] === `{dchagin}` include::static/pgpkeys/dchagin.key[] === `{perky}` include::static/pgpkeys/perky.key[] === `{jon}` include::static/pgpkeys/jon.key[] === `{jonathan}` include::static/pgpkeys/jonathan.key[] === `{loader}` include::static/pgpkeys/loader.key[] === `{luoqi}` include::static/pgpkeys/luoqi.key[] === `{ache}` include::static/pgpkeys/ache.key[] === `{melifaro}` include::static/pgpkeys/melifaro.key[] === `{seanc}` include::static/pgpkeys/seanc.key[] === `{cjh}` include::static/pgpkeys/cjh.key[] === `{davidch}` include::static/pgpkeys/davidch.key[] === `{milki}` include::static/pgpkeys/milki.key[] === `{cjc}` include::static/pgpkeys/cjc.key[] === `{marcus}` include::static/pgpkeys/marcus.key[] === `{nik}` include::static/pgpkeys/nik.key[] === `{benjsc}` include::static/pgpkeys/benjsc.key[] === `{lcook}` include::static/pgpkeys/lcook.key[] === `{ngie}` include::static/pgpkeys/ngie.key[] === `{tijl}` include::static/pgpkeys/tijl.key[] === `{rakuco}` include::static/pgpkeys/rakuco.key[] === `{dch}` include::static/pgpkeys/dch.key[] === `{alc}` include::static/pgpkeys/alc.key[] === `{olivier}` include::static/pgpkeys/olivier.key[] === `{jeb}` include::static/pgpkeys/jeb.key[] === `{bcran}` include::static/pgpkeys/bcran.key[] === `{culot}` include::static/pgpkeys/culot.key[] === `{aaron}` include::static/pgpkeys/aaron.key[] === `{alfredo}` include::static/pgpkeys/alfredo.key[] === `{bapt}` include::static/pgpkeys/bapt.key[] === `{ceri}` include::static/pgpkeys/ceri.key[] === `{brd}` include::static/pgpkeys/brd.key[] === `{edavis}` include::static/pgpkeys/edavis.key[] === `{pjd}` include::static/pgpkeys/pjd.key[] === `{alexey}` include::static/pgpkeys/alexey.key[] === `{bsd}` include::static/pgpkeys/bsd.key[] === `{carl}` include::static/pgpkeys/carl.key[] === `{carlavilla}` include::static/pgpkeys/carlavilla.key[] === `{jmd}` include::static/pgpkeys/jmd.key[] === `{vd}` include::static/pgpkeys/vd.key[] === `{rdivacky}` include::static/pgpkeys/rdivacky.key[] === `{danfe}` include::static/pgpkeys/danfe.key[] === `{dd}` include::static/pgpkeys/dd.key[] === `{bdrewery}` include::static/pgpkeys/bdrewery.key[] === `{gad}` include::static/pgpkeys/gad.key[] === `{olivierd}` include::static/pgpkeys/olivierd.key[] === `{bruno}` include::static/pgpkeys/bruno.key[] === `{ale}` include::static/pgpkeys/ale.key[] === `{nemysis}` include::static/pgpkeys/nemysis.key[] === `{peadar}` include::static/pgpkeys/peadar.key[] === `{deischen}` include::static/pgpkeys/deischen.key[] === `{josef}` include::static/pgpkeys/josef.key[] === `{lme}` include::static/pgpkeys/lme.key[] === `{ue}` include::static/pgpkeys/ue.key[] === `{ru}` include::static/pgpkeys/ru.key[] === `{le}` include::static/pgpkeys/le.key[] === `{se}` include::static/pgpkeys/se.key[] === `{kevans}` include::static/pgpkeys/kevans.key[] === `{bf}` include::static/pgpkeys/bf.key[] === `{sef}` include::static/pgpkeys/sef.key[] === `{madpilot}` include::static/pgpkeys/madpilot.key[] === `{rafan}` include::static/pgpkeys/rafan.key[] === `{kami}` include::static/pgpkeys/kami.key[] === `{stefanf}` include::static/pgpkeys/stefanf.key[] === `{farrokhi}` include::static/pgpkeys/farrokhi.key[] === `{jedgar}` include::static/pgpkeys/jedgar.key[] === `{mfechner}` include::static/pgpkeys/mfechner.key[] === `{feld}` include::static/pgpkeys/feld.key[] === `{green}` include::static/pgpkeys/green.key[] === `{lioux}` include::static/pgpkeys/lioux.key[] === `{mdf}` include::static/pgpkeys/mdf.key[] === `{fanf}` include::static/pgpkeys/fanf.key[] === `{blackend}` include::static/pgpkeys/blackend.key[] === `{petef}` include::static/pgpkeys/petef.key[] === `{decke}` include::static/pgpkeys/decke.key[] === `{landonf}` include::static/pgpkeys/landonf.key[] === `{billf}` include::static/pgpkeys/billf.key[] === `{sg}` include::static/pgpkeys/sg.key[] === `{sgalabov}` include::static/pgpkeys/sgalabov.key[] === `{ultima}` include::static/pgpkeys/ultima.key[] === `{avg}` include::static/pgpkeys/avg.key[] === `{beat}` include::static/pgpkeys/beat.key[] === `{danger}` include::static/pgpkeys/danger.key[] === `{sjg}` include::static/pgpkeys/sjg.key[] === `{gibbs}` include::static/pgpkeys/gibbs.key[] === `{pfg}` include::static/pgpkeys/pfg.key[] === `{girgen}` include::static/pgpkeys/girgen.key[] === `{eugen}` include::static/pgpkeys/eugen.key[] === `{pgollucci}` include::static/pgpkeys/pgollucci.key[] === `{trociny}` include::static/pgpkeys/trociny.key[] === `{danilo}` include::static/pgpkeys/danilo.key[] === `{dmgk}` include::static/pgpkeys/dmgk.key[] === `{daichi}` include::static/pgpkeys/daichi.key[] === `{mnag}` include::static/pgpkeys/mnag.key[] === `{grehan}` include::static/pgpkeys/grehan.key[] === `{jamie}` include::static/pgpkeys/jamie.key[] === `{adridg}` include::static/pgpkeys/adridg.key[] === `{edwin}` include::static/pgpkeys/edwin.key[] === `{wg}` include::static/pgpkeys/wg.key[] === `{bar}` include::static/pgpkeys/bar.key[] === `{anish}` include::static/pgpkeys/anish.key[] === `{jmg}` include::static/pgpkeys/jmg.key[] === `{mjg}` include::static/pgpkeys/mjg.key[] === `{jhale}` include::static/pgpkeys/jhale.key[] === `{jah}` include::static/pgpkeys/jah.key[] === `{dannyboy}` include::static/pgpkeys/dannyboy.key[] === `{dhartmei}` include::static/pgpkeys/dhartmei.key[] === `{ohauer}` include::static/pgpkeys/ohauer.key[] === `{ehaupt}` include::static/pgpkeys/ehaupt.key[] === `{jhay}` include::static/pgpkeys/jhay.key[] === `{bhd}` include::static/pgpkeys/bhd.key[] === `{sheldonh}` include::static/pgpkeys/sheldonh.key[] === `{mikeh}` include::static/pgpkeys/mikeh.key[] === `{mheinen}` include::static/pgpkeys/mheinen.key[] === `{niels}` include::static/pgpkeys/niels.key[] === `{jh}` include::static/pgpkeys/jh.key[] === `{jgh}` include::static/pgpkeys/jgh.key[] === `{ghelmer}` include::static/pgpkeys/ghelmer.key[] === `{mux}` include::static/pgpkeys/mux.key[] === `{wen}` include::static/pgpkeys/wen.key[] === `{dhn}` include::static/pgpkeys/dhn.key[] === `{jhibbits}` include::static/pgpkeys/jhibbits.key[] === `{jhixson}` include::static/pgpkeys/jhixson.key[] === `{pho}` include::static/pgpkeys/pho.key[] === `{oh}` include::static/pgpkeys/oh.key[] === `{mhorne}` include::static/pgpkeys/mhorne.key[] === `{bhughes}` include::static/pgpkeys/bhughes.key[] === `{mich}` include::static/pgpkeys/mich.key[] === `{sunpoet}` include::static/pgpkeys/sunpoet.key[] === `{lwhsu}` include::static/pgpkeys/lwhsu.key[] === `{foxfair}` include::static/pgpkeys/foxfair.key[] === `{whu}` include::static/pgpkeys/whu.key[] === `{chinsan}` include::static/pgpkeys/chinsan.key[] === `{shurd}` include::static/pgpkeys/shurd.key[] === `{kibab}` include::static/pgpkeys/kibab.key[] === `{davide}` include::static/pgpkeys/davide.key[] === `{jkh}` include::static/pgpkeys/jkh.key[] === `{sevan}` include::static/pgpkeys/sevan.key[] === `{versus}` include::static/pgpkeys/versus.key[] === `{pi}` include::static/pgpkeys/pi.key[] === `{weongyo}` include::static/pgpkeys/weongyo.key[] === `{peterj}` include::static/pgpkeys/peterj.key[] === `{jinmei}` include::static/pgpkeys/jinmei.key[] === `{ahze}` include::static/pgpkeys/ahze.key[] === `{markj}` include::static/pgpkeys/markj.key[] === `{trevor}` include::static/pgpkeys/trevor.key[] === `{thj}` include::static/pgpkeys/thj.key[] === `{mjoras}` include::static/pgpkeys/mjoras.key[] === `{erj}` include::static/pgpkeys/erj.key[] === `{allanjude}` include::static/pgpkeys/allanjude.key[] === `{tj}` include::static/pgpkeys/tj.key[] === `{kan}` include::static/pgpkeys/kan.key[] === `{bjk}` include::static/pgpkeys/bjk.key[] === `{phk}` include::static/pgpkeys/phk.key[] === `{pluknet}` include::static/pgpkeys/pluknet.key[] === `{cokane}` include::static/pgpkeys/cokane.key[] === `{karels}` include::static/pgpkeys/karels.key[] === `{kato}` include::static/pgpkeys/kato.key[] === `{joe}` include::static/pgpkeys/joe.key[] === `{vkashyap}` include::static/pgpkeys/vkashyap.key[] === `{pkelsey}` include::static/pgpkeys/pkelsey.key[] === `{pkubaj}` include::static/pgpkeys/pkubaj.key[] === `{kris}` include::static/pgpkeys/kris.key[] === `{keramida}` include::static/pgpkeys/keramida.key[] === `{fjoe}` include::static/pgpkeys/fjoe.key[] === `{manolis}` include::static/pgpkeys/manolis.key[] === `{stevek}` include::static/pgpkeys/stevek.key[] === `{jkim}` include::static/pgpkeys/jkim.key[] === `{zack}` include::static/pgpkeys/zack.key[] === `{jceel}` include::static/pgpkeys/jceel.key[] === `{andreas}` include::static/pgpkeys/andreas.key[] === `{kai}` include::static/pgpkeys/kai.key[] === `{jkois}` include::static/pgpkeys/jkois.key[] === `{sergei}` include::static/pgpkeys/sergei.key[] === `{wulf}` include::static/pgpkeys/wulf.key[] === `{maxim}` include::static/pgpkeys/maxim.key[] === `{taras}` include::static/pgpkeys/taras.key[] === `{tobik}` include::static/pgpkeys/tobik.key[] === `{jkoshy}` include::static/pgpkeys/jkoshy.key[] === `{wkoszek}` include::static/pgpkeys/wkoszek.key[] === `{ak}` include::static/pgpkeys/ak.key[] === `{skra}` include::static/pgpkeys/skra.key[] === `{skreuzer}` include::static/pgpkeys/skreuzer.key[] === `{gabor}` include::static/pgpkeys/gabor.key[] === `{anchie}` include::static/pgpkeys/anchie.key[] === `{rik}` include::static/pgpkeys/rik.key[] === `{rushani}` include::static/pgpkeys/rushani.key[] === `{kuriyama}` include::static/pgpkeys/kuriyama.key[] === `{gleb}` include::static/pgpkeys/gleb.key[] === `{rene}` include::static/pgpkeys/rene.key[] === `{jlaffaye}` include::static/pgpkeys/jlaffaye.key[] === `{clement}` include::static/pgpkeys/clement.key[] === `{mlaier}` include::static/pgpkeys/mlaier.key[] === `{dvl}` include::static/pgpkeys/dvl.key[] === `{erwin}` include::static/pgpkeys/erwin.key[] === `{martymac}` include::static/pgpkeys/martymac.key[] === `{glarkin}` include::static/pgpkeys/glarkin.key[] === `{laszlof}` include::static/pgpkeys/laszlof.key[] === `{dru}` include::static/pgpkeys/dru.key[] === `{lawrance}` include::static/pgpkeys/lawrance.key[] === `{njl}` include::static/pgpkeys/njl.key[] === `{jlh}` include::static/pgpkeys/jlh.key[] === `{leeym}` include::static/pgpkeys/leeym.key[] === `{sam}` include::static/pgpkeys/sam.key[] === `{jylefort}` include::static/pgpkeys/jylefort.key[] === `{grog}` include::static/pgpkeys/grog.key[] === `{oliver}` include::static/pgpkeys/oliver.key[] === `{netchild}` include::static/pgpkeys/netchild.key[] === `{leitao}` include::static/pgpkeys/leitao.key[] === `{ae}` include::static/pgpkeys/ae.key[] === `{lesi}` include::static/pgpkeys/lesi.key[] === `{achim}` include::static/pgpkeys/achim.key[] === `{cel}` include::static/pgpkeys/cel.key[] === `{truckman}` include::static/pgpkeys/truckman.key[] === `{glewis}` include::static/pgpkeys/glewis.key[] === `{vishwin}` include::static/pgpkeys/vishwin.key[] === `{qingli}` include::static/pgpkeys/qingli.key[] === `{delphij}` include::static/pgpkeys/delphij.key[] === `{avatar}` include::static/pgpkeys/avatar.key[] === `{ijliao}` include::static/pgpkeys/ijliao.key[] === `{rlibby}` include::static/pgpkeys/rlibby.key[] === `{lidl}` include::static/pgpkeys/lidl.key[] === `{lifanov}` include::static/pgpkeys/lifanov.key[] === `{lulf}` include::static/pgpkeys/lulf.key[] === `{clive}` include::static/pgpkeys/clive.key[] === `{pclin}` include::static/pgpkeys/pclin.key[] === `{yzlin}` include::static/pgpkeys/yzlin.key[] === `{linimon}` include::static/pgpkeys/linimon.key[] === `{arved}` include::static/pgpkeys/arved.key[] === `{dryice}` include::static/pgpkeys/dryice.key[] === `{nemoliu}` include::static/pgpkeys/nemoliu.key[] === `{kevlo}` include::static/pgpkeys/kevlo.key[] === `{zml}` include::static/pgpkeys/zml.key[] === `{nox}` include::static/pgpkeys/nox.key[] === `{remko}` include::static/pgpkeys/remko.key[] === `{avl}` include::static/pgpkeys/avl.key[] === `{issyl0}` include::static/pgpkeys/issyl0.key[] === `{scottl}` include::static/pgpkeys/scottl.key[] === `{jtl}` include::static/pgpkeys/jtl.key[] === `{luporl}` include::static/pgpkeys/luporl.key[] === `{wma}` include::static/pgpkeys/wma.key[] === `{rmacklem}` include::static/pgpkeys/rmacklem.key[] === `{vmaffione}` include::static/pgpkeys/vmaffione.key[] === `{bmah}` include::static/pgpkeys/bmah.key[] === `{rm}` include::static/pgpkeys/rm.key[] === `{mtm}` include::static/pgpkeys/mtm.key[] === `{dwmalone}` include::static/pgpkeys/dwmalone.key[] === `{amdmi3}` include::static/pgpkeys/amdmi3.key[] === `{marino}` include::static/pgpkeys/marino.key[] === `{kwm}` include::static/pgpkeys/kwm.key[] === `{emaste}` include::static/pgpkeys/emaste.key[] === `{cherry}` include::static/pgpkeys/cherry.key[] === `{matusita}` include::static/pgpkeys/matusita.key[] === `{mm}` include::static/pgpkeys/mm.key[] === `{sem}` include::static/pgpkeys/sem.key[] === `{slm}` include::static/pgpkeys/slm.key[] === `{mckay}` include::static/pgpkeys/mckay.key[] === `{mckusick}` include::static/pgpkeys/mckusick.key[] === `{tmclaugh}` include::static/pgpkeys/tmclaugh.key[] === `{jmcneill}` include::static/pgpkeys/jmcneill.key[] === `{xmj}` include::static/pgpkeys/xmj.key[] === `{jmelo}` include::static/pgpkeys/jmelo.key[] === `{mmel}` include::static/pgpkeys/mmel.key[] === `{jmmv}` include::static/pgpkeys/jmmv.key[] === `{kadesai}` include::static/pgpkeys/kadesai.key[] === `{ken}` include::static/pgpkeys/ken.key[] === `{markm}` include::static/pgpkeys/markm.key[] === `{dinoex}` include::static/pgpkeys/dinoex.key[] === `{sanpei}` include::static/pgpkeys/sanpei.key[] === `{rmh}` include::static/pgpkeys/rmh.key[] === `{jrm}` include::static/pgpkeys/jrm.key[] === `{freqlabs}` include::static/pgpkeys/freqlabs.key[] === `{mmokhi}` include::static/pgpkeys/mmokhi.key[] === `{mmoll}` include::static/pgpkeys/mmoll.key[] === `{cmt}` include::static/pgpkeys/cmt.key[] === `{stephen}` include::static/pgpkeys/stephen.key[] === `{marcel}` include::static/pgpkeys/marcel.key[] === `{dougm}` include::static/pgpkeys/dougm.key[] === `{kmoore}` include::static/pgpkeys/kmoore.key[] === `{marck}` include::static/pgpkeys/marck.key[] === `{mav}` include::static/pgpkeys/mav.key[] === `{lippe}` include::static/pgpkeys/lippe.key[] === `{rich}` include::static/pgpkeys/rich.key[] === `{knu}` include::static/pgpkeys/knu.key[] === `{tmm}` include::static/pgpkeys/tmm.key[] === `{jsm}` include::static/pgpkeys/jsm.key[] === `{max}` include::static/pgpkeys/max.key[] === `{maho}` include::static/pgpkeys/maho.key[] === `{yoichi}` include::static/pgpkeys/yoichi.key[] === `{trasz}` include::static/pgpkeys/trasz.key[] === `{neel}` include::static/pgpkeys/neel.key[] === `{dbn}` include::static/pgpkeys/dbn.key[] === `{bland}` include::static/pgpkeys/bland.key[] === `{joneum}` include::static/pgpkeys/joneum.key[] === `{gnn}` include::static/pgpkeys/gnn.key[] === `{khng}` include::static/pgpkeys/khng.key[] === `{simon}` include::static/pgpkeys/simon.key[] === `{rnoland}` include::static/pgpkeys/rnoland.key[] === `{anders}` include::static/pgpkeys/anders.key[] === `{lofi}` include::static/pgpkeys/lofi.key[] === `{obrien}` include::static/pgpkeys/obrien.key[] === `{olgeni}` include::static/pgpkeys/olgeni.key[] === `{phil}` include::static/pgpkeys/phil.key[] === `{philip}` include::static/pgpkeys/philip.key[] === `{jpaetzel}` include::static/pgpkeys/jpaetzel.key[] === `{pgj}` include::static/pgpkeys/pgj.key[] === `{hiren}` include::static/pgpkeys/hiren.key[] === `{hmp}` include::static/pgpkeys/hmp.key[] === `{yuripv}` include::static/pgpkeys/yuripv.key[] === `{fluffy}` include::static/pgpkeys/fluffy.key[] === `{sat}` include::static/pgpkeys/sat.key[] === `{np}` include::static/pgpkeys/np.key[] === `{royger}` include::static/pgpkeys/royger.key[] === `{rpaulo}` include::static/pgpkeys/rpaulo.key[] === `{misha}` include::static/pgpkeys/misha.key[] === `{dumbbell}` include::static/pgpkeys/dumbbell.key[] === `{mp}` include::static/pgpkeys/mp.key[] === `{roam}` include::static/pgpkeys/roam.key[] === `{den}` include::static/pgpkeys/den.key[] === `{csjp}` include::static/pgpkeys/csjp.key[] === `{gerald}` include::static/pgpkeys/gerald.key[] === `{scottph}` include::static/pgpkeys/scottph.key[] === `{jacula}` include::static/pgpkeys/jacula.key[] === `{0mp}` include::static/pgpkeys/0mp.key[] === `{pizzamig}` include::static/pgpkeys/pizzamig.key[] === `{rpokala}` include::static/pgpkeys/rpokala.key[] === `{jdp}` include::static/pgpkeys/jdp.key[] === `{krion}` include::static/pgpkeys/krion.key[] === `{sepotvin}` include::static/pgpkeys/sepotvin.key[] === `{cpm}` include::static/pgpkeys/cpm.key[] === `{markp}` include::static/pgpkeys/markp.key[] === `{alepulver}` include::static/pgpkeys/alepulver.key[] === `{kp}` include::static/pgpkeys/kp.key[] === `{thomas}` include::static/pgpkeys/thomas.key[] === `{hq}` include::static/pgpkeys/hq.key[] === `{dfr}` include::static/pgpkeys/dfr.key[] === `{bofh}` include::static/pgpkeys/bofh.key[] === `{fox}` include::static/pgpkeys/fox.key[] === `{lbr}` include::static/pgpkeys/lbr.key[] === `{crees}` include::static/pgpkeys/crees.key[] === `{rees}` include::static/pgpkeys/rees.key[] === `{mr}` include::static/pgpkeys/mr.key[] === `{bcr}` include::static/pgpkeys/bcr.key[] === `{rezny}` include::static/pgpkeys/rezny.key[] === `{trhodes}` include::static/pgpkeys/trhodes.key[] === `{benno}` include::static/pgpkeys/benno.key[] === `{arichardson}` include::static/pgpkeys/arichardson.key[] === `{beech}` include::static/pgpkeys/beech.key[] === `{matteo}` include::static/pgpkeys/matteo.key[] === `{roberto}` include::static/pgpkeys/roberto.key[] === `{rodrigc}` include::static/pgpkeys/rodrigc.key[] === `{ler}` include::static/pgpkeys/ler.key[] === `{leres}` include::static/pgpkeys/leres.key[] === `{robak}` include::static/pgpkeys/robak.key[] === `{guido}` include::static/pgpkeys/guido.key[] === `{rea}` include::static/pgpkeys/rea.key[] === `{ray}` include::static/pgpkeys/ray.key[] === `{arybchik}` include::static/pgpkeys/arybchik.key[] === `{niklas}` include::static/pgpkeys/niklas.key[] === `{salvadore}` include::static/pgpkeys/salvadore.key[] === `{bsam}` include::static/pgpkeys/bsam.key[] === `{marks}` include::static/pgpkeys/marks.key[] === `{alonso}` include::static/pgpkeys/alonso.key[] === `{bschmidt}` include::static/pgpkeys/bschmidt.key[] === `{wosch}` include::static/pgpkeys/wosch.key[] === `{ed}` include::static/pgpkeys/ed.key[] === `{cy}` include::static/pgpkeys/cy.key[] === `{das}` include::static/pgpkeys/das.key[] === `{scheidell}` include::static/pgpkeys/scheidell.key[] === `{schweikh}` include::static/pgpkeys/schweikh.key[] === `{matthew}` include::static/pgpkeys/matthew.key[] === `{tmseck}` include::static/pgpkeys/tmseck.key[] === `{stas}` include::static/pgpkeys/stas.key[] === `{johalun}` include::static/pgpkeys/johalun.key[] === `{johans}` include::static/pgpkeys/johans.key[] === `{lev}` include::static/pgpkeys/lev.key[] === `{bakul}` include::static/pgpkeys/bakul.key[] === `{gshapiro}` include::static/pgpkeys/gshapiro.key[] === `{arun}` include::static/pgpkeys/arun.key[] === `{wxs}` include::static/pgpkeys/wxs.key[] === `{nork}` include::static/pgpkeys/nork.key[] === `{syrinx}` include::static/pgpkeys/syrinx.key[] === `{vanilla}` include::static/pgpkeys/vanilla.key[] === `{ashish}` include::static/pgpkeys/ashish.key[] === `{chs}` include::static/pgpkeys/chs.key[] === `{bms}` include::static/pgpkeys/bms.key[] === `{demon}` include::static/pgpkeys/demon.key[] === `{jesper}` include::static/pgpkeys/jesper.key[] === `{scop}` include::static/pgpkeys/scop.key[] === `{anray}` include::static/pgpkeys/anray.key[] === `{flo}` include::static/pgpkeys/flo.key[] === `{glebius}` include::static/pgpkeys/glebius.key[] === `{kensmith}` include::static/pgpkeys/kensmith.key[] === `{ben}` include::static/pgpkeys/ben.key[] === `{des}` include::static/pgpkeys/des.key[] === `{sobomax}` include::static/pgpkeys/sobomax.key[] === `{asomers}` include::static/pgpkeys/asomers.key[] === `{brian}` include::static/pgpkeys/brian.key[] === `{sson}` include::static/pgpkeys/sson.key[] === `{nsouch}` include::static/pgpkeys/nsouch.key[] === `{ssouhlal}` include::static/pgpkeys/ssouhlal.key[] === `{tsoome}` include::static/pgpkeys/tsoome.key[] === `{loos}` include::static/pgpkeys/loos.key[] === `{brnrd}` include::static/pgpkeys/brnrd.key[] === `{uqs}` include::static/pgpkeys/uqs.key[] === `{rink}` include::static/pgpkeys/rink.key[] === `{vsevolod}` include::static/pgpkeys/vsevolod.key[] === `{pstef}` include::static/pgpkeys/pstef.key[] === `{zi}` include::static/pgpkeys/zi.key[] === `{lstewart}` include::static/pgpkeys/lstewart.key[] === `{rrs}` include::static/pgpkeys/rrs.key[] === `{murray}` include::static/pgpkeys/murray.key[] === `{vs}` include::static/pgpkeys/vs.key[] === `{rstone}` include::static/pgpkeys/rstone.key[] === `{xride}` include::static/pgpkeys/xride.key[] === `{marius}` include::static/pgpkeys/marius.key[] === `{cs}` include::static/pgpkeys/cs.key[] === `{clsung}` include::static/pgpkeys/clsung.key[] === `{gsutter}` include::static/pgpkeys/gsutter.key[] === `{metal}` include::static/pgpkeys/metal.key[] === `{ryusuke}` include::static/pgpkeys/ryusuke.key[] === `{garys}` include::static/pgpkeys/garys.key[] === `{nyan}` include::static/pgpkeys/nyan.key[] === `{sahil}` include::static/pgpkeys/sahil.key[] === `{tota}` include::static/pgpkeys/tota.key[] === `{romain}` include::static/pgpkeys/romain.key[] === `{sylvio}` include::static/pgpkeys/sylvio.key[] === `{dteske}` include::static/pgpkeys/dteske.key[] === `{itetcu}` include::static/pgpkeys/itetcu.key[] === `{mi}` include::static/pgpkeys/mi.key[] === `{gordon}` include::static/pgpkeys/gordon.key[] === `{lth}` include::static/pgpkeys/lth.key[] === `{jase}` include::static/pgpkeys/jase.key[] === `{lx}` include::static/pgpkeys/lx.key[] === `{fabient}` include::static/pgpkeys/fabient.key[] === `{thierry}` include::static/pgpkeys/thierry.key[] === `{thompsa}` include::static/pgpkeys/thompsa.key[] === `{flz}` include::static/pgpkeys/flz.key[] === `{jilles}` include::static/pgpkeys/jilles.key[] === `{ganbold}` include::static/pgpkeys/ganbold.key[] === `{tuexen}` include::static/pgpkeys/tuexen.key[] === `{andrew}` include::static/pgpkeys/andrew.key[] === `{gonzo}` include::static/pgpkeys/gonzo.key[] === `{ume}` include::static/pgpkeys/ume.key[] === `{junovitch}` include::static/pgpkeys/junovitch.key[] === `{ups}` include::static/pgpkeys/ups.key[] === `{fsu}` include::static/pgpkeys/fsu.key[] === `{mikael}` include::static/pgpkeys/mikael.key[] === `{ivadasz}` include::static/pgpkeys/ivadasz.key[] === `{manu}` include::static/pgpkeys/manu.key[] === `{vangyzen}` include::static/pgpkeys/vangyzen.key[] === `{ram}` include::static/pgpkeys/ram.key[] === `{bryanv}` include::static/pgpkeys/bryanv.key[] === `{nectar}` include::static/pgpkeys/nectar.key[] === `{avilla}` include::static/pgpkeys/avilla.key[] === `{nivit}` include::static/pgpkeys/nivit.key[] === `{ivoras}` include::static/pgpkeys/ivoras.key[] === `{avos}` include::static/pgpkeys/avos.key[] === `{stefan}` include::static/pgpkeys/stefan.key[] === `{kaiw}` include::static/pgpkeys/kaiw.key[] === `{adamw}` include::static/pgpkeys/adamw.key[] === `{naddy}` include::static/pgpkeys/naddy.key[] === `{peter}` include::static/pgpkeys/peter.key[] === `{nwhitehorn}` include::static/pgpkeys/nwhitehorn.key[] === `{miwi}` include::static/pgpkeys/miwi.key[] === `{nate}` include::static/pgpkeys/nate.key[] === `{swills}` include::static/pgpkeys/swills.key[] === `{twinterg}` include::static/pgpkeys/twinterg.key[] === `{def}` include::static/pgpkeys/def.key[] === `{mw}` include::static/pgpkeys/mw.key[] === `{wollman}` include::static/pgpkeys/wollman.key[] === `{woodsb02}` include::static/pgpkeys/woodsb02.key[] === `{joerg}` include::static/pgpkeys/joerg.key[] === `{davidxu}` include::static/pgpkeys/davidxu.key[] === `{ygy}` include::static/pgpkeys/ygy.key[] === `{emax}` include::static/pgpkeys/emax.key[] === `{yongari}` include::static/pgpkeys/yongari.key[] === `{rcyu}` include::static/pgpkeys/rcyu.key[] === `{oshogbo}` include::static/pgpkeys/oshogbo.key[] === `{riggs}` include::static/pgpkeys/riggs.key[] === `{egypcio}` include::static/pgpkeys/egypcio.key[] === `{bz}` include::static/pgpkeys/bz.key[] === `{zeising}` include::static/pgpkeys/zeising.key[] === `{phantom}` include::static/pgpkeys/phantom.key[] === `{sephe}` include::static/pgpkeys/sephe.key[] === `{mizhka}` include::static/pgpkeys/mizhka.key[] === `{zont}` include::static/pgpkeys/zont.key[] === `{tz}` include::static/pgpkeys/tz.key[] === `{yuri}` include::static/pgpkeys/yuri.key[] === `{slavash}` include::static/pgpkeys/slavash.key[] === `{arrowd}` include::static/pgpkeys/arrowd.key[] === `{rigoletto}` include::static/pgpkeys/rigoletto.key[] === `{kaktus}` include::static/pgpkeys/kaktus.key[] === `{samm}` include::static/pgpkeys/samm.key[] [[pgpkeys-other]] == Other Cluster Account Holders === `{arundel}` include::static/pgpkeys/arundel.key[] === `{bhaga}` include::static/pgpkeys/bhaga.key[] === `{bk}` include::static/pgpkeys/bk.key[] === `{deb}` include::static/pgpkeys/deb.key[] === `{debdrup}` include::static/pgpkeys/debdrup.key[] === `{dutchdaemon}` include::static/pgpkeys/dutchdaemon.key[] === `{keymaster}` include::static/pgpkeys/keymaster.key[] === `{plosher}` include::static/pgpkeys/plosher.key[] === `{mwlucas}` include::static/pgpkeys/mwlucas.key[] === `{dhw}` include::static/pgpkeys/dhw.key[] === `{eduardo}` include::static/pgpkeys/eduardo.key[] diff --git a/documentation/content/en/articles/port-mentor-guidelines/_index.adoc b/documentation/content/en/articles/port-mentor-guidelines/_index.adoc index aa414b7db9..0d1c2b72b1 100644 --- a/documentation/content/en/articles/port-mentor-guidelines/_index.adoc +++ b/documentation/content/en/articles/port-mentor-guidelines/_index.adoc @@ -1,122 +1,132 @@ --- title: Port Mentor Guidelines organizations: - organization: The FreeBSD Ports Management Team copyright: 2011 Thomas Abthorpe, Chris Rees description: Port Mentor Guidelines for FreeBSD Mentors tags: ["port", "mentor", "mentee", "guidelines", "FreeBSD"] --- = Port Mentor Guidelines :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] ''' toc::[] [[port-mentor.guidelines]] == Guideline for Mentor/Mentee Relationships This section is intended to help demystify the mentoring process, as well as a way to openly promote a constructive discussion to adapt and grow the guidelines. In our lives we have too many rules; we are not a government organization that inflicts regulation, but rather a collective of like minded individuals working toward a common goal, maintaining the quality assurance of the product we call the Ports Tree. [[why.mentor]] === Why Mentor? * For most of us, we were mentored into the Project, so return the favor by offering to mentor somebody else in. * You have an irresistible urge to inflict knowledge on others. * The usual punishment applies because you are sick and tired of committing somebody else's good work! [[mentor.comentor]] === Mentor/Co-Mentor Reasons for a co-mentorship: * Significant timezone differential. Accessible, interactive mentor(s) available via IM is extremely helpful! * Potential language barrier. Yes, FreeBSD is very English oriented, as is most software development, however, having a mentor who can speak a native language can be very useful. * ENOTIME! Until there is a 30 hour day, and an 8 day week, some of us only have so much time to give. Sharing the load with somebody else will make it easier. * A rookie mentor can benefit from the experience of a senior committer/mentor. * Two heads are better than one. Reasons for sole mentorship: * You do not play nicely with others. * You prefer to have a one-on-one relationship. * The reasons for co-mentorship do not apply to you. [[mentor.expectations]] === Expectations We expect mentors to review and test-build all proposed patches, at least for an initial period lasting more than a week or two. We expect that mentors should take responsibility for the actions of their mentee. A mentor should follow up with all commits the mentee makes, both approved and implicit. We expect mentors to make sure their mentees read the link:{porters-handbook}[Porter's Handbook], the link:{pr-guidelines}[PR handling guide], and the link:{committers-guide}[Committer's Guide]. While it is not necessary to memorize all the details, every committer needs to have an overview of these things to be an effective part of the community (and avoid as many rookie mistakes as possible). [[mentees]] === Selecting a Mentee There is no defined rule for what makes a candidate ready; it can be a combination of number of PRs they have submitted, the number of ports maintained, frequency of ports updates and/or level of participation in a particular area of interest like GNOME, KDE, Gecko or others. A candidate should have almost no timeouts, be responsive to requests, and generally helpful in supporting their ports. There must be a history of commitment, as it is widely understood that training a committer requires time and effort. If somebody has been around longer, and spent the time observing how things are done, there is some anticipation of accumulated knowledge. All too often we have seen a maintainer submit a few PRs, show up in IRC and ask when they will be given a commit bit. Being subscribed to, and following the mailing lists is very beneficial. There is no real expectation that submitting posts on the lists will make somebody a committer, but it demonstrates a commitment. Some mails offer insights into the knowledge of a candidate as well how they interact with others. Similarly participating in IRC can give somebody a higher profile. Ask six different committers how many PRs a maintainer should submit prior to being nominated, and you will get six different answers. Ask those same individuals how long somebody should have been participating, same dilemma. How many ports should they have at a minimum? Now we have a bikeshed! Some things are just hard to quantify, a mentor will just have to use their best judgement, and hope that portmgr agrees. [[mentorship.duration]] === Mentorship Duration As the trust level develops and grows, the mentee may be granted "implicit" commit rights. This can include trivial changes to a [.filename]#Makefile#, [.filename]#pkg-descr# etc. Similarly, it may include `PORTVERSION` updates that do not include `plist` changes. Other circumstances may be formulated at the discretion of the Mentor. However, during the period of mentorship, a port version bump that affects dependent ports should be checked by a mentor. Just as we are all varied individuals, each mentee has different learning curves, time commitments, and other influencing factors that will contribute to the time required before they can "fly solo". Empirically, a mentee should be observed for at least 3 months. 90-100 commits is another target that a mentor could use before releasing a mentee. Other factors to consider prior releasing a mentee are the number of mistakes they may have made, QATs received etc. If they are still making rookie mistakes, they still require mentor guidance. [[mentor.comentor.debate]] === Mentor/Co-Mentor Debate When a request gets to portmgr, it usually reads as, "I propose 'foo' for a ports commit bit, I will co-mentor with 'bar'". Proposal received, voted, and carried. The mentor is the primary point of contact or the "first among equals", the co-mentor is the backup. Some reprobate, whose name shall be withheld, made the https://lists.freebsd.org/pipermail/cvs-ports/2007-September/134614.html[first recorded co-mentor commit]. Similar co-mentor commits have also been spotted in the src tree. Does this make it right? Does this make it wrong? It seems to be part of the evolution of how things are done. [[mentee.expectations]] === Expectations We expect mentees to be prepared for constructive criticism from the community. There's still a lot of "lore" that is not written down. Responding well to constructive criticism is what we hope we are selecting for by first reviewing their existing contributions on IRC and mailing lists. We warn mentees that some of the criticism they receive may be less "constructive" than others, (whether through language communication problems, or excessive nit-picking), and that dealing with this gracefully is just part of being in a large community. In case of specific problems with specific people, or any questions, we hope that they will approach a portmgr member on IRC or by email. diff --git a/documentation/content/en/articles/pr-guidelines/_index.adoc b/documentation/content/en/articles/pr-guidelines/_index.adoc index c82135ef00..787caf1a3e 100644 --- a/documentation/content/en/articles/pr-guidelines/_index.adoc +++ b/documentation/content/en/articles/pr-guidelines/_index.adoc @@ -1,497 +1,509 @@ --- title: Problem Report Handling Guidelines authors: - author: Dag-Erling Smørgrav - author: Hiten Pandya description: These guidelines describe recommended handling practices for FreeBSD Problem Reports (PRs). trademarks: ["freebsd", "general"] tags: ["PR", "guideline", "bugs", "maintenance", "BugZilla", "FreeBSD"] --- = Problem Report Handling Guidelines :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract These guidelines describe recommended handling practices for FreeBSD Problem Reports (PRs). Whilst developed for the FreeBSD PR Database Maintenance Team mailto:freebsd-bugbusters@FreeBSD.org[freebsd-bugbusters@FreeBSD.org], these guidelines should be followed by anyone working with FreeBSD PRs. ''' toc::[] [[intro]] == Introduction Bugzilla is an issue management system used by the FreeBSD Project. As accurate tracking of outstanding software defects is important to FreeBSD's quality, the correct use of the software is essential to the forward progress of the Project. Access to Bugzilla is available to the entire FreeBSD community. In order to maintain consistency within the database and provide a consistent user experience, guidelines have been established covering common aspects of bug management such as presenting followup, handling close requests, and so forth. [[pr-lifecycle]] == Problem Report Life-cycle * The Reporter submits a bug report on the website. The bug is in the `Needs Triage` state. * Jane Random BugBuster confirms that the bug report has sufficient information to be reproducible. If not, she goes back and forth with the reporter to obtain the needed information. At this point the bug is set to the `Open` state. * Joe Random Committer takes interest in the PR and assigns it to himself, or Jane Random BugBuster decides that Joe is best suited to handle it and assigns it to him. The bug should be set to the `In Discussion` state. * Joe has a brief exchange with the originator (making sure it all goes into the audit trail) and determines the cause of the problem. * Joe pulls an all-nighter and whips up a patch that he thinks fixes the problem, and submits it in a follow-up, asking the originator to test it. He then sets the PRs state to `Patch Ready`. * A couple of iterations later, both Joe and the originator are satisfied with the patch, and Joe commits it to `-CURRENT` (or directly to `-STABLE` if the problem does not exist in `-CURRENT`), making sure to reference the Problem Report in his commit log (and credit the originator if they submitted all or part of the patch) and, if appropriate, start an MFC countdown. The bug is set to the `Needs MFC` state. * If the patch does not need MFCing, Joe then closes the PR as `Issue Resolved`. [NOTE] ==== Many PRs are submitted with very little information about the problem, and some are either very complex to solve, or just scratch the surface of a larger problem; in these cases, it is very important to obtain all the necessary information needed to solve the problem. If the problem contained within cannot be solved, or has occurred again, it is necessary to re-open the PR. ==== [[pr-states]] == Problem Report State It is important to update the state of a PR when certain actions are taken. The state should accurately reflect the current state of work on the PR. .A small example on when to change PR state [example] ==== When a PR has been worked on and the developer(s) responsible feel comfortable about the fix, they will submit a followup to the PR and change its state to "feedback". At this point, the originator should evaluate the fix in their context and respond indicating whether the defect has indeed been remedied. ==== A Problem Report may be in one of the following states: open:: Initial state; the problem has been pointed out and it needs reviewing. analyzed:: The problem has been reviewed and a solution is being sought. feedback:: Further work requires additional information from the originator or the community; possibly information regarding the proposed solution. patched:: A patch has been committed, but something (MFC, or maybe confirmation from originator) is still pending. suspended:: The problem is not being worked on, due to lack of information or resources. This is a prime candidate for somebody who is looking for a project to take on. If the problem cannot be solved at all, it will be closed, rather than suspended. The documentation project uses suspended for wish-list items that entail a significant amount of work which no one currently has time for. closed:: A problem report is closed when any changes have been integrated, documented, and tested, or when fixing the problem is abandoned. [NOTE] ==== The "patched" state is directly related to feedback, so you may go directly to "closed" state if the originator cannot test the patch, and it works in your own testing. ==== [[pr-types]] == Types of Problem Reports While handling problem reports, either as a developer who has direct access to the Problem Reports database or as a contributor who browses the database and submits followups with patches, comments, suggestions or change requests, you will come across several different types of PRs. * <> * <> * <> * <> * <> The following sections describe what each different type of PRs is used for, when a PR belongs to one of these types, and what treatment each different type receives. [[pr-unassigned]] == Unassigned PRs When PRs arrive, they are initially assigned to a generic (placeholder) assignee. These are always prepended with `freebsd-`. The exact value for this default depends on the category; in most cases, it corresponds to a specific FreeBSD mailing list. Here is the current list, with the most common ones listed first: [[default-assignees-common]] .Default Assignees - most common [cols="1,1,1", options="header"] |=== | Type | Categories | Default Assignee |base system |bin, conf, gnu, kern, misc |freebsd-bugs |architecture-specific |alpha, amd64, arm, i386, ia64, powerpc, sparc64 |freebsd-_arch_ |ports collection |ports |freebsd-ports-bugs |documentation shipped with the system |docs |freebsd-doc |FreeBSD web pages (not including docs) |Website |freebsd-www |=== [[default-assignees-other]] .Default Assignees - other [cols="1,1,1", options="header"] |=== | Type | Categories | Default Assignee |advocacy efforts |advocacy |freebsd-advocacy |Java Virtual Machine(TM) problems |java |freebsd-java |standards compliance |standards |freebsd-standards |threading libraries |threads |freebsd-threads |man:usb[4] subsystem |usb |freebsd-usb |=== Do not be surprised to find that the submitter of the PR has assigned it to the wrong category. If you fix the category, do not forget to fix the assignment as well. (In particular, our submitters seem to have a hard time understanding that just because their problem manifested on an i386 system, that it might be generic to all of FreeBSD, and thus be more appropriate for `kern`. The converse is also true, of course.) Certain PRs may be reassigned away from these generic assignees by anyone. There are several types of assignees: specialized mailing lists; mail aliases (used for certain limited-interest items); and individuals. For assignees which are mailing lists, please use the long form when making the assignment (e.g., `freebsd-foo` instead of `foo`); this will avoid duplicate emails sent to the mailing list. [NOTE] ==== Since the list of individuals who have volunteered to be the default assignee for certain types of PRs changes so often, it is much more suitable for https://wiki.freebsd.org/AssigningPRs[the FreeBSD wiki]. ==== Here is a sample list of such entities; it is probably not complete. [[common-assignees-base]] .Common Assignees - base system [cols="1,1,1,1", options="header"] |=== | Type | Suggested Category | Suggested Assignee | Assignee Type |problem specific to the ARM(R) architecture |arm |freebsd-arm |mailing list |problem specific to the MIPS(R) architecture |kern |freebsd-mips |mailing list |problem specific to the PowerPC(R) architecture |kern |freebsd-ppc |mailing list |problem with Advanced Configuration and Power Management (man:acpi[4]) |kern |freebsd-acpi |mailing list |problem with Asynchronous Transfer Mode (ATM) drivers |kern |freebsd-atm |mailing list |problem with embedded or small-footprint FreeBSD systems (e.g., NanoBSD/PicoBSD/FreeBSD-arm) |kern |freebsd-embedded |mailing list |problem with FireWire(R) drivers |kern |freebsd-firewire |mailing list |problem with the filesystem code |kern |freebsd-fs |mailing list |problem with the man:geom[4] subsystem |kern |freebsd-geom |mailing list |problem with the man:ipfw[4] subsystem |kern |freebsd-ipfw |mailing list |problem with Integrated Services Digital Network (ISDN) drivers |kern |freebsd-isdn |mailing list |man:jail[8] subsystem |kern |freebsd-jail |mailing list |problem with Linux(R) or SVR4 emulation |kern |freebsd-emulation |mailing list |problem with the networking stack |kern |freebsd-net |mailing list |problem with the man:pf[4] subsystem |kern |freebsd-pf |mailing list |problem with the man:scsi[4] subsystem |kern |freebsd-scsi |mailing list |problem with the man:sound[4] subsystem |kern |freebsd-multimedia |mailing list |problems with the man:wlan[4] subsystem and wireless drivers |kern |freebsd-wireless |mailing list |problem with man:sysinstall[8] or man:bsdinstall[8] |bin |freebsd-sysinstall |mailing list |problem with the system startup scripts (man:rc[8]) |kern |freebsd-rc |mailing list |problem with VIMAGE or VNET functionality and related code |kern |freebsd-virtualization |mailing list |problem with Xen emulation |kern |freebsd-xen |mailing list |=== [[common-assignees-ports]] .Common Assignees - Ports Collection [cols="1,1,1,1", options="header"] |=== | Type | Suggested Category | Suggested Assignee | Assignee Type |problem with the ports framework (__not__ with an individual port!) |ports |portmgr |alias |port which is maintained by apache@FreeBSD.org |ports |apache |mailing list |port which is maintained by autotools@FreeBSD.org |ports |autotools |alias |port which is maintained by doceng@FreeBSD.org |ports |doceng |alias |port which is maintained by eclipse@FreeBSD.org |ports |freebsd-eclipse |mailing list |port which is maintained by gecko@FreeBSD.org |ports |gecko |mailing list |port which is maintained by gnome@FreeBSD.org |ports |gnome |mailing list |port which is maintained by hamradio@FreeBSD.org |ports |hamradio |alias |port which is maintained by haskell@FreeBSD.org |ports |haskell |alias |port which is maintained by java@FreeBSD.org |ports |freebsd-java |mailing list |port which is maintained by kde@FreeBSD.org |ports |kde |mailing list |port which is maintained by mono@FreeBSD.org |ports |mono |mailing list |port which is maintained by office@FreeBSD.org |ports |freebsd-office |mailing list |port which is maintained by perl@FreeBSD.org |ports |perl |mailing list |port which is maintained by python@FreeBSD.org |ports |freebsd-python |mailing list |port which is maintained by ruby@FreeBSD.org |ports |freebsd-ruby |mailing list |port which is maintained by secteam@FreeBSD.org |ports |secteam |alias |port which is maintained by vbox@FreeBSD.org |ports |vbox |alias |port which is maintained by x11@FreeBSD.org |ports |freebsd-x11 |mailing list |=== Ports PRs which have a maintainer who is a ports committer may be reassigned by anyone (but note that not every FreeBSD committer is necessarily a ports committer, so you cannot simply go by the email address alone.) For other PRs, please do not reassign them to individuals (other than yourself) unless you are certain that the assignee really wants to track the PR. This will help to avoid the case where no one looks at fixing a particular problem because everyone assumes that the assignee is already working on it. [[common-assignees-other]] .Common Assignees - Other [cols="1,1,1,1", options="header"] |=== | Type | Suggested Category | Suggested Assignee | Assignee Type |problem with PR database |bin |bugmeister |alias |problem with Bugzilla https://bugs.freebsd.org/submit/[web form]. |doc |bugmeister |alias |=== [[pr-assigned]] == Assigned PRs If a PR has the `responsible` field set to the username of a FreeBSD developer, it means that the PR has been handed over to that particular person for further work. Assigned PRs should not be touched by anyone but the assignee or bugmeister. If you have comments, submit a followup. If for some reason you think the PR should change state or be reassigned, send a message to the assignee. If the assignee does not respond within two weeks, unassign the PR and do as you please. [[pr-dups]] == Duplicate PRs If you find more than one PR that describe the same issue, choose the one that contains the largest amount of useful information and close the others, stating clearly the number of the superseding PR. If several PRs contain non-overlapping useful information, submit all the missing information to one in a followup, including references to the others; then close the other PRs (which are now completely superseded). [[pr-stale]] == Stale PRs A PR is considered stale if it has not been modified in more than six months. Apply the following procedure to deal with stale PRs: * If the PR contains sufficient detail, try to reproduce the problem in `-CURRENT` and `-STABLE`. If you succeed, submit a followup detailing your findings and try to find someone to assign it to. Set the state to "analyzed" if appropriate. * If the PR describes an issue which you know is the result of a usage error (incorrect configuration or otherwise), submit a followup explaining what the originator did wrong, then close the PR with the reason "User error" or "Configuration error". * If the PR describes an error which you know has been corrected in both `-CURRENT` and `-STABLE`, close it with a message stating when it was fixed in each branch. * If the PR describes an error which you know has been corrected in `-CURRENT`, but not in `-STABLE`, try to find out when the person who corrected it is planning to MFC it, or try to find someone else (maybe yourself?) to do it. Set the state to "patched" and assign it to whomever will do the MFC. * In other cases, ask the originator to confirm if the problem still exists in newer versions. If the originator does not reply within a month, close the PR with the notation "Feedback timeout". [[pr-misfiled-notpr]] == Non-Bug PRs Developers that come across PRs that look like they should have been posted to {freebsd-bugs} or some other list should close the PR, informing the submitter in a comment why this is not really a PR and where the message should be posted. The email addresses that Bugzilla listens to for incoming PRs have been published as part of the FreeBSD documentation, have been announced and listed on the web-site. This means that spammers found them. Whenever you close one of these PRs, please do the following: * Set the component to `junk` (under `Supporting Services`. * Set Responsible to `nobody@FreeBSD.org`. * Set State to `Issue Resolved`. Setting the category to `junk` makes it obvious that there is no useful content within the PR, and helps to reduce the clutter within the main categories. [[references]] == Further Reading This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete. * link:{problem-reports}[How to Write FreeBSD Problem Reports]-guidelines for PR originators. diff --git a/documentation/content/en/articles/problem-reports/_index.adoc b/documentation/content/en/articles/problem-reports/_index.adoc index c474d7786f..d92d95945e 100644 --- a/documentation/content/en/articles/problem-reports/_index.adoc +++ b/documentation/content/en/articles/problem-reports/_index.adoc @@ -1,272 +1,284 @@ --- title: Writing FreeBSD Problem Reports authors: - author: Dag-Erling Smørgrav - author: Mark Linimon description: How to best formulate and submit a problem report to the FreeBSD Project trademarks: ["freebsd", "ibm", "intel", "sun", "general"] tags: ["formulate", "submit", "FreeBSD", "PR"] --- = Writing FreeBSD Problem Reports :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article describes how to best formulate and submit a problem report to the FreeBSD Project. ''' toc::[] [[pr-intro]] == Introduction One of the most frustrating experiences one can have as a software user is to submit a problem report only to have it summarily closed with a terse and unhelpful explanation like "not a bug" or "bogus PR". Similarly, one of the most frustrating experiences as a software developer is to be flooded with problem reports that are not really problem reports but requests for support, or that contain little or no information about what the problem is and how to reproduce it. This document attempts to describe how to write good problem reports. What, one asks, is a good problem report? Well, to go straight to the bottom line, a good problem report is one that can be analyzed and dealt with swiftly, to the mutual satisfaction of both user and developer. Although the primary focus of this article is on FreeBSD problem reports, most of it should apply quite well to other software projects. Note that this article is organized thematically, not chronologically. Read the entire document before submitting a problem report, rather than treating it as a step-by-step tutorial. [[pr-when]] == When to Submit a Problem Report There are many types of problems, and not all of them should engender a problem report. Of course, nobody is perfect, and there will be times when what seems to be a bug in a program is, in fact, a misunderstanding of the syntax for a command or a typographical error in a configuration file (though that in itself may sometimes be indicative of poor documentation or poor error handling in the application). There are still many cases where submitting a problem report is clearly _not_ the right course of action, and will only serve to frustrate both the submitter and the developers. Conversely, there are cases where it might be appropriate to submit a problem report about something else than a bug-an enhancement or a new feature, for instance. So how does one determine what is a bug and what is not? As a simple rule of thumb, the problem is _not_ a bug if it can be expressed as a question (usually of the form "How do I do X?" or "Where can I find Y?"). It is not always quite so black and white, but the question rule covers a large majority of cases. When looking for an answer, consider posing the question to the {freebsd-questions}. Consider these factors when submitting PRs about ports or other software that is not part of FreeBSD itself: * Please do not submit problem reports that simply state that a newer version of an application is available. Ports maintainers are automatically notified by portscout when a new version of an application becomes available. Actual patches to update a port to the latest version are welcome. * For unmaintained ports (`MAINTAINER` is `ports@FreeBSD.org`), a PR without an included patch is unlikely to get picked up by a committer. To become the maintainer of an unmaintained port, submit a PR with the request (patch preferred but not required). * In either case, following the process described in link:{porters-handbook}#port-upgrading[Porter's Handbook] will yield the best results. (You might also wish to read link:{contributing}#ports-contributing[Contributing to the FreeBSD Ports Collection].) A bug that cannot be reproduced can rarely be fixed. If the bug only occurred once and you cannot reproduce it, and it does not seem to happen to anybody else, chances are none of the developers will be able to reproduce it or figure out what is wrong. That does not mean it did not happen, but it does mean that the chances of your problem report ever leading to a bug fix are very slim. To make matters worse, often these kinds of bugs are actually caused by failing hard drives or overheating processors - you should always try to rule out these causes, whenever possible, before submitting a PR. Next, to decide to whom you should file your problem report, you need to understand that the software that makes up FreeBSD is composed of several different elements: * Code in the base system that is written and maintained by FreeBSD contributors, such as the kernel, the C library, and the device drivers (categorized as `kern`); the binary utilities (`bin`); the manual pages and documentation (`docs`); and the web pages (`www`). All bugs in these areas should be reported to the FreeBSD developers. * Code in the base system that is written and maintained by others, and imported into FreeBSD and adapted. Examples include man:clang[1], and man:sendmail[8]. Most bugs in these areas should be reported to the FreeBSD developers; but in some cases they may need to be reported to the original authors instead if the problems are not FreeBSD-specific. * Individual applications that are not in the base system but are instead part of the FreeBSD Ports Collection (category `ports`). Most of these applications are not written by FreeBSD developers; what FreeBSD provides is merely a framework for installing the application. Therefore, only report a problem to the FreeBSD developers when the problem is believed to be FreeBSD-specific; otherwise, report it to the authors of the software. Then, ascertain whether the problem is timely. There are few things that will annoy a developer more than receiving a problem report about a bug she has already fixed. If the problem is in the base system, first read the FAQ section on link:{faq}#LATEST-VERSION[FreeBSD versions], if you are not already familiar with the topic. It is not possible for FreeBSD to fix problems in anything other than certain recent branches of the base system, so filing a bug report about an older version will probably only result in a developer advising you to upgrade to a supported version to see if the problem still recurs. The Security Officer team maintains the link:https://www.FreeBSD.org/security/[list of supported versions]. If the problem is in a port, consider filing a bug with the upstream. The FreeBSD Project can not fix all bugs in all software. [[pr-prep]] == Preparations A good rule to follow is to always do a background search before submitting a problem report. Maybe the problem has already been reported; maybe it is being discussed on the mailing lists, or recently was; it may even already be fixed in a newer version than what you are running. You should therefore check all the obvious places before submitting your problem report. For FreeBSD, this means: * The FreeBSD link:{faq}[Frequently Asked Questions] (FAQ) list. The FAQ attempts to provide answers for a wide range of questions, such as those concerning link:{faq}#hardware[hardware compatibility], link:{faq}#applications[user applications], and link:{faq}#kernelconfig[kernel configuration]. * The link:{handbook}#eresources-mail[mailing lists]-if you are not subscribed, use https://www.FreeBSD.org/search/#mailinglists[the searchable archives] on the FreeBSD web site. If the problem has not been discussed on the lists, you might try posting a message about it and waiting a few days to see if someone can spot something that has been overlooked. * Optionally, the entire web-use your favorite search engine to locate any references to the problem. You may even get hits from archived mailing lists or newsgroups you did not know of or had not thought to search through. * Next, the searchable https://bugs.freebsd.org/bugzilla/query.cgi[FreeBSD PR database] (Bugzilla). Unless the problem is recent or obscure, there is a fair chance it has already been reported. * Most importantly, attempt to see if existing documentation in the source base addresses your problem. + For the base FreeBSD code, you should carefully study the contents of [.filename]#/usr/src/UPDATING# on your system or the latest version at https://cgit.freebsd.org/src/tree/UPDATING[https://cgit.freebsd.org/src/tree/UPDATING]. (This is vital information if you are upgrading from one version to another-especially if you are upgrading to the FreeBSD-CURRENT branch). + However, if the problem is in something that was installed as a part of the FreeBSD Ports Collection, you should refer to [.filename]#/usr/ports/UPDATING# (for individual ports) or [.filename]#/usr/ports/CHANGES# (for changes that affect the entire Ports Collection). https://cgit.freebsd.org/ports/tree/UPDATING[https://cgit.freebsd.org/ports/tree/UPDATING] and https://cgit.freebsd.org/ports/tree/CHANGES[https://cgit.freebsd.org/ports/tree/CHANGES] are also available via cgit. [[pr-writing]] == Writing the Problem Report Now that you have decided that your issue merits a problem report, and that it is a FreeBSD problem, it is time to write the actual problem report. Before we get into the mechanics of the program used to generate and submit PRs, here are some tips and tricks to help make sure that your PR will be most effective. [[pr-writing-tips]] == Tips and Tricks for Writing a Good Problem Report * _Do not leave the "Summary" line empty._ The PRs go both onto a mailing list that goes all over the world (where the "Summary" is used for the `Subject:` line), but also into a database. Anyone who comes along later and browses the database by synopsis, and finds a PR with a blank subject line, tends just to skip over it. Remember that PRs stay in this database until they are closed by someone; an anonymous one will usually just disappear in the noise. * _Avoid using a weak "Summary" line._ You should not assume that anyone reading your PR has any context for your submission, so the more you provide, the better. For instance, what part of the system does the problem apply to? Do you only see the problem while installing, or while running? To illustrate, instead of `Summary: portupgrade is broken`, see how much more informative this seems: `Summary: port ports-mgmt/portupgrade coredumps on -current`. (In the case of ports, it is especially helpful to have both the category and portname in the "Summary" line.) * _If you have a patch, say so._ A PR with a patch included is much more likely to be looked at than one without. Please set the `patch` Keyword in Bugzilla. * _If you are a maintainer, say so._ If you are maintaining a part of the source code (for instance, an existing port), you definitely should set the "Class" of your PR to `maintainer-update`. This way any committer that handles your PR will not have to check. * _Be specific._ The more information you supply about what problem you are having, the better your chance of getting a response. ** Include the version of FreeBSD you are running (there is a place to put that, see below) and on which architecture. You should include whether you are running from a release (e.g., from a CD-ROM or download), or from a system maintained by Git (and, if so, what hash and branch you are at). If you are tracking the FreeBSD-CURRENT branch, that is the very first thing someone will ask, because fixes (especially for high-profile problems) tend to get committed very quickly, and FreeBSD-CURRENT users are expected to keep up. ** Include which global options you have specified in your [.filename]#make.conf#, [.filename]#src.conf#, and [.filename]#src-env.conf#. Given the infinite number of options, not every combination may be fully supported. ** If the problem can be reproduced easily, include information that will help a developer to reproduce it themselves. If a problem can be demonstrated with specific input then include an example of that input if possible, and include both the actual and the expected output. If this data is large or cannot be made public, then do try to create a minimal file that exhibits the same issue and that can be included within the PR. ** If this is a kernel problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): *** your kernel configuration (including which hardware devices you have installed) *** whether or not you have debugging options enabled (such as `WITNESS`), and if so, whether the problem persists when you change the sense of that option *** the full text of any backtrace, panic or other console output, or entries in [.filename]#/var/log/messages#, if any were generated *** the output of `pciconf -l` and relevant parts of your `dmesg` output if your problem relates to a specific piece of hardware *** the fact that you have read [.filename]#src/UPDATING# and that your problem is not listed there (someone is guaranteed to ask) *** whether or not you can run any other kernel as a fallback (this is to rule out hardware-related issues such as failing disks and overheating CPUs, which can masquerade as kernel problems) ** If this is a ports problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): *** which ports you have installed *** any environment variables that override the defaults in [.filename]#bsd.port.mk#, such as `PORTSDIR` *** the fact that you have read [.filename]#ports/UPDATING# and that your problem is not listed there (someone is guaranteed to ask) * _Avoid vague requests for features._ PRs of the form "someone should really implement something that does so-and-so" are less likely to get results than very specific requests. Remember, the source is available to everyone, so if you want a feature, the best way to ensure it being included is to get to work! Also consider the fact that many things like this would make a better topic for discussion on `freebsd-questions` than an entry in the PR database, as discussed above. * _Make sure no one else has already submitted a similar PR._ Although this has already been mentioned above, it bears repeating here. It only take a minute or two to use the web-based search engine at https://bugs.freebsd.org/bugzilla/query.cgi[https://bugs.freebsd.org/bugzilla/query.cgi]. (Of course, everyone is guilty of forgetting to do this now and then.) * _Report only one issue per Problem Report._ Avoid including two or more problems within the same report unless they are related. When submitting patches, avoid adding multiple features or fixing multiple bugs in the same PR unless they are closely related-such PRs often take longer to resolve. * _Avoid controversial requests._ If your PR addresses an area that has been controversial in the past, you should probably be prepared to not only offer patches, but also justification for why the patches are "The Right Thing To Do". As noted above, a careful search of the mailing lists using the archives at https://www.FreeBSD.org/search/#mailinglists[https://www.FreeBSD.org/search/#mailinglists] is always good preparation. * _Be polite._ Almost anyone who would potentially work on your PR is a volunteer. No one likes to be told that they have to do something when they are already doing it for some motivation other than monetary gain. This is a good thing to keep in mind at all times on Open Source projects. [[pr-writing-before-beginning]] == Before Beginning Similar considerations apply to use of the https://bugs.freebsd.org/bugzilla/enter_bug.cgi[web-based PR submission form]. Be careful of cut-and-paste operations that might change whitespace or other text formatting. Finally, if the submission is lengthy, prepare the work offline so that nothing will be lost if there is a problem submitting it. [[pr-writing-attaching-patches]] == Attaching Patches or Files When attaching a patch, be sure to use either `git diff` or man:diff[1] with the `-u` option to create a unified diff and make sure to specify the Git hash and branch of the repository against which you modified files, so the developers who read your report will be able to apply them easily. For problems with the kernel or the base utilities, a patch against FreeBSD-CURRENT (the main Git branch) is preferred since all new code should be applied and tested there first. After appropriate or substantial testing has been done, the code will be merged/migrated to the FreeBSD-STABLE branch. If you attach a patch inline, instead of as an attachment, note that the most common problem by far is the tendency of some email programs to render tabs as spaces, which will completely ruin anything intended to be part of a Makefile. Do not send patches as attachments using `Content-Transfer-Encoding: quoted-printable`. These will perform character escaping and the entire patch will be useless. Also note that while including small patches in a PR is generally all right-particularly when they fix the problem described in the PR-large patches and especially new code which may require substantial review before committing should be placed on a web or ftp server, and the URL should be included in the PR instead of the patch. Patches in email tend to get mangled, and the larger the patch, the harder it will be for interested parties to unmangle it. Also, posting a patch on the web allows you to modify it without having to resubmit the entire patch in a followup to the original PR. Finally, large patches simply increase the size of the database, since closed PRs are not actually deleted but instead kept and simply marked as complete. You should also take note that unless you explicitly specify otherwise in your PR or in the patch itself, any patches you submit will be assumed to be licensed under the same terms as the original file you modified. [[pr-writing-filling-template]] == Filling out the Form [NOTE] ==== The email address you use will become public information and may become available to spammers. You should either have spam handling procedures in place, or use a temporary email account. However, please note that if you do not use a valid email account at all, we will not be able to ask you questions about your PR. ==== When you file a bug, you will find the following fields: * _Summary:_ Fill this out with a short and accurate description of the problem. The synopsis is used as the subject of the problem report email, and is used in problem report listings and summaries; problem reports with obscure synopses tend to get ignored. * _Severity:_ One of `Affects only me`, `Affects some people` or `Affects many people`. Do not overreact; refrain from labeling your problem `Affects many people` unless it really does. FreeBSD developers will not necessarily work on your problem faster if you inflate its importance since there are so many other people who have done exactly that. * _Category:_ Choose an appropriate category. + The first thing you need to do is to decide what part of the system your problem lies in. Remember, FreeBSD is a complete operating system, which installs both a kernel, the standard libraries, many peripheral drivers, and a large number of utilities (the "base system"). However, there are thousands of additional applications in the Ports Collection. You'll first need to decide if the problem is in the base system or something installed via the Ports Collection. + Here is a description of the major categories: ** If a problem is with the kernel, the libraries (such as standard C library `libc`), or a peripheral driver in the base system, in general you will use the `kern` category. (There are a few exceptions; see below). In general these are things that are described in section 2, 3, or 4 of the manual pages. ** If a problem is with a binary program such as man:sh[1] or man:mount[8], you will first need to determine whether these programs are in the base system or were added via the Ports Collection. If you are unsure, you can do `whereis _programname_`. FreeBSD's convention for the Ports Collection is to install everything underneath [.filename]#/usr/local#, although this can be overridden by a system administrator. For these, you will use the `ports` category (yes, even if the port's category is `www`; see below). If the location is [.filename]#/bin#, [.filename]#/usr/bin#, [.filename]#/sbin#, or [.filename]#/usr/sbin#, it is part of the base system, and you should use the `bin` category. These are all things that are described in section 1 or 8 of the manual pages. ** If you believe that the error is in the startup `(rc)` scripts, or in some kind of other non-executable configuration file, then the right category is `conf` (configuration). These are things that are described in section 5 of the manual pages. ** If you have found a problem in the documentation set (articles, books, man pages) or website the correct choice is `docs`. + [NOTE] ==== if you are having a problem with something from a port named `www/_someportname_`, this nevertheless goes in the `ports` category. ==== + There are a few more specialized categories. ** If the problem would otherwise be filed in `kern` but has to do with the USB subsystem, the correct choice is `usb`. ** If the problem would otherwise be filed in `kern` but has to do with the threading libraries, the correct choice is `threads`. ** If the problem would otherwise be in the base system, but has to do with our adherence to standards such as POSIX(R), the correct choice is `standards`. ** If you are convinced that the problem will only occur under the processor architecture you are using, select one of the architecture-specific categories: commonly `i386` for Intel-compatible machines in 32-bit mode; `amd64` for AMD machines running in 64-bit mode (this also includes Intel-compatible machines running in EMT64 mode); and less commonly `arm` or `powerpc`. + [NOTE] ==== These categories are quite often misused for "I do not know" problems. Rather than guessing, please just use `misc`. ==== + .Correct Use of Arch-Specific Category [example] ==== You have a common PC-based machine, and think you have encountered a problem specific to a particular chipset or a particular motherboard: `i386` is the right category. ==== + .Incorrect Use of Arch-Specific Category [example] ==== You are having a problem with an add-in peripheral card on a commonly seen bus, or a problem with a particular type of hard disk drive: in this case, it probably applies to more than one architecture, and `kern` is the right category. ==== ** If you really do not know where the problem lies (or the explanation does not seem to fit into the ones above), use the `misc` category. Before you do so, you may wish to ask for help on the {freebsd-questions} first. You may be advised that one of the existing categories really is a better choice. * _Environment:_ This should describe, as accurately as possible, the environment in which the problem has been observed. This includes the operating system version, the version of the specific program or file that contains the problem, and any other relevant items such as system configuration, other installed software that influences the problem, etc.-quite simply everything a developer needs to know to reconstruct the environment in which the problem occurs. * __Description:__A complete and accurate description of the problem you are experiencing. Try to avoid speculating about the causes of the problem unless you are certain that you are on the right track, as it may mislead a developer into making incorrect assumptions about the problem. It should include the actions you need to take to reproduce the problem. If you know any workaround, include it. It not only helps other people with the same problem work around it, but may also help a developer understand the cause for the problem. [[pr-followup]] == Follow-up Once the problem report has been filed, you will receive a confirmation by email which will include the tracking number that was assigned to your problem report and a URL you can use to check its status. With a little luck, someone will take an interest in your problem and try to address it, or, as the case may be, explain why it is not a problem. You will be automatically notified of any change of status, and you will receive copies of any comments or patches someone may attach to your problem report's audit trail. If someone requests additional information from you, or you remember or discover something you did not mention in the initial report, please submit a follow up. The number one reason for a bug not getting fixed is lack of communication with the originator. The easiest way is to use the comment option on the individual PR's web page, which you can reach from the https://bugs.freebsd.org/bugzilla/query.cgi[PR search page]. If the problem report remains open after the problem has gone away, just add a comment saying that the problem report can be closed, and, if possible, explaining how or when the problem was fixed. Sometimes there is a delay of a week or two where the problem report remains untouched, not assigned or commented on by anyone. This can happen when there is an increased problem report backlog or during a holiday season. When a problem report has not received attention after several weeks, it is worth finding a committer particularly interested in working on it. There are a few ways to do so, ideally in the following order, with a few days between attempting each communication channel: * Find the relevant FreeBSD mailing list for the problem report from the link:{handbook}#eresources-mail[list in the Handbook] and send a message to that list asking about assistance or comments on the problem report. * Join the relevant IRC channels. A partial listing is here: https://wiki.freebsd.org/IrcChannels[]. Inform the people in that channel about the problem report and ask for assistance. Be patient and stay in the channel after posting, so that the people from different time zones around the world have a chance to catch up. * Find committers interested in the problem that was reported. If the problem was in a particular tool, binary, port, document, or source file, check the https://cgit.FreeBSD.org[Git Repository]. Locate the last few committers who made substantive changes to the file, and try to reach them via IRC or email. A list of committers and their emails can be found in the link:{contributors}[Contributors to FreeBSD] article. Remember that these people are volunteers, just like maintainers and users, so they might not be immediately available to assist with the problem report. Patience and consistency in the follow-ups is highly advised and appreciated. With enough care and effort dedicated to that follow-up process, finding a committer to take care of the problem report is just a matter of time. [[pr-problems]] == If There Are Problems If you found an issue with the bug system, file a bug! There is a category for exactly this purpose. If you are unable to do so, contact the bug wranglers at mailto:bugmeister@FreeBSD.org[bugmeister@FreeBSD.org]. [[pr-further]] == Further Reading This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete. * https://github.com/smileytechguy/reporting-bugs-effectively/blob/master/ENGLISH.md[How to Report Bugs Effectively]-an excellent essay by Simon G. Tatham on composing useful (non-FreeBSD-specific) problem reports. * link:{pr-guidelines}[Problem Report Handling Guidelines]-valuable insight into how problem reports are handled by the FreeBSD developers. diff --git a/documentation/content/en/articles/rc-scripting/_index.adoc b/documentation/content/en/articles/rc-scripting/_index.adoc index 35ee3f3f30..4d12be53b5 100644 --- a/documentation/content/en/articles/rc-scripting/_index.adoc +++ b/documentation/content/en/articles/rc-scripting/_index.adoc @@ -1,804 +1,814 @@ --- title: Practical rc.d scripting in BSD authors: - author: Yar Tikhiy email: yar@FreeBSD.org copyright: 2005-2006, 2012 The FreeBSD Project description: A guide to writing new rc.d scripts and understanding those already written trademarks: ["freebsd", "netbsd", "general"] tags: ["rc.d", "scripting", "guide", "tutorial", "FreeBSD"] --- = Practical rc.d scripting in BSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract Beginners may find it difficult to relate the facts from the formal documentation on the BSD [.filename]#rc.d# framework with the practical tasks of [.filename]#rc.d# scripting. In this article, we consider a few typical cases of increasing complexity, show [.filename]#rc.d# features suited for each case, and discuss how they work. Such an examination should provide reference points for further study of the design and efficient application of [.filename]#rc.d#. ''' toc::[] [[rcng-intro]] == Introduction The historical BSD had a monolithic startup script, [.filename]#/etc/rc#. It was invoked by man:init[8] at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on. The precise list of tasks was not the same in every system; admins needed to customize it. With few exceptions, [.filename]#/etc/rc# had to be modified, and true hackers liked it. The real problem with the monolithic approach was that it provided no control over the individual components started from [.filename]#/etc/rc#. For instance, [.filename]#/etc/rc# could not restart a single daemon. The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through [.filename]#/etc/rc# for the flags, and finally type the full command line to start the daemon again. The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions. In a few words, the single script failed to fulfil what scripts are for: to make the system admin's life easier. Later there was an attempt to split out some parts of [.filename]#/etc/rc# for the sake of starting the most important subsystems separately. The notorious example was [.filename]#/etc/netstart# to bring up networking. It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking. That was why [.filename]#/etc/netstart# mutated into [.filename]#/etc/rc.network#. The latter was no longer an ordinary script; it comprised of large, tangled man:sh[1] functions called from [.filename]#/etc/rc# at different stages of system startup. However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic [.filename]#/etc/rc# had been. Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems. It became obvious at last that more steps are necessary on the way to a fine-grained and extensible [.filename]#rc# system. Thus BSD [.filename]#rc.d# was born. Its acknowledged fathers were Luke Mewburn and the NetBSD community. Later it was imported into FreeBSD. Its name refers to the location of system scripts for individual services, which is in [.filename]#/etc/rc.d#. Soon we will learn about more components of the [.filename]#rc.d# system and see how the individual scripts are invoked. The basic ideas behind BSD [.filename]#rc.d# are _fine modularity_ and __code reuse__. _Fine modularity_ means that each basic "service" such as a system daemon or primitive startup task gets its own man:sh[1] script able to start the service, stop it, reload it, check its status. A particular action is chosen by the command-line argument to the script. The [.filename]#/etc/rc# script still drives system startup, but now it merely invokes the smaller scripts one by one with the `start` argument. It is easy to perform shutdown tasks as well by running the same set of scripts with the `stop` argument, which is done by [.filename]#/etc/rc.shutdown#. Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible. _Code reuse_ means that common operations are implemented as man:sh[1] functions and collected in [.filename]#/etc/rc.subr#. Now a typical script can be just a few lines' worth of man:sh[1] code. Finally, an important part of the [.filename]#rc.d# framework is man:rcorder[8], which helps [.filename]#/etc/rc# to run the small scripts orderly with respect to dependencies between them. It can help [.filename]#/etc/rc.shutdown#, too, because the proper order for the shutdown sequence is opposite to that of startup. The BSD [.filename]#rc.d# design is described in <>, and the [.filename]#rc.d# components are documented in great detail in <>. However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together in order to create a well-styled script for a particular task. Therefore this article will try a different approach to describe [.filename]#rc.d#. It will show which features should be used in a number of typical cases, and why. Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the [.filename]#rc.d# realm. Neither is this article a replacement for the relevant manual pages. Do not hesitate to refer to them for more formal and complete documentation while reading this article. There are prerequisites to understanding this article. First of all, you should be familiar with the man:sh[1] scripting language in order to master [.filename]#rc.d#. In addition, you should know how the system performs userland startup and shutdown tasks, which is described in man:rc[8]. This article focuses on the FreeBSD branch of [.filename]#rc.d#. Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD [.filename]#rc.d# not only share the same design but also stay similar in their aspects visible to script authors. [[rcng-task]] == Outlining the task A little consideration before starting `$EDITOR` will not hurt. In order to write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first: * Is the service mandatory or optional? * Will the script serve a single program, e.g., a daemon, or perform more complex actions? * Which other services will our service depend on, and vice versa? From the examples that follow we will see why it is important to know the answers to these questions. [[rcng-dummy]] == A dummy script The following script just emits a message each time the system boots up: [.programlisting] .... #!/bin/sh <.> . /etc/rc.subr <.> name="dummy" <.> start_cmd="${name}_start" <.> stop_cmd=":" <.> dummy_start() <.> { echo "Nothing started." } load_rc_config $name <.> run_rc_command "$1" <.> .... Things to note are: ➊ An interpreted script should begin with the magic "shebang" line. That line specifies the interpreter program for the script. Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set. (See man:chmod[1].) For example, a system admin can run our script manually, from the command line: [source,shell] .... # /etc/rc.d/dummy start .... [NOTE] ==== In order to be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language. If you have a service or port that uses a binary control utility or a startup routine written in another language, install that element in [.filename]#/usr/sbin# (for the system) or [.filename]#/usr/local/sbin# (for ports) and call it from a man:sh[1] script in the appropriate [.filename]#rc.d# directory. ==== [TIP] ==== If you would like to learn the details of why [.filename]#rc.d# scripts must be written in the man:sh[1] language, see how [.filename]#/etc/rc# invokes them by means of `run_rc_script`, then study the implementation of `run_rc_script` in [.filename]#/etc/rc.subr#. ==== ➋ In [.filename]#/etc/rc.subr#, a number of man:sh[1] functions are defined for an [.filename]#rc.d# script to use. The functions are documented in man:rc.subr[8]. While it is theoretically possible to write an [.filename]#rc.d# script without ever using man:rc.subr[8], its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to man:rc.subr[8] in [.filename]#rc.d# scripts. We are not going to be an exception. An [.filename]#rc.d# script must "source"[.filename]#/etc/rc.subr# (include it using "`.`") _before_ it calls man:rc.subr[8] functions so that man:sh[1] has an opportunity to learn the functions. The preferred style is to source [.filename]#/etc/rc.subr# first of all. [NOTE] ==== Some useful functions related to networking are provided by another include file, [.filename]#/etc/network.subr#. ==== ➌ [[name-var]]The mandatory variable `name` specifies the name of our script. It is required by man:rc.subr[8]. That is, each [.filename]#rc.d# script _must_ set `name` before it calls man:rc.subr[8] functions. Now it is the right time to choose a unique name for our script once and for all. We will use it in a number of places while developing the script. For a start, let us give the same name to the script file, too. [NOTE] ==== The current style of [.filename]#rc.d# scripting is to enclose values assigned to variables in double quotes. Keep in mind that it is just a style issue that may not always be applicable. You can safely omit quotes from around simple words without man:sh[1] metacharacters in them, while in certain cases you will need single quotes to prevent any interpretation of the value by man:sh[1]. A programmer should be able to tell the language syntax from style conventions and use both of them wisely. ==== ➍ The main idea behind man:rc.subr[8] is that an [.filename]#rc.d# script provides handlers, or methods, for man:rc.subr[8] to invoke. In particular, `start`, `stop`, and other arguments to an [.filename]#rc.d# script are handled this way. A method is a man:sh[1] expression stored in a variable named `argument_cmd`, where _argument_ corresponds to what can be specified on the script's command line. We will see later how man:rc.subr[8] provides default methods for the standard arguments. [NOTE] ==== To make the code in [.filename]#rc.d# more uniform, it is common to use `${name}` wherever appropriate. Thus a number of lines can be just copied from one script to another. ==== ➎ We should keep in mind that man:rc.subr[8] provides default methods for the standard arguments. Consequently, we must override a standard method with a no-op man:sh[1] expression if we want it to do nothing. ➏ The body of a sophisticated method can be implemented as a function. It is a good idea to make the function name meaningful. [IMPORTANT] ==== It is strongly recommended to add the prefix `${name}` to the names of all functions defined in our script so they never clash with the functions from man:rc.subr[8] or another common include file. ==== ➐ This call to man:rc.subr[8] loads man:rc.conf[5] variables. Our script makes no use of them yet, but it still is recommended to load man:rc.conf[5] because there can be man:rc.conf[5] variables controlling man:rc.subr[8] itself. ➑ Usually this is the last command in an [.filename]#rc.d# script. It invokes the man:rc.subr[8] machinery to perform the requested action using the variables and methods our script has provided. [[rcng-confdummy]] == A configurable dummy script Now let us add some controls to our dummy script. As you may know, [.filename]#rc.d# scripts are controlled with man:rc.conf[5]. Fortunately, man:rc.subr[8] hides all the complications from us. The following script uses man:rc.conf[5] via man:rc.subr[8] to see whether it is enabled in the first place, and to fetch a message to show at boot time. These two tasks in fact are independent. On the one hand, an [.filename]#rc.d# script can just support enabling and disabling its service. On the other hand, a mandatory [.filename]#rc.d# script can have configuration variables. We will do both things in the same script though: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=dummy rcvar=dummy_enable <.> start_cmd="${name}_start" stop_cmd=":" load_rc_config $name <.> : ${dummy_enable:=no} <.> : ${dummy_msg="Nothing started."} <.> dummy_start() { echo "$dummy_msg" <.> } run_rc_command "$1" .... What changed in this example? ➊ The variable `rcvar` specifies the name of the ON/OFF knob variable. ➋ Now `load_rc_config` is invoked earlier in the script, before any man:rc.conf[5] variables are accessed. [NOTE] ==== While examining [.filename]#rc.d# scripts, keep in mind that man:sh[1] defers the evaluation of expressions in a function until the latter is called. Therefore it is not an error to invoke `load_rc_config` as late as just before `run_rc_command` and still access man:rc.conf[5] variables from the method functions exported to `run_rc_command`. This is because the method functions are to be called by `run_rc_command`, which is invoked _after_ `load_rc_config`. ==== ➌ A warning will be emitted by `run_rc_command` if `rcvar` itself is set, but the indicated knob variable is unset. If your [.filename]#rc.d# script is for the base system, you should add a default setting for the knob to [.filename]#/etc/defaults/rc.conf# and document it in man:rc.conf[5]. Otherwise it is your script that should provide a default setting for the knob. The canonical approach to the latter case is shown in the example. [NOTE] ==== You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective of its current setting, by prefixing the argument to the script with `one` or `force`, as in `onestart` or `forcestop`. Keep in mind though that `force` has other dangerous effects we will touch upon below, while `one` just overrides the ON/OFF knob. E.g., assume that `dummy_enable` is `OFF`. The following command will run the `start` method in spite of the setting: [source,shell] .... # /etc/rc.d/dummy onestart .... ==== ➍ Now the message to be shown at boot time is no longer hard-coded in the script. It is specified by an man:rc.conf[5] variable named `dummy_msg`. This is a trivial example of how man:rc.conf[5] variables can control an [.filename]#rc.d# script. [IMPORTANT] ==== The names of all man:rc.conf[5] variables used exclusively by our script _must_ have the same prefix: `${name}_`. For example: `dummy_mode`, `dummy_state_file`, and so on. ==== [NOTE] ==== While it is possible to use a shorter name internally, e.g., just `msg`, adding the unique prefix `${name}_` to all global names introduced by our script will save us from possible collisions with the man:rc.subr[8] namespace. As a rule, [.filename]#rc.d# scripts of the base system need not provide defaults for their man:rc.conf[5] variables because the defaults should be set in [.filename]#/etc/defaults/rc.conf# instead. On the other hand, [.filename]#rc.d# scripts for ports should provide the defaults as shown in the example. ==== ➎ Here we use `dummy_msg` to actually control our script, i.e., to emit a variable message. Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is: [.programlisting] .... start_cmd="echo \"$dummy_msg\"" .... [[rcng-daemon]] == Startup and shutdown of a simple daemon We said earlier that man:rc.subr[8] could provide default methods. Obviously, such defaults cannot be too general. They are suited for the common case of starting and shutting down a simple daemon program. Let us assume now that we need to write an [.filename]#rc.d# script for such a daemon called `mumbled`. Here it is: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" <.> load_rc_config $name run_rc_command "$1" .... Pleasingly simple, isn't it? Let us examine our little script. The only new thing to note is as follows: ➊ The `command` variable is meaningful to man:rc.subr[8]. If it is set, man:rc.subr[8] will act according to the scenario of serving a conventional daemon. In particular, the default methods will be provided for such arguments: `start`, `stop`, `restart`, `poll`, and `status`. The daemon will be started by running `$command` with command-line flags specified by `$mumbled_flags`. Thus all the input data for the default `start` method are available in the variables set by our script. Unlike `start`, other methods may require additional information about the process started. For instance, `stop` must know the PID of the process to terminate it. In the present case, man:rc.subr[8] will scan through the list of all processes, looking for a process with its name equal to `procname`. The latter is another variable of meaning to man:rc.subr[8], and its value defaults to that of `command`. In other words, when we set `command`, `procname` is effectively set to the same value. This enables our script to kill the daemon and to check if it is running in the first place. [NOTE] ==== Some programs are in fact executable scripts. The system runs such a script by starting its interpreter and passing the name of the script to it as a command-line argument. This is reflected in the list of processes, which can confuse man:rc.subr[8]. You should additionally set `command_interpreter` to let man:rc.subr[8] know the actual name of the process if `$command` is a script. For each [.filename]#rc.d# script, there is an optional man:rc.conf[5] variable that takes precedence over `command`. Its name is constructed as follows: `${name}_program`, where `name` is the mandatory variable we discussed <>. E.g., in this case it will be `mumbled_program`. It is man:rc.subr[8] that arranges `${name}_program` to override `command`. Of course, man:sh[1] will permit you to set `${name}_program` from man:rc.conf[5] or the script itself even if `command` is unset. In that case, the special properties of `${name}_program` are lost, and it becomes an ordinary variable your script can use for its own purposes. However, the sole use of `${name}_program` is discouraged because using it together with `command` became an idiom of [.filename]#rc.d# scripting. ==== For more detailed information on default methods, refer to man:rc.subr[8]. [[rcng-daemon-adv]] == Startup and shutdown of an advanced daemon Let us add some meat onto the bones of the previous script and make it more complex and featureful. The default methods can do a good job for us, but we may need some of their aspects tweaked. Now we will learn how to tune the default methods to our needs. [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" command_args="mock arguments > /dev/null 2>&1" <.> pidfile="/var/run/${name}.pid" <.> required_files="/etc/${name}.conf /usr/share/misc/${name}.rules" <.> sig_reload="USR1" <.> start_precmd="${name}_prestart" <.> stop_postcmd="echo Bye-bye" <.> extra_commands="reload plugh xyzzy" <.> plugh_cmd="mumbled_plugh" <.> xyzzy_cmd="echo 'Nothing happens.'" mumbled_prestart() { if checkyesno mumbled_smart; then <.> rc_flags="-o smart ${rc_flags}" <.> fi case "$mumbled_mode" in foo) rc_flags="-frotz ${rc_flags}" ;; bar) rc_flags="-baz ${rc_flags}" ;; *) warn "Invalid value for mumbled_mode" <.> return 1 <.> ;; esac run_rc_command xyzzy <.> return 0 } mumbled_plugh() <.> { echo 'A hollow voice says "plugh".' } load_rc_config $name run_rc_command "$1" .... ➊ Additional arguments to `$command` can be passed in `command_args`. They will be added to the command line after `$mumbled_flags`. Since the final command line is passed to `eval` for its actual execution, input and output redirections can be specified in `command_args`. [NOTE] ==== _Never_ include dashed options, like `-X` or `--foo`, in `command_args`. The contents of `command_args` will appear at the end of the final command line, hence they are likely to follow arguments present in `${name}_flags`; but most commands will not recognize dashed options after ordinary arguments. A better way of passing additional options to `$command` is to add them to the beginning of `${name}_flags`. Another way is to modify `rc_flags` <>. ==== ➋ A good-mannered daemon should create a _pidfile_ so that its process can be found more easily and reliably. The variable `pidfile`, if set, tells man:rc.subr[8] where it can find the pidfile for its default methods to use. [NOTE] ==== In fact, man:rc.subr[8] will also use the pidfile to see if the daemon is already running before starting it. This check can be skipped by using the `faststart` argument. ==== ➌ If the daemon cannot run unless certain files exist, just list them in `required_files`, and man:rc.subr[8] will check that those files do exist before starting the daemon. There also are `required_dirs` and `required_vars` for directories and environment variables, respectively. They all are described in detail in man:rc.subr[8]. [NOTE] ==== The default method from man:rc.subr[8] can be forced to skip the prerequisite checks by using `forcestart` as the argument to the script. ==== ➍ We can customize signals to send to the daemon in case they differ from the well-known ones. In particular, `sig_reload` specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default. Another signal is sent to stop the daemon process; the default is SIGTERM, but this can be changed by setting `sig_stop` appropriately. [NOTE] ==== The signal names should be specified to man:rc.subr[8] without the `SIG` prefix, as it is shown in the example. The FreeBSD version of man:kill[1] can recognize the `SIG` prefix, but the versions from other OS types may not. ==== ➎➏ Performing additional tasks before or after the default methods is easy. For each command-argument supported by our script, we can define `argument_precmd` and `argument_postcmd`. These man:sh[1] commands are invoked before and after the respective method, as it is evident from their names. [NOTE] ==== Overriding a default method with a custom `argument_cmd` still does not prevent us from making use of `argument_precmd` or `argument_postcmd` if we need to. In particular, the former is good for checking custom, sophisticated conditions that should be met before performing the command itself. Using `argument_precmd` along with `argument_cmd` lets us logically separate the checks from the action. Do not forget that you can cram any valid man:sh[1] expressions into the methods, pre-, and post-commands you define. Just invoking a function that makes the real job is a good style in most cases, but never let style limit your understanding of what is going on behind the curtain. ==== ➐ If we would like to implement custom arguments, which can also be thought of as _commands_ to our script, we need to list them in `extra_commands` and provide methods to handle them. [NOTE] ==== The `reload` command is special. On the one hand, it has a preset method in man:rc.subr[8]. On the other hand, `reload` is not offered by default. The reason is that not all daemons use the same reload mechanism and some have nothing to reload at all. So we need to ask explicitly that the builtin functionality be provided. We can do so via `extra_commands`. What do we get from the default method for `reload`? Quite often daemons reload their configuration upon reception of a signal - typically, SIGHUP. Therefore man:rc.subr[8] attempts to reload the daemon by sending a signal to it. The signal is preset to SIGHUP but can be customized via `sig_reload` if necessary. ==== ➑⓮ Our script supports two non-standard commands, `plugh` and `xyzzy`. We saw them listed in `extra_commands`, and now it is time to provide methods for them. The method for `xyzzy` is just inlined while that for `plugh` is implemented as the `mumbled_plugh` function. Non-standard commands are not invoked during startup or shutdown. Usually they are for the system admin's convenience. They can also be used from other subsystems, e.g., man:devd[8] if specified in man:devd.conf[5]. The full list of available commands can be found in the usage line printed by man:rc.subr[8] when the script is invoked without arguments. For example, here is the usage line from the script under study: [source,shell] .... # /etc/rc.d/mumbled Usage: /etc/rc.d/mumbled [fast|force|one](start|stop|restart|rcvar|reload|plugh|xyzzy|status|poll) .... ⓭ A script can invoke its own standard or non-standard commands if needed. This may look similar to calling functions, but we know that commands and shell functions are not always the same thing. For instance, `xyzzy` is not implemented as a function here. In addition, there can be a pre-command and post-command, which should be invoked orderly. So the proper way for a script to run its own command is by means of man:rc.subr[8], as shown in the example. ➒ A handy function named `checkyesno` is provided by man:rc.subr[8]. It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to `YES`, or `TRUE`, or `ON`, or `1`, case insensitive; a non-zero exit code is returned otherwise. In the latter case, the function tests the variable for being set to `NO`, `FALSE`, `OFF`, or `0`, case insensitive; it prints a warning message if the variable contains anything else, i.e., junk. Keep in mind that for man:sh[1] a zero exit code means true and a non-zero exit code means false. [IMPORTANT] ==== The `checkyesno` function takes a __variable name__. Do not pass the expanded _value_ of a variable to it; it will not work as expected. The following is the correct usage of `checkyesno`: [.programlisting] .... if checkyesno mumbled_enable; then foo fi .... On the contrary, calling `checkyesno` as shown below will not work - at least not as expected: [.programlisting] .... if checkyesno "${mumbled_enable}"; then foo fi .... ==== ➓ [[rc-flags]]We can affect the flags to be passed to `$command` by modifying `rc_flags` in `$start_precmd`. ⓫ In certain cases we may need to emit an important message that should go to `syslog` as well. This can be done easily with the following man:rc.subr[8] functions: `debug`, `info`, `warn`, and `err`. The latter function then exits the script with the code specified. ⓬ The exit codes from methods and their pre-commands are not just ignored by default. If `argument_precmd` returns a non-zero exit code, the main method will not be performed. In turn, `argument_postcmd` will not be invoked unless the main method returns a zero exit code. [NOTE] ==== However, man:rc.subr[8] can be instructed from the command line to ignore those exit codes and invoke all commands anyway by prefixing an argument with `force`, as in `forcestart`. ==== [[rcng-hookup]] == Connecting a script to the rc.d framework After a script has been written, it needs to be integrated into [.filename]#rc.d#. The crucial step is to install the script in [.filename]#/etc/rc.d# (for the base system) or [.filename]#/usr/local/etc/rc.d# (for ports). Both [.filename]#bsd.prog.mk# and [.filename]#bsd.port.mk# provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode. System scripts should be installed from [.filename]#src/etc/rc.d# through the [.filename]#Makefile# found there. Port scripts can be installed using `USE_RC_SUBR` as described link:{porters-handbook}#rc-scripts[in the Porter's Handbook]. However, we should consider beforehand the place of our script in the system startup sequence. The service handled by our script is likely to depend on other services. For instance, a network daemon cannot function without the network interfaces and routing up and running. Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted. We mentioned man:rcorder[8] already. Now it is time to have a close look at it. In a nutshell, man:rcorder[8] takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to `stdout`. The point is to keep dependency information _inside_ the files so that each file can speak for itself only. A file can specify the following information: * the names of the "conditions" (which means services to us) it __provides__; * the names of the "conditions" it __requires__; * the names of the "conditions" this file should run __before__; * additional _keywords_ that can be used to select a subset from the whole set of files (man:rcorder[8] can be instructed via options to include or omit the files having particular keywords listed.) It is no surprise that man:rcorder[8] can handle only text files with a syntax close to that of man:sh[1]. That is, special lines understood by man:rcorder[8] look like man:sh[1] comments. The syntax of such special lines is rather rigid to simplify their processing. See man:rcorder[8] for details. Besides using man:rcorder[8] special lines, a script can insist on its dependency upon another service by just starting it forcibly. This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in man:rc.conf[5]. With this general knowledge in mind, let us consider the simple daemon script enhanced with dependency stuff: [.programlisting] .... #!/bin/sh # PROVIDE: mumbled oldmumble <.> # REQUIRE: DAEMON cleanvar frotz <.> # BEFORE: LOGIN <.> # KEYWORD: nojail shutdown <.> . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" start_precmd="${name}_prestart" mumbled_prestart() { if ! checkyesno frotz_enable && \ ! /etc/rc.d/frotz forcestatus 1>/dev/null 2>&1; then force_depend frotz || return 1 <.> fi return 0 } load_rc_config $name run_rc_command "$1" .... As before, detailed analysis follows: ➊ That line declares the names of "conditions" our script provides. Now other scripts can record a dependency on our script by those names. [NOTE] ==== Usually a script specifies a single condition provided. However, nothing prevents us from listing several conditions there, e.g., for compatibility reasons. In any case, the name of the main, or the only, `PROVIDE:` condition should be the same as `${name}`. ==== ➋➌ So our script indicates which "conditions" provided by other scripts it depends on. According to the lines, our script asks man:rcorder[8] to put it after the script(s) providing [.filename]#DAEMON# and [.filename]#cleanvar#, but before that providing [.filename]#LOGIN#. [NOTE] ==== The `BEFORE:` line should not be abused to work around an incomplete dependency list in the other script. The appropriate case for using `BEFORE:` is when the other script does not care about ours, but our script can do its task better if run before the other one. A typical real-life example is the network interfaces vs. the firewall: While the interfaces do not depend on the firewall in doing their job, the system security will benefit from the firewall being ready before there is any network traffic. Besides conditions corresponding to a single service each, there are meta-conditions and their "placeholder" scripts used to ensure that certain groups of operations are performed before others. These are denoted by [.filename]#UPPERCASE# names. Their list and purposes can be found in man:rc[8]. Keep in mind that putting a service name in the `REQUIRE:` line does not guarantee that the service will actually be running by the time our script starts. The required service may fail to start or just be disabled in man:rc.conf[5]. Obviously, man:rcorder[8] cannot track such details, and man:rc[8] will not do that either. Consequently, the application started by our script should be able to cope with any required services being unavailable. In certain cases, we can help it as discussed <> ==== [[keywords]]➍ As we remember from the above text, man:rcorder[8] keywords can be used to select or leave out some scripts. Namely any man:rcorder[8] consumer can specify through `-k` and `-s` options which keywords are on the "keep list" and "skip list", respectively. From all the files to be dependency sorted, man:rcorder[8] will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list. In FreeBSD, man:rcorder[8] is used by [.filename]#/etc/rc# and [.filename]#/etc/rc.shutdown#. These two scripts define the standard list of FreeBSD [.filename]#rc.d# keywords and their meanings as follows: [[forcedep]]➎ To begin with, `force_depend` should be used with much care. It is generally better to revise the hierarchy of configuration variables for your [.filename]#rc.d# scripts if they are interdependent. If you still cannot do without `force_depend`, the example offers an idiom of how to invoke it conditionally. In the example, our `mumbled` daemon requires that another one, `frotz`, be started in advance. However, `frotz` is optional, too; and man:rcorder[8] knows nothing about such details. Fortunately, our script has access to all man:rc.conf[5] variables. If `frotz_enable` is true, we hope for the best and rely on [.filename]#rc.d# to have started `frotz`. Otherwise we forcibly check the status of `frotz`. Finally, we enforce our dependency on `frotz` if it is found to be not running. A warning message will be emitted by `force_depend` because it should be invoked only if a misconfiguration has been detected. [[rcng-args]] == Giving more flexibility to an rc.d script When invoked during startup or shutdown, an [.filename]#rc.d# script is supposed to act on the entire subsystem it is responsible for. E.g., [.filename]#/etc/rc.d/netif# should start or stop all network interfaces described by man:rc.conf[5]. Either task can be uniquely indicated by a single command argument such as `start` or `stop`. Between startup and shutdown, [.filename]#rc.d# scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises. For instance, the admin may want to add the settings of a new network interface to man:rc.conf[5] and then to start it without interfering with the operation of the existing interfaces. Next time the admin may need to shut down a single network interface. In the spirit of the command line, the respective [.filename]#rc.d# script calls for an extra argument, the interface name. Fortunately, man:rc.subr[8] allows for passing any number of arguments to script's methods (within the system limits). Due to that, the changes in the script itself can be minimal. How can man:rc.subr[8] gain access to the extra command-line arguments. Should it just grab them directly? Not by any means. Firstly, an man:sh[1] function has no access to the positional parameters of its caller, but man:rc.subr[8] is just a sack of such functions. Secondly, the good manner of [.filename]#rc.d# dictates that it is for the main script to decide which arguments are to be passed to its methods. So the approach adopted by man:rc.subr[8] is as follows: `run_rc_command` passes on all its arguments but the first one to the respective method verbatim. The first, omitted, argument is the name of the method itself: `start`, `stop`, etc. It will be shifted out by `run_rc_command`, so what is `$2` in the original command line will be presented as `$1` to the method, and so on. To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied. Here we go: [.programlisting] .... #!/bin/sh . /etc/rc.subr name="dummy" start_cmd="${name}_start" stop_cmd=":" kiss_cmd="${name}_kiss" extra_commands="kiss" dummy_start() { if [ $# -gt 0 ]; then <.> echo "Greeting message: $*" else echo "Nothing started." fi } dummy_kiss() { echo -n "A ghost gives you a kiss" if [ $# -gt 0 ]; then <.> echo -n " and whispers: $*" fi case "$*" in *[.!?]) echo ;; *) echo . ;; esac } load_rc_config $name run_rc_command "$@" <.> .... What essential changes can we notice in the script? ➊ All arguments you type after `start` can end up as positional parameters to the respective method. We can use them in any way according to our task, skills, and fancy. In the current example, we just pass all of them to man:echo[1] as one string in the next line - note `$*` within the double quotes. Here is how the script can be invoked now: [source,shell] .... # /etc/rc.d/dummy start Nothing started. # /etc/rc.d/dummy start Hello world! Greeting message: Hello world! .... ➋ The same applies to any method our script provides, not only to a standard one. We have added a custom method named `kiss`, and it can take advantage of the extra arguments not less than `start` does. E.g.: [source,shell] .... # /etc/rc.d/dummy kiss A ghost gives you a kiss. # /etc/rc.d/dummy kiss Once I was Etaoin Shrdlu... A ghost gives you a kiss and whispers: Once I was Etaoin Shrdlu... .... ➌ If we want just to pass all extra arguments to any method, we can merely substitute `"$@"` for `"$1"` in the last line of our script, where we invoke `run_rc_command`. [IMPORTANT] ==== An man:sh[1] programmer ought to understand the subtle difference between `$*` and `$@` as the ways to designate all positional parameters. For its in-depth discussion, refer to a good handbook on man:sh[1] scripting. _Do not_ use the expressions until you fully understand them because their misuse will result in buggy and insecure scripts. ==== [NOTE] ==== Currently `run_rc_command` may have a bug that prevents it from keeping the original boundaries between arguments. That is, arguments with embedded whitespace may not be processed correctly. The bug stems from `$*` misuse. ==== [[rcng-furthur]] == Further reading [[lukem]]http://www.mewburn.net/luke/papers/rc.d.pdf[The original article by Luke Mewburn] offers a general overview of [.filename]#rc.d# and detailed rationale for its design decisions. It provides insight on the whole [.filename]#rc.d# framework and its place in a modern BSD operating system. [[manpages]]The manual pages man:rc[8], man:rc.subr[8], and man:rcorder[8] document the [.filename]#rc.d# components in great detail. You cannot fully use the [.filename]#rc.d# power without studying the manual pages and referring to them while writing your own scripts. The major source of working, real-life examples is [.filename]#/etc/rc.d# in a live system. Its contents are easy and pleasant to read because most rough corners are hidden deep in man:rc.subr[8]. Keep in mind though that the [.filename]#/etc/rc.d# scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions. Now you can improve them! diff --git a/documentation/content/en/articles/releng/_index.adoc b/documentation/content/en/articles/releng/_index.adoc index 4d53c368e4..8ed11ec631 100644 --- a/documentation/content/en/articles/releng/_index.adoc +++ b/documentation/content/en/articles/releng/_index.adoc @@ -1,445 +1,455 @@ --- title: Legacy FreeBSD Release Engineering authors: - author: Murray Stokely email: murray@FreeBSD.org webpage: https://people.FreeBSD.org/~murray/ description: This paper describes the approach used by past the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System trademarks: ["freebsd", "intel", "general"] tags: ["Release", "Engineering", "Historical", "FreeBSD"] --- = FreeBSD Release Engineering :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :xrefstyle: full + +ifeval::["{backend}" == "html5"] include::shared/releases.adoc[] include::shared/authors.adoc[] include::shared/en/teams.adoc[lines=16..-1] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] - -ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/releng/ endif::[] ifeval::["{backend}" == "pdf"] +include::../../../../shared/releases.adoc[] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/teams.adoc[lines=16..-1] +include::../../../../shared/en/mailing-lists.adoc[] +include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/articles/releng/ endif::[] ifeval::["{backend}" == "epub3"] +include::shared/releases.adoc[] +include::shared/authors.adoc[] +include::shared/en/teams.adoc[lines=16..-1] +include::shared/en/mailing-lists.adoc[] +include::shared/en/urls.adoc[] :imagesdir: ../../../../static/images/articles/releng/ endif::[] [.abstract-title] Abstract [NOTE] ==== This document is outdated and does not accurately describe the current release procedures of the FreeBSD Release Engineering team. It is retained for historical purposes. The current procedures used by the FreeBSD Release Engineering team are available in the link:{freebsd-releng}[FreeBSD Release Engineering] article. ==== This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It details the methodology used for the official FreeBSD releases and describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization. ''' toc::[] [[introduction]] == Introduction The development of FreeBSD is a very open process. FreeBSD is comprised of contributions from thousands of people around the world. The FreeBSD Project provides Subversion footnote:[Subversion, http://subversion.apache.org] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides. This has been a huge help in attracting more talented developers to FreeBSD. However, I think everyone would agree that chaos would soon manifest if write access to the main repository was opened up to everyone on the Internet. Therefore only a "select" group of nearly 300 people are given write access to the Subversion repository. These link:{contributors}#staff-committers[FreeBSD committers]footnote:[link:{contributors}#staff-committers[FreeBSD committers]] are usually the people who do the bulk of FreeBSD development. An elected link:https://www.FreeBSD.org/administration/#t-core[Core Team]footnote:[link:https://www.FreeBSD.org/administration/#t-core[FreeBSD Core Team]] of developers provide some level of direction over the project. The rapid pace of `FreeBSD` development makes the main development branch unsuitable for the everyday use by the general public. In particular, stabilizing efforts are required for polishing the development system into a production quality release. To solve this conflict, development continues on several parallel tracks. The main development branch is the _HEAD_ or _trunk_ of our Subversion tree, known as "FreeBSD-CURRENT" or "-CURRENT" for short. A set of more stable branches are maintained, known as "FreeBSD-STABLE" or "-STABLE" for short. All branches live in a master Subversion repository maintained by the FreeBSD Project. FreeBSD-CURRENT is the "bleeding-edge" of FreeBSD development where all new changes first enter the system. FreeBSD-STABLE is the development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community. The term _stable_ in the name of the branch refers to the presumed Application Binary Interface stability, which is promised by the project. This means that a user application compiled on an older version of the system from the same branch works on a newer system from the same branch. The ABI stability has improved greatly from the compared to previous releases. In most cases, binaries from the older _STABLE_ systems run unmodified on newer systems, including __HEAD__, assuming that the system management interfaces are not used. In the interim period between releases, weekly snapshots are built automatically by the FreeBSD Project build machines and made available for download from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/`. The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with Subversion and "`make buildworld`" footnote:[link:{handbook}#makeworld[Rebuilding world]] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release. In addition to installation ISO snapshots, weekly virtual machine images are also provided for use with VirtualBox, qemu, or other popular emulation software. The virtual machine images can be downloaded from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/VM-IMAGES/`. The virtual machine images are approximately 150MB man:xz[1] compressed, and contain a 10GB sparse filesystem when attached to a virtual machine. Bug reports and feature requests are continuously submitted by users throughout the release cycle. Problems reports are entered into our Bugzilla database through the web interface provided at https://www.freebsd.org/support/bugreports/[https://www.freebsd.org/support/bugreports/]. To service our most conservative users, individual release branches were introduced with FreeBSD 4.3. These release branches are created shortly before a final release is made. After the release goes out, only the most critical security fixes and additions are merged onto the release branch. In addition to source updates via Subversion, binary patchkits are available to keep systems on the _releng/X.Y_ branches updated. === What This Article Describes The following sections of this article describe: <>:: The different phases of the release engineering process leading up to the actual system build. <>:: The actual build process. <>:: How the base release may be extended by third parties. <>:: Some of the lessons learned through the release of FreeBSD 4.4. <>:: Future directions of development. [[release-proc]] == Release Process New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals. The FreeBSD release process begins to ramp up 70-80 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze. During this time, many developers perform what have become known as "MFC sweeps". MFC stands for "Merge From CURRENT" and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch. Project policy requires any change to be first applied to trunk, and merged to the -STABLE branches after sufficient external testing was done by -CURRENT users (developers are expected to extensively test the change before committing to -CURRENT, but it is impossible for a person to exercise all usages of the general-purpose operating system). Minimal MFC period is 3 days, which is typically used only for trivial or critical bugfixes. === Code Review Sixty days before the anticipated release, the source repository enters a "code freeze". During this time, all commits to the -STABLE branch must be approved by `{re}`. The approval process is technically enforced by a pre-commit hook. The kinds of changes that are allowed during this period include: * Bug fixes. * Documentation updates. * Security-related fixes of any kind. * Minor changes to device drivers, such as adding new Device IDs. * Driver updates from the vendors. * Any additional change that the release engineering team feels is justified, given the potential risk. Shortly after the code freeze is started, a _BETA1_ image is built and released for widespread testing. During the code freeze, at least one beta image or release candidate is released every two weeks until the final release is ready. During the days preceding the final release, the release engineering team is in constant communication with the security-officer team, the documentation maintainers, and the port maintainers to ensure that all of the different components required for a successful release are available. After the quality of the BETA images is satisfying enough, and no large and potentially risky changes are planned, the release branch is created and _Release Candidate_ (RC) images are built from the release branch, instead of the BETA images from the STABLE branch. Also, the freeze on the STABLE branch is lifted and release branch enters a "hard code freeze" where it becomes much harder to justify new changes to the system unless a serious bug-fix or security issue is involved. === Final Release Checklist When several BETA images have been made available for widespread testing and all major issues have been resolved, the final release "polishing" can begin. [[rel-branch]] ==== Creating the Release Branch [NOTE] ==== In all examples below, `$FSVN` refers to the location of the FreeBSD Subversion repository, `svn+ssh://svn.FreeBSD.org/base/`. ==== The layout of FreeBSD branches in Subversion is described in the link:{committers-guide}#subversion-primer-base-layout[Committer's Guide]. The first step in creating a branch is to identify the revision of the `stable/_X_` sources that you want to branch _from_. [source,shell] .... # svn log -v $FSVN/stable/9 .... The next step is to create the _release branch_ [source,shell] .... # svn cp $FSVN/stable/9@REVISION $FSVN/releng/9.2 .... This branch can be checked out: [source,shell] .... # svn co $FSVN/releng/9.2 src .... [NOTE] ==== Creating the `releng` branch and `release` tags is done by the link:https://www.FreeBSD.org/administration/#t-re[Release Engineering Team]. ==== image::branches-head.png[FreeBSD Development Branch] image::branches-releng3.png[FreeBSD 3.x STABLE Branch] image::branches-releng4.png[FreeBSD 4.x STABLE Branch] image::branches-releng5.png[FreeBSD 5.x STABLE Branch] image::branches-releng6.png[FreeBSD 6.x STABLE Branch] image::branches-releng7.png[FreeBSD 7.x STABLE Branch] image::branches-releng8.png[FreeBSD 8.x STABLE Branch] image::branches-releng9.png[FreeBSD 9.x STABLE Branch] [[versionbump]] ==== Bumping up the Version Number Before the final release can be tagged, built, and released, the following files need to be modified to reflect the correct version of FreeBSD: * [.filename]#doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.xml# * [.filename]#doc/en_US.ISO8859-1/books/porters-handbook/book.xml# * [.filename]#doc/en_US.ISO8859-1/htdocs/cgi/ports.cgi# * [.filename]#ports/Tools/scripts/release/config# * [.filename]#doc/shared/xml/freebsd.ent# * [.filename]#src/Makefile.inc1# * [.filename]#src/UPDATING# * [.filename]#src/gnu/usr.bin/groff/tmac/mdoc.local# * [.filename]#src/release/Makefile# * [.filename]#src/release/doc/en_US.ISO8859-1/shared/xml/release.dsl# * [.filename]#src/release/doc/shared/examples/Makefile.relnotesng# * [.filename]#src/release/doc/shared/xml/release.ent# * [.filename]#src/sys/conf/newvers.sh# * [.filename]#src/sys/sys/param.h# * [.filename]#src/usr.sbin/pkg_install/add/main.c# * [.filename]#doc/en_US.ISO8859-1/htdocs/search/opensearch/man.xml# The release notes and errata files also need to be adjusted for the new release (on the release branch) and truncated appropriately (on the stable/current branch): * [.filename]#src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml# * [.filename]#src/release/doc/en_US.ISO8859-1/errata/article.xml# Sysinstall should be updated to note the number of available ports and the amount of disk space required for the Ports Collection. footnote:[FreeBSD Ports Collection https://www.FreeBSD.org/ports] This information is currently kept in [.filename]#src/usr.sbin/bsdinstall/dist.c#. After the release has been built, a number of files should be updated to announce the release to the world. These files are relative to `head/` within the `doc/` subversion tree. * [.filename]#share/images/articles/releng/branches-relengX.pic# * [.filename]#head/shared/xml/release.ent# * [.filename]#en_US.ISO8859-1/htdocs/releases/*# * [.filename]#en_US.ISO8859-1/htdocs/releng/index.xml# * [.filename]#share/xml/news.xml# Additionally, update the "BSD Family Tree" file: * [.filename]#src/shared/misc/bsd-family-tree# ==== Creating the Release Tag When the final release is ready, the following command will create the `release/9.2.0` tag. [source,shell] .... # svn cp $FSVN/releng/9.2 $FSVN/release/9.2.0 .... The Documentation and Ports managers are responsible for tagging their respective trees with the `tags/RELEASE_9_2_0` tag. When the Subversion `svn cp` command is used to create a __release tag__, this identifies the source at a specific point in time. By creating tags, we ensure that future release builders will always be able to use the exact same source we used to create the official FreeBSD Project releases. [[release-build]] == Release Building FreeBSD "releases" can be built by anyone with a fast machine and access to a source repository. (That should be everyone, since we offer Subversion access! See the link:{handbook}#svn[Subversion section in the Handbook] for details.) The _only_ special requirement is that the man:md[4] device must be available. If the device is not loaded into your kernel, then the kernel module should be automatically loaded when man:mdconfig[8] is executed during the boot media creation phase. All of the tools necessary to build a release are available from the Subversion repository in [.filename]#src/release#. These tools aim to provide a consistent way to build FreeBSD releases. A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM or DVD, and an FTP install directory. man:release[7] fully documents the `src/release/generate-release.sh` script which is used to build a release. `generate-release.sh` is a wrapper around the Makefile target: `make release`. === Building a Release man:release[7] documents the exact commands required to build a FreeBSD release. The following sequences of commands can build an 9.2.0 release: [source,shell] .... # cd /usr/src/release # sh generate-release.sh release/9.2.0 /local3/release .... After running these commands, all prepared release files are available in [.filename]#/local3/release/R# directory. The release [.filename]#Makefile# can be broken down into several distinct steps. * Creation of a sanitized system environment in a separate directory hierarchy with "`make installworld`". * Checkout from Subversion of a clean version of the system source, documentation, and ports into the release build hierarchy. * Population of [.filename]#/etc# and [.filename]#/dev# in the chrooted environment. * chroot into the release build hierarchy, to make it harder for the outside environment to taint this build. * `make world` in the chrooted environment. * Build of Kerberos-related binaries. * Build [.filename]#GENERIC# kernel. * Creation of a staging directory tree where the binary distributions will be built and packaged. * Build and installation of the documentation toolchain needed to convert the documentation source (SGML) into HTML and text documents that will accompany the release. * Build and installation of the actual documentation (user manuals, tutorials, release notes, hardware compatibility lists, and so on.) * Package up distribution tarballs of the binaries and sources. * Create FTP installation hierarchy. * _(optionally)_ Create ISO images for CDROM/DVD media. For more information about the release build infrastructure, please see man:release[7]. [NOTE] ==== It is important to remove any site-specific settings from [.filename]#/etc/make.conf#. For example, it would be unwise to distribute binaries that were built on a system with `CPUTYPE` set to a specific processor. ==== === Contributed Software ("ports") The https://www.FreeBSD.org/ports[FreeBSD Ports collection] is a collection of over {numports} third-party software packages available for FreeBSD. The `{portmgr}` is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany official FreeBSD releases. === Release ISOs Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the _BSDi/Wind River Systems/FreeBSD Mall_ "official" CDROM distributions. Each of the four discs must contain a [.filename]#README.TXT# file that explains the contents of the disc, a [.filename]#CDROM.INF# file that provides meta-data for the disc so that man:bsdinstall[8] can validate and use the contents, and a [.filename]#filename.txt# file that provides a manifest for the disc. This _manifest_ can be created with a simple command: [source,shell] .... /stage/cdrom# find . -type f | sed -e 's/^\.\///' | sort > filename.txt .... The specific requirements of each CD are outlined below. ==== Disc 1 The first disc is almost completely created by `make release`. The only changes that should be made to the [.filename]#disc1# directory are the addition of a [.filename]#tools# directory, and as many popular third party software packages as will fit on the disc. The [.filename]#tools# directory contains software that allow users to create installation floppies from other operating systems. This disc should be made bootable so that users of modern PCs do not need to create installation floppy disks. If a custom kernel of FreeBSD is to be included, then man:bsdinstall[8] and man:release[7] must be updated to include installation instructions. The relevant code is contained in [.filename]#src/release# and [.filename]#src/usr.sbin/bsdinstall#. Specifically, the file [.filename]#src/release/Makefile#, and [.filename]#dist.c#, [.filename]#dist.h#, [.filename]#menus.c#, [.filename]#install.c#, and [.filename]#Makefile# will need to be updated under [.filename]#src/usr.sbin/bsdinstall#. Optionally, you may choose to update [.filename]#bsdinstall.8#. ==== Disc 2 The second disc is also largely created by `make release`. This disc contains a "live filesystem" that can be used from man:bsdinstall[8] to troubleshoot a FreeBSD installation. This disc should be bootable and should also contain a compressed copy of the CVS repository in the [.filename]#CVSROOT# directory and commercial software demos in the [.filename]#commerce# directory. ==== Multi-volume Support Sysinstall supports multiple volume package installations. This requires that each disc have an [.filename]#INDEX# file containing all of the packages on all volumes of a set, along with an extra field that indicates which volume that particular package is on. Each volume in the set must also have the `CD_VOLUME` variable set in the [.filename]#cdrom.inf# file so that bsdinstall can tell which volume is which. When a user attempts to install a package that is not on the current disc, bsdinstall will prompt the user to insert the appropriate one. [[distribution]] == Distribution [[dist-ftp]] === FTP Sites When the release has been thoroughly tested and packaged for distribution, the master FTP site must be updated. The official FreeBSD public FTP sites are all mirrors of a master server that is open only to other FTP sites. This site is known as `ftp-master`. When the release is ready, the following files must be modified on `ftp-master`: [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/#:: The installable FTP directory as output from `make release`. [.filename]#/pub/FreeBSD/ports/arch/packages-X.Y-release/#:: The complete package build for this release. [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/tools#:: A symlink to [.filename]#../../../tools#. [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/packages#:: A symlink to [.filename]#../../../ports/arch/packages-X.Y-release#. [.filename]#/pub/FreeBSD/releases/arch/ISO-IMAGES/X.Y/X.Y-RELEASE-arch-*.iso#:: The ISO images. The "*" is [.filename]#disc1#, [.filename]#disc2#, etc. Only if there is a [.filename]#disc1# and there is an alternative first installation CD (for example a stripped-down install with no windowing system) there may be a [.filename]#mini# as well. For more information about the distribution mirror architecture of the FreeBSD FTP sites, please see the link:{hubs}[Mirroring FreeBSD] article. It may take many hours to two days after updating `ftp-master` before a majority of the Tier-1 FTP sites have the new software depending on whether or not a package set got loaded at the same time. It is imperative that the release engineers coordinate with the {mirror-announce} before announcing the general availability of new software on the FTP sites. Ideally the release package set should be loaded at least four days prior to release day. The release bits should be loaded between 24 and 48 hours before the planned release time with "other" file permissions turned off. This will allow the mirror sites to download it but the general public will not be able to download it from the mirror sites. Mail should be sent to {mirror-announce} at the time the release bits get posted saying the release has been staged and giving the time that the mirror sites should begin allowing access. Be sure to include a time zone with the time, for example make it relative to GMT. [[dist-cdrom]] === CD-ROM Replication Coming soon: Tips for sending FreeBSD ISOs to a replicator and quality assurance measures to be taken. [[extensibility]] == Extensibility Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution. We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of. The only "rule" we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements! The FreeBSD community can only help support users of the software we provide. We certainly encourage innovation in the form of advanced installation and administration tools, for example, but we cannot be expected to answer questions about it. === Scripting `bsdinstall` The FreeBSD system installation and configuration tool, man:bsdinstall[8], can be scripted to provide automated installs for large sites. This functionality can be used in conjunction with Intel(R) PXE footnote:[link:{handbook}#network-diskless[Diskless Operation with PXE]] to bootstrap systems from the network. [[lessons-learned]] == Lessons Learned from FreeBSD 4.4 The release engineering process for 4.4 formally began on August 1st, 2001. After that date all commits to the `RELENG_4` branch of FreeBSD had to be explicitly approved by the `{re}`. The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th. The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates. A total of over _500_ emails were sent to the `{re}` in little over a month. Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates. The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent. This will become even more important as FreeBSD is ported to new platforms. [[future]] == Future Directions It is imperative for our release engineering activities to scale with our growing userbase. Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases. * _Parallelism_ - Certain portions of the release build are actually "embarrassingly parallel". Most of the tasks are very I/O intensive, so having multiple high-speed disk drives is actually more important than using multiple processors in speeding up the `make release` process. If multiple disks are used for different hierarchies in the man:chroot[2] environment, then the CVS checkout of the [.filename]#ports# and [.filename]#doc# trees can be happening simultaneously as the `make world` on another disk. Using a RAID solution (hardware or software) can significantly decrease the overall build time. * _Cross-building releases_ - Building IA-64 or Alpha release on x86 hardware? `make TARGET=ia64 release`. * _Regression Testing_ - We need better automated correctness testing for FreeBSD. * _Installation Tools_ - Our installation program has long since outlived its intended life span. Several projects are under development to provide a more advanced installation mechanism. The libh project was one such project that aimed to provide an intelligent new package framework and GUI installation program. [[ackno]] == Acknowledgements I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today. Of course the release would not have been possible without all of the release-related work done by `{asami}`, `{steve}`, `{bmah}`, `{nik}`, `{obrien}`, `{kris}`, `{jhb}` and the rest of the FreeBSD development community. I would also like to thank `{rgrimes}`, `{phk}`, and others who worked on the release engineering tools in the very early days of FreeBSD. This article was influenced by release engineering documents from the CSRG footnote:[Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: link:http://docs.FreeBSD.org/44doc/papers/releng.html[The Release Engineering of 4.3BSD]] , the NetBSD Project, footnote:[NetBSD Developer Documentation: Release Engineering http://www.NetBSD.org/developers/releng/index.html] , and John Baldwin's proposed release engineering process notes. footnote:[John Baldwin's FreeBSD Release Engineering Proposal https://people.FreeBSD.org/~jhb/docs/releng.txt] diff --git a/documentation/content/en/articles/remote-install/_index.adoc b/documentation/content/en/articles/remote-install/_index.adoc index 8baa70f1ab..5d6a8b5e4a 100644 --- a/documentation/content/en/articles/remote-install/_index.adoc +++ b/documentation/content/en/articles/remote-install/_index.adoc @@ -1,377 +1,389 @@ --- title: Remote Installation of the FreeBSD Operating System Without a Remote Console authors: - author: Daniel Gerzo email: danger@FreeBSD.org copyright: 2008-2021 The FreeBSD Documentation Project description: Describes the remote installation of the FreeBSD operating system when the console of the remote system is unavailable trademarks: ["freebsd", "general"] tags: ["Remote", "Installation", "FreeBSD"] --- = Remote Installation of the FreeBSD Operating System Without a Remote Console :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article documents the remote installation of the FreeBSD operating system when the console of the remote system is unavailable. The main idea behind this article is the result of a collaboration with `{mm}` with valuable input provided by `{pjd}`. ''' toc::[] [[background]] == Background There are many server hosting providers in the world, but very few of them are officially supporting FreeBSD. They usually provide support for a Linux(R) distribution to be installed on the servers they offer. In some cases, these companies will install your preferred Linux(R) distribution if you request it. Using this option, we will attempt to install FreeBSD. In other cases, they may offer a rescue system which would be used in an emergency. It is possible to use this for our purposes as well. This article covers the basic installation and configuration steps required to bootstrap a remote installation of FreeBSD with RAID-1 and ZFS capabilities. [[intro]] == Introduction This section will summarize the purpose of this article and better explain what is covered herein. The instructions included in this article will benefit those using services provided by colocation facilities not supporting FreeBSD. [.procedure] ==== . As we have mentioned in the <> section, many of the reputable server hosting companies provide some kind of rescue system, which is booted from their LAN and accessible over SSH. They usually provide this support in order to help their customers fix broken operating systems. As this article will explain, it is possible to install FreeBSD with the help of these rescue systems. + . The next section of this article will describe how to configure, and build minimalistic FreeBSD on the local machine. That version will eventually be running on the remote machine from a ramdisk, which will allow us to install a complete FreeBSD operating system from an FTP mirror using the sysinstall utility. . The rest of this article will describe the installation procedure itself, as well as the configuration of the ZFS file system. ==== [[requirements]] === Requirements To continue successfully, you must: * Have a network accessible operating system with SSH access * Understand the FreeBSD installation process * Be familiar with the man:sysinstall[8] utility * Have the FreeBSD installation SO image or CD handy [[preparation]] == Preparation - mfsBSD Before FreeBSD may be installed on the target system, it is necessary to build the minimal FreeBSD operating system image which will boot from the hard drive. This way the new system can be accessed from the network, and the rest of the installation can be done without remote access to the system console. The mfsBSD tool-set can be used to build a tiny FreeBSD image. As the name of mfsBSD suggests ("mfs" means "memory file system"), the resulting image runs entirely from a ramdisk. Thanks to this feature, the manipulation of hard drives will not be limited, therefore it will be possible to install a complete FreeBSD operating system. The mfsBSD http://mfsbsd.vx.sk/[home page] includes pointers to the latest release of the toolset. Please note that the internals of mfsBSD and how it all fits together is beyond the scope of this article. The interested reader should consult the original documentation of mfsBSD for more details. Download and extract the latest mfsBSD release and change your working directory to the directory where the mfsBSD scripts will reside: [source,shell] .... # fetch http://mfsbsd.vx.sk/release/mfsbsd-2.1.tar.gz # tar xvzf mfsbsd-2.1.tar.gz # cd mfsbsd-2.1/ .... [[mfsbsd-config]] === Configuration of mfsBSD Before booting mfsBSD, a few important configuration options have to be set. The most important that we have to get right is, naturally, the network setup. The most suitable method to configure networking options depends on whether we know beforehand the type of the network interface we will use, and the network interface driver to be loaded for our hardware. We will see how mfsBSD can be configured in either case. Another important thing to set is the `root` password. This can be done by editing [.filename]#conf/loader.conf#. Please see the included comments. ==== The [.filename]#conf/interfaces.conf# method When the installed network interface card is unknown, it is possible to use the auto-detection features of mfsBSD. The startup scripts of mfsBSD can detect the correct driver to use, based on the MAC address of the interface, if we set the following options in [.filename]#conf/interfaces.conf#: [.programlisting] .... mac_interfaces="ext1" ifconfig_ext1_mac="00:00:00:00:00:00" ifconfig_ext1="inet 192.168.0.2/24" .... Do not forget to add the `defaultrouter` information to [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" .... ==== The [.filename]#conf/rc.conf# Method When the network interface driver is known, it is more convenient to use [.filename]#conf/rc.conf# for networking options. The syntax of this file is the same as the one used in the standard man:rc.conf[5] file of FreeBSD. For example, if you know that a man:re[4] network interface is going to be available, you can set the following options in [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" ifconfig_re0="inet 192.168.0.2/24" .... [[mfsbsd-build]] === Building an mfsBSD Image The process of building an mfsBSD image is pretty straightforward. The first step is to mount the FreeBSD installation CD, or the installation ISO image to [.filename]#/cdrom#. For the sake of example, in this article we will assume that you have downloaded the FreeBSD 10.1-RELEASE ISO. Mounting this ISO image to the [.filename]#/cdrom# directory is easy with the man:mdconfig[8] utility: [source,shell] .... # mdconfig -a -t vnode -u 10 -f FreeBSD-10.1-RELEASE-amd64-disc1.iso # mount_cd9660 /dev/md10 /cdrom .... Since the recent FreeBSD releases do not contain regular distribution sets, it is required to extract the FreeBSD distribution files from the distribution archives located on the ISO image: [source,shell] .... # mkdir DIST # tar -xvf /cdrom/usr/freebsd-dist/base.txz -C DIST # tar -xvf /cdrom/usr/freebsd-dist/kernel.txz -C DIST .... Next, build the bootable mfsBSD image: [source,shell] .... # make BASE=DIST .... [NOTE] ==== The above `make` has to be run from the top level of the mfsBSD directory tree, for example [.filename]#~/mfsbsd-2.1/#. ==== === Booting mfsBSD Now that the mfsBSD image is ready, it must be uploaded to the remote system running a live rescue system or pre-installed Linux(R) distribution. The most suitable tool for this task is scp: [source,shell] .... # scp disk.img root@192.168.0.2:. .... To boot mfsBSD image properly, it must be placed on the first (bootable) device of the given machine. This may be accomplished using this example providing that [.filename]#sda# is the first bootable disk device: [source,shell] .... # dd if=/root/disk.img of=/dev/sda bs=1m .... If all went well, the image should now be in the MBR of the first device and the machine can be rebooted. Watch for the machine to boot up properly with the man:ping[8] tool. Once it has came back on-line, it should be possible to access it over man:ssh[1] as user `root` with the configured password. [[installation]] == Installation of the FreeBSD Operating System The mfsBSD has been successfully booted and it should be possible to log in through man:ssh[1]. This section will describe how to create and label slices, set up `gmirror` for RAID-1, and how to use `sysinstall` to install a minimal distribution of the FreeBSD operating system. === Preparation of Hard Drives The first task is to allocate disk space for FreeBSD, i.e.: to create slices and partitions. Obviously, the currently running system is fully loaded in system memory and therefore there will be no problems with manipulating hard drives. To complete this task, it is possible to use either `sysinstall` or man:fdisk[8] in conjunction to man:bsdlabel[8]. At the start, mark all system disks as empty. Repeat the following command for each hard drive: [source,shell] .... # dd if=/dev/zero of=/dev/ad0 count=2 .... Next, create slices and label them with your preferred tool. While it is considered easier to use `sysinstall`, a powerful and also probably less buggy method will be to use standard text-based UNIX(R) tools, such as man:fdisk[8] and man:bsdlabel[8], which will also be covered in this section. The former option is well documented in the link:{handbook}#install-steps[Installing FreeBSD] chapter of the FreeBSD Handbook. As it was mentioned in the introduction, this article will present how to set up a system with RAID-1 and ZFS capabilities. Our set up will consist of a small man:gmirror[8] mirrored [.filename]#/# (root), [.filename]#/usr# and [.filename]#/var# dataset, and the rest of the disk space will be allocated for a man:zpool[8] mirrored ZFS file system. Please note, that the ZFS file system will be configured after the FreeBSD operating system is successfully installed and booted. The following example will describe how to create slices and labels, initialize man:gmirror[8] on each partition and how to create a UFS2 file system in each mirrored partition: [source,shell] .... # fdisk -BI /dev/ad0 <.> # fdisk -BI /dev/ad1 # bsdlabel -wB /dev/ad0s1 <.> # bsdlabel -wB /dev/ad1s1 # bsdlabel -e /dev/ad0s1 <.> # bsdlabel /dev/ad0s1 > /tmp/bsdlabel.txt && bsdlabel -R /dev/ad1s1 /tmp/bsdlabel.txt <.> # gmirror label root /dev/ad[01]s1a <.> # gmirror label var /dev/ad[01]s1d # gmirror label usr /dev/ad[01]s1e # gmirror label -F swap /dev/ad[01]s1b <.> # newfs /dev/mirror/root <.> # newfs /dev/mirror/var # newfs /dev/mirror/usr .... <.> Create a slice covering the entire disk and initialize the boot code contained in sector 0 of the given disk. Repeat this command for all hard drives in the system. <.> Write a standard label for each disk including the bootstrap code. <.> Now, manually edit the label of the given disk. Refer to the man:bsdlabel[8] manual page in order to find out how to create partitions. Create partitions `a` for [.filename]#/# (root) file system, `b` for swap, `d` for [.filename]#/var#, `e` for [.filename]#/usr# and finally `f` which will later be used for ZFS. <.> Import the recently created label for the second hard drive, so both hard drives will be labeled in the same way. <.> Initialize man:gmirror[8] on each partition. <.> Note that `-F` is used for the swap partition. This instructs man:gmirror[8] to assume that the device is in the consistent state after the power/system failure. <.> Create a UFS2 file system on each mirrored partition. === System Installation This is the most important part. This section will describe how to actually install the minimal distribution of FreeBSD on the hard drives that we have prepared in the previous section. To accomplish this goal, all file systems need to be mounted so `sysinstall` may write the contents of FreeBSD to the hard drives: [source,shell] .... # mount /dev/mirror/root /mnt # mkdir /mnt/var /mnt/usr # mount /dev/mirror/var /mnt/var # mount /dev/mirror/usr /mnt/usr .... When you are done, start man:sysinstall[8]. Select the [.guimenuitem]#Custom# installation from the main menu. Select [.guimenuitem]#Options# and press kbd:[Enter]. With the help of arrow keys, move the cursor on the `Install Root` item, press kbd:[Space] and change it to [.filename]#/mnt#. Press kbd:[Enter] to submit your changes and exit the [.guimenuitem]#Options# menu by pressing kbd:[q]. [WARNING] ==== Note that this step is very important and if skipped, `sysinstall` will be unable to install FreeBSD. ==== Go to the [.guimenuitem]#Distributions# menu, move the cursor with the arrow keys to `Minimal`, and check it by pressing kbd:[Space]. This article uses the Minimal distribution in order to save network traffic, because the system itself will be installed over ftp. Exit this menu by choosing `Exit`. [NOTE] ==== The [.guimenuitem]#Partition# and [.guimenuitem]#Label# menus will be skipped, as these are useless now. ==== In the [.guimenuitem]#Media# menu, select `FTP`. Select the nearest mirror and let `sysinstall` assume that the network is already configured. You will be returned back to the [.guimenuitem]#Custom# menu. Finally, perform the system installation by selecting the last option, [.guimenuitem]#Commit#. Exit `sysinstall` when it finishes the installation. === Post Installation Steps The FreeBSD operating system should be installed now; however, the process is not finished yet. It is necessary to perform some post installation steps in order to allow FreeBSD to boot in the future and to be able to log in to the system. You must now man:chroot[8] into the freshly installed system in order to finish the installation. Use the following command: [source,shell] .... # chroot /mnt .... To complete our goal, perform these steps: * Copy the `GENERIC` kernel to the [.filename]#/boot/kernel# directory: + [source,shell] .... # cp -Rp /boot/GENERIC/* /boot/kernel .... * Create the [.filename]#/etc/rc.conf#, [.filename]#/etc/resolv.conf# and [.filename]#/etc/fstab# files. Do not forget to properly set the network information and to enable sshd in [.filename]#/etc/rc.conf#. The contents of [.filename]#/etc/fstab# will be similar to the following: + [.programlisting] .... # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 /dev/mirror/root / ufs rw 1 1 /dev/mirror/usr /usr ufs rw 2 2 /dev/mirror/var /var ufs rw 2 2 /dev/cd0 /cdrom cd9660 ro,noauto 0 0 .... * Create [.filename]#/boot/loader.conf# with the following contents: + [.programlisting] .... geom_mirror_load="YES" zfs_load="YES" .... * Perform the following command, which will make ZFS available on the next boot: + [source,shell] .... # sysrc zfs_enable="YES" .... * Add additional users to the system using the man:adduser[8] tool. Do not forget to add a user to the `wheel` group so you may obtain root access after the reboot. * Double-check all your settings. The system should now be ready for the next boot. Use the man:reboot[8] command to reboot your system. [[zfs]] == ZFS If your system survived the reboot, it should now be possible to log in. Welcome to the fresh FreeBSD installation, performed remotely without the use of a remote console! The only remaining step is to configure man:zpool[8] and create some man:zfs[8] file systems. Creating and administering ZFS is very straightforward. First, create a mirrored pool: [source,shell] .... # zpool create tank mirror /dev/ad[01]s1f .... Next, create some file systems: [source,shell] .... # zfs create tank/ports # zfs create tank/src # zfs set compression=gzip tank/ports # zfs set compression=on tank/src # zfs set mountpoint=/usr/ports tank/ports # zfs set mountpoint=/usr/src tank/src .... That is all. If you are interested in more details about ZFS on FreeBSD, please refer to the https://wiki.freebsd.org/ZFS[ZFS] section of the FreeBSD Wiki. diff --git a/documentation/content/en/articles/serial-uart/_index.adoc b/documentation/content/en/articles/serial-uart/_index.adoc index 6e4f646700..084f4493d6 100644 --- a/documentation/content/en/articles/serial-uart/_index.adoc +++ b/documentation/content/en/articles/serial-uart/_index.adoc @@ -1,1164 +1,1177 @@ --- title: Serial and UART Tutorial authors: - author: Frank Durda email: uhclem@FreeBSD.org description: Detailed information about the use of serial ports and UART with FreeBSD trademarks: ["freebsd", "microsoft", "general"] tags: ["Serial", "hardware", "UART", "Tutorial", "FreeBSD"] --- = Serial and UART Tutorial :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: + +ifeval::["{backend}" == "html5"] include::shared/authors.adoc[] include::shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "pdf"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] + +ifeval::["{backend}" == "epub3"] +include::../../../../shared/authors.adoc[] +include::../../../../shared/en/urls.adoc[] +endif::[] [.abstract-title] Abstract This article talks about using serial hardware with FreeBSD. ''' toc::[] [[uart]] == The UART: What it is and how it works _Copyright (R) 1996 `{uhclem}`, All Rights Reserved. 13 January 1996._ The Universal Asynchronous Receiver/Transmitter (UART) controller is the key component of the serial communications subsystem of a computer. The UART takes bytes of data and transmits the individual bits in a sequential fashion. At the destination, a second UART re-assembles the bits into complete bytes. Serial transmission is commonly used with modems and for non-networked communication between computers, terminals and other devices. There are two primary forms of serial transmission: Synchronous and Asynchronous. Depending on the modes that are supported by the hardware, the name of the communication sub-system will usually include a `A` if it supports Asynchronous communications, and a `S` if it supports Synchronous communications. Both forms are described below. Some common acronyms are: [.blockquote] UART Universal Asynchronous Receiver/Transmitter [.blockquote] USART Universal Synchronous-Asynchronous Receiver/Transmitter === Synchronous Serial Transmission Synchronous serial transmission requires that the sender and receiver share a clock with one another, or that the sender provide a strobe or other timing signal so that the receiver knows when to "read" the next bit of the data. In most forms of serial Synchronous communication, if there is no data available at a given instant to transmit, a fill character must be sent instead so that data is always being transmitted. Synchronous communication is usually more efficient because only data bits are transmitted between sender and receiver, and synchronous communication can be more costly if extra wiring and circuits are required to share a clock signal between the sender and receiver. A form of Synchronous transmission is used with printers and fixed disk devices in that the data is sent on one set of wires while a clock or strobe is sent on a different wire. Printers and fixed disk devices are not normally serial devices because most fixed disk interface standards send an entire word of data for each clock or strobe signal by using a separate wire for each bit of the word. In the PC industry, these are known as Parallel devices. The standard serial communications hardware in the PC does not support Synchronous operations. This mode is described here for comparison purposes only. === Asynchronous Serial Transmission Asynchronous transmission allows data to be transmitted without the sender having to send a clock signal to the receiver. Instead, the sender and receiver must agree on timing parameters in advance and special bits are added to each word which are used to synchronize the sending and receiving units. When a word is given to the UART for Asynchronous transmissions, a bit called the "Start Bit" is added to the beginning of each word that is to be transmitted. The Start Bit is used to alert the receiver that a word of data is about to be sent, and to force the clock in the receiver into synchronization with the clock in the transmitter. These two clocks must be accurate enough to not have the frequency drift by more than 10% during the transmission of the remaining bits in the word. (This requirement was set in the days of mechanical teleprinters and is easily met by modern electronic equipment.) After the Start Bit, the individual bits of the word of data are sent, with the Least Significant Bit (LSB) being sent first. Each bit in the transmission is transmitted for exactly the same amount of time as all of the other bits, and the receiver "looks" at the wire at approximately halfway through the period assigned to each bit to determine if the bit is a `1` or a `0`. For example, if it takes two seconds to send each bit, the receiver will examine the signal to determine if it is a `1` or a `0` after one second has passed, then it will wait two seconds and then examine the value of the next bit, and so on. The sender does not know when the receiver has "looked" at the value of the bit. The sender only knows when the clock says to begin transmitting the next bit of the word. When the entire data word has been sent, the transmitter may add a Parity Bit that the transmitter generates. The Parity Bit may be used by the receiver to perform simple error checking. Then at least one Stop Bit is sent by the transmitter. When the receiver has received all of the bits in the data word, it may check for the Parity Bits (both sender and receiver must agree on whether a Parity Bit is to be used), and then the receiver looks for a Stop Bit. If the Stop Bit does not appear when it is supposed to, the UART considers the entire word to be garbled and will report a Framing Error to the host processor when the data word is read. The usual cause of a Framing Error is that the sender and receiver clocks were not running at the same speed, or that the signal was interrupted. Regardless of whether the data was received correctly or not, the UART automatically discards the Start, Parity and Stop bits. If the sender and receiver are configured identically, these bits are not passed to the host. If another word is ready for transmission, the Start Bit for the new word can be sent as soon as the Stop Bit for the previous word has been sent. As asynchronous data is "self synchronizing", if there is no data to transmit, the transmission line can be idle. === Other UART Functions In addition to the basic job of converting data from parallel to serial for transmission and from serial to parallel on reception, a UART will usually provide additional circuits for signals that can be used to indicate the state of the transmission media, and to regulate the flow of data in the event that the remote device is not prepared to accept more data. For example, when the device connected to the UART is a modem, the modem may report the presence of a carrier on the phone line while the computer may be able to instruct the modem to reset itself or to not take calls by raising or lowering one more of these extra signals. The function of each of these additional signals is defined in the EIA RS232-C standard. === The RS232-C and V.24 Standards In most computer systems, the UART is connected to circuitry that generates signals that comply with the EIA RS232-C specification. There is also a CCITT standard named V.24 that mirrors the specifications included in RS232-C. ==== RS232-C Bit Assignments (Marks and Spaces) In RS232-C, a value of `1` is called a `Mark` and a value of `0` is called a `Space`. When a communication line is idle, the line is said to be "Marking", or transmitting continuous `1` values. The Start bit always has a value of `0` (a Space). The Stop Bit always has a value of `1` (a Mark). This means that there will always be a Mark (1) to Space (0) transition on the line at the start of every word, even when multiple word are transmitted back to back. This guarantees that sender and receiver can resynchronize their clocks regardless of the content of the data bits that are being transmitted. The idle time between Stop and Start bits does not have to be an exact multiple (including zero) of the bit rate of the communication link, but most UARTs are designed this way for simplicity. In RS232-C, the "Marking" signal (a `1`) is represented by a voltage between -2 VDC and -12 VDC, and a "Spacing" signal (a `0`) is represented by a voltage between 0 and +12 VDC. The transmitter is supposed to send +12 VDC or -12 VDC, and the receiver is supposed to allow for some voltage loss in long cables. Some transmitters in low power devices (like portable computers) sometimes use only +5 VDC and -5 VDC, but these values are still acceptable to a RS232-C receiver, provided that the cable lengths are short. ==== RS232-C Break Signal RS232-C also specifies a signal called a `Break`, which is caused by sending continuous Spacing values (no Start or Stop bits). When there is no electricity present on the data circuit, the line is considered to be sending `Break`. The `Break` signal must be of a duration longer than the time it takes to send a complete byte plus Start, Stop and Parity bits. Most UARTs can distinguish between a Framing Error and a Break, but if the UART cannot do this, the Framing Error detection can be used to identify Breaks. In the days of teleprinters, when numerous printers around the country were wired in series (such as news services), any unit could cause a `Break` by temporarily opening the entire circuit so that no current flowed. This was used to allow a location with urgent news to interrupt some other location that was currently sending information. In modern systems there are two types of Break signals. If the Break is longer than 1.6 seconds, it is considered a "Modem Break", and some modems can be programmed to terminate the conversation and go on-hook or enter the modems' command mode when the modem detects this signal. If the Break is smaller than 1.6 seconds, it signifies a Data Break and it is up to the remote computer to respond to this signal. Sometimes this form of Break is used as an Attention or Interrupt signal and sometimes is accepted as a substitute for the ASCII CONTROL-C character. Marks and Spaces are also equivalent to "Holes" and "No Holes" in paper tape systems. [NOTE] ==== Breaks cannot be generated from paper tape or from any other byte value, since bytes are always sent with Start and Stop bit. The UART is usually capable of generating the continuous Spacing signal in response to a special command from the host processor. ==== ==== RS232-C DTE and DCE Devices The RS232-C specification defines two types of equipment: the Data Terminal Equipment (DTE) and the Data Carrier Equipment (DCE). Usually, the DTE device is the terminal (or computer), and the DCE is a modem. Across the phone line at the other end of a conversation, the receiving modem is also a DCE device and the computer that is connected to that modem is a DTE device. The DCE device receives signals on the pins that the DTE device transmits on, and vice versa. When two devices that are both DTE or both DCE must be connected together without a modem or a similar media translator between them, a NULL modem must be used. The NULL modem electrically re-arranges the cabling so that the transmitter output is connected to the receiver input on the other device, and vice versa. Similar translations are performed on all of the control signals so that each device will see what it thinks are DCE (or DTE) signals from the other device. The number of signals generated by the DTE and DCE devices are not symmetrical. The DTE device generates fewer signals for the DCE device than the DTE device receives from the DCE. ==== RS232-C Pin Assignments The EIA RS232-C specification (and the ITU equivalent, V.24) calls for a twenty-five pin connector (usually a DB25) and defines the purpose of most of the pins in that connector. In the IBM Personal Computer and similar systems, a subset of RS232-C signals are provided via nine pin connectors (DB9). The signals that are not included on the PC connector deal mainly with synchronous operation, and this transmission mode is not supported by the UART that IBM selected for use in the IBM PC. Depending on the computer manufacturer, a DB25, a DB9, or both types of connector may be used for RS232-C communications. (The IBM PC also uses a DB25 connector for the parallel printer interface which causes some confusion.) Below is a table of the RS232-C signal assignments in the DB25 and DB9 connectors. [.informaltable] [cols="1,1,1,1,1,1,1", frame="none", options="header"] |=== | DB25 RS232-C Pin | DB9 IBM PC Pin | EIA Circuit Symbol | CCITT Circuit Symbol | Common Name | Signal Source | Description |1 |- |AA |101 |PG/FG |- |Frame/Protective Ground |2 |3 |BA |103 |TD |DTE |Transmit Data |3 |2 |BB |104 |RD |DCE |Receive Data |4 |7 |CA |105 |RTS |DTE |Request to Send |5 |8 |CB |106 |CTS |DCE |Clear to Send |6 |6 |CC |107 |DSR |DCE |Data Set Ready |7 |5 |AV |102 |SG/GND |- |Signal Ground |8 |1 |CF |109 |DCD/CD |DCE |Data Carrier Detect |9 |- |- |- |- |- |Reserved for Test |10 |- |- |- |- |- |Reserved for Test |11 |- |- |- |- |- |Reserved for Test |12 |- |CI |122 |SRLSD |DCE |Sec. Recv. Line Signal Detector |13 |- |SCB |121 |SCTS |DCE |Secondary Clear to Send |14 |- |SBA |118 |STD |DTE |Secondary Transmit Data |15 |- |DB |114 |TSET |DCE |Trans. Sig. Element Timing |16 |- |SBB |119 |SRD |DCE |Secondary Received Data |17 |- |DD |115 |RSET |DCE |Receiver Signal Element Timing |18 |- |- |141 |LOOP |DTE |Local Loopback |19 |- |SCA |120 |SRS |DTE |Secondary Request to Send |20 |4 |CD |108.2 |DTR |DTE |Data Terminal Ready |21 |- |- |- |RDL |DTE |Remote Digital Loopback |22 |9 |CE |125 |RI |DCE |Ring Indicator |23 |- |CH |111 |DSRS |DTE |Data Signal Rate Selector |24 |- |DA |113 |TSET |DTE |Trans. Sig. Element Timing |25 |- |- |142 |- |DCE |Test Mode |=== === Bits, Baud and Symbols Baud is a measurement of transmission speed in asynchronous communication. Due to advances in modem communication technology, this term is frequently misused when describing the data rates in newer devices. Traditionally, a Baud Rate represents the number of bits that are actually being sent over the media, not the amount of data that is actually moved from one DTE device to the other. The Baud count includes the overhead bits Start, Stop and Parity that are generated by the sending UART and removed by the receiving UART. This means that seven-bit words of data actually take 10 bits to be completely transmitted. Therefore, a modem capable of moving 300 bits per second from one place to another can normally only move 30 7-bit words if Parity is used and one Start and Stop bit are present. If 8-bit data words are used and Parity bits are also used, the data rate falls to 27.27 words per second, because it now takes 11 bits to send the eight-bit words, and the modem still only sends 300 bits per second. The formula for converting bytes per second into a baud rate and vice versa was simple until error-correcting modems came along. These modems receive the serial stream of bits from the UART in the host computer (even when internal modems are used the data is still frequently serialized) and converts the bits back into bytes. These bytes are then combined into packets and sent over the phone line using a Synchronous transmission method. This means that the Stop, Start, and Parity bits added by the UART in the DTE (the computer) were removed by the modem before transmission by the sending modem. When these bytes are received by the remote modem, the remote modem adds Start, Stop and Parity bits to the words, converts them to a serial format and then sends them to the receiving UART in the remote computer, who then strips the Start, Stop and Parity bits. The reason all these extra conversions are done is so that the two modems can perform error correction, which means that the receiving modem is able to ask the sending modem to resend a block of data that was not received with the correct checksum. This checking is handled by the modems, and the DTE devices are usually unaware that the process is occurring. By striping the Start, Stop and Parity bits, the additional bits of data that the two modems must share between themselves to perform error-correction are mostly concealed from the effective transmission rate seen by the sending and receiving DTE equipment. For example, if a modem sends ten 7-bit words to another modem without including the Start, Stop and Parity bits, the sending modem will be able to add 30 bits of its own information that the receiving modem can use to do error-correction without impacting the transmission speed of the real data. The use of the term Baud is further confused by modems that perform compression. A single 8-bit word passed over the telephone line might represent a dozen words that were transmitted to the sending modem. The receiving modem will expand the data back to its original content and pass that data to the receiving DTE. Modern modems also include buffers that allow the rate that bits move across the phone line (DCE to DCE) to be a different speed than the speed that the bits move between the DTE and DCE on both ends of the conversation. Normally the speed between the DTE and DCE is higher than the DCE to DCE speed because of the use of compression by the modems. As the number of bits needed to describe a byte varied during the trip between the two machines plus the differing bits-per-seconds speeds that are used present on the DTE-DCE and DCE-DCE links, the usage of the term Baud to describe the overall communication speed causes problems and can misrepresent the true transmission speed. So Bits Per Second (bps) is the correct term to use to describe the transmission rate seen at the DCE to DCE interface and Baud or Bits Per Second are acceptable terms to use when a connection is made between two systems with a wired connection, or if a modem is in use that is not performing error-correction or compression. Modern high speed modems (2400, 9600, 14,400, and 19,200bps) in reality still operate at or below 2400 baud, or more accurately, 2400 Symbols per second. High speed modem are able to encode more bits of data into each Symbol using a technique called Constellation Stuffing, which is why the effective bits per second rate of the modem is higher, but the modem continues to operate within the limited audio bandwidth that the telephone system provides. Modems operating at 28,800 and higher speeds have variable Symbol rates, but the technique is the same. === The IBM Personal Computer UART Starting with the original IBM Personal Computer, IBM selected the National Semiconductor INS8250 UART for use in the IBM PC Parallel/Serial Adapter. Subsequent generations of compatible computers from IBM and other vendors continued to use the INS8250 or improved versions of the National Semiconductor UART family. ==== National Semiconductor UART Family Tree There have been several versions and subsequent generations of the INS8250 UART. Each major version is described below. [.programlisting] .... INS8250 -> INS8250B \ \ \-> INS8250A -> INS82C50A \ \ \-> NS16450 -> NS16C450 \ \ \-> NS16550 -> NS16550A -> PC16550D .... INS8250:: This part was used in the original IBM PC and IBM PC/XT. The original name for this part was the INS8250 ACE (Asynchronous Communications Element) and it is made from NMOS technology. + The 8250 uses eight I/O ports and has a one-byte send and a one-byte receive buffer. This original UART has several race conditions and other flaws. The original IBM BIOS includes code to work around these flaws, but this made the BIOS dependent on the flaws being present, so subsequent parts like the 8250A, 16450 or 16550 could not be used in the original IBM PC or IBM PC/XT. INS8250-B:: This is the slower speed of the INS8250 made from NMOS technology. It contains the same problems as the original INS8250. INS8250A:: An improved version of the INS8250 using XMOS technology with various functional flaws corrected. The INS8250A was used initially in PC clone computers by vendors who used "clean" BIOS designs. Due to the corrections in the chip, this part could not be used with a BIOS compatible with the INS8250 or INS8250B. INS82C50A:: This is a CMOS version (low power consumption) of the INS8250A and has similar functional characteristics. NS16450:: Same as NS8250A with improvements so it can be used with faster CPU bus designs. IBM used this part in the IBM AT and updated the IBM BIOS to no longer rely on the bugs in the INS8250. NS16C450:: This is a CMOS version (low power consumption) of the NS16450. NS16550:: Same as NS16450 with a 16-byte send and receive buffer but the buffer design was flawed and could not be reliably be used. NS16550A:: Same as NS16550 with the buffer flaws corrected. The 16550A and its successors have become the most popular UART design in the PC industry, mainly due to its ability to reliably handle higher data rates on operating systems with sluggish interrupt response times. NS16C552:: This component consists of two NS16C550A CMOS UARTs in a single package. PC16550D:: Same as NS16550A with subtle flaws corrected. This is revision D of the 16550 family and is the latest design available from National Semiconductor. ==== The NS16550AF and the PC16550D are the same thing National reorganized their part numbering system a few years ago, and the NS16550AFN no longer exists by that name. (If you have a NS16550AFN, look at the date code on the part, which is a four digit number that usually starts with a nine. The first two digits of the number are the year, and the last two digits are the week in that year when the part was packaged. If you have a NS16550AFN, it is probably a few years old.) The new numbers are like PC16550DV, with minor differences in the suffix letters depending on the package material and its shape. (A description of the numbering system can be found below.) It is important to understand that in some stores, you may pay $15(US) for a NS16550AFN made in 1990 and in the next bin are the new PC16550DN parts with minor fixes that National has made since the AFN part was in production, the PC16550DN was probably made in the past six months and it costs half (as low as $5(US) in volume) as much as the NS16550AFN because they are readily available. As the supply of NS16550AFN chips continues to shrink, the price will probably continue to increase until more people discover and accept that the PC16550DN really has the same function as the old part number. ==== National Semiconductor Part Numbering System The older NS``__nnnnnrqp__`` part numbers are now of the format PC``__nnnnnrgp__``. The `_r_` is the revision field. The current revision of the 16550 from National Semiconductor is `D`. The `_p_` is the package-type field. The types are: [.informaltable] [cols="1,1,1", frame="none"] |=== |"F" |QFP |(quad flat pack) L lead type |"N" |DIP |(dual inline package) through hole straight lead type |"V" |LPCC |(lead plastic chip carrier) J lead type |=== The _g_ is the product grade field. If an `I` precedes the package-type letter, it indicates an "industrial" grade part, which has higher specs than a standard part but not as high as Military Specification (Milspec) component. This is an optional field. So what we used to call a NS16550AFN (DIP Package) is now called a PC16550DN or PC16550DIN. === Other Vendors and Similar UARTs Over the years, the 8250, 8250A, 16450 and 16550 have been licensed or copied by other chip vendors. In the case of the 8250, 8250A and 16450, the exact circuit (the "megacell") was licensed to many vendors, including Western Digital and Intel. Other vendors reverse-engineered the part or produced emulations that had similar behavior. In internal modems, the modem designer will frequently emulate the 8250A/16450 with the modem microprocessor, and the emulated UART will frequently have a hidden buffer consisting of several hundred bytes. Due to the size of the buffer, these emulations can be as reliable as a 16550A in their ability to handle high speed data. However, most operating systems will still report that the UART is only a 8250A or 16450, and may not make effective use of the extra buffering present in the emulated UART unless special drivers are used. Some modem makers are driven by market forces to abandon a design that has hundreds of bytes of buffer and instead use a 16550A UART so that the product will compare favorably in market comparisons even though the effective performance may be lowered by this action. A common misconception is that all parts with "16550A" written on them are identical in performance. There are differences, and in some cases, outright flaws in most of these 16550A clones. When the NS16550 was developed, the National Semiconductor obtained several patents on the design and they also limited licensing, making it harder for other vendors to provide a chip with similar features. As a result of the patents, reverse-engineered designs and emulations had to avoid infringing the claims covered by the patents. Subsequently, these copies almost never perform exactly the same as the NS16550A or PC16550D, which are the parts most computer and modem makers want to buy but are sometimes unwilling to pay the price required to get the genuine part. Some of the differences in the clone 16550A parts are unimportant, while others can prevent the device from being used at all with a given operating system or driver. These differences may show up when using other drivers, or when particular combinations of events occur that were not well tested or considered in the Windows(R) driver. This is because most modem vendors and 16550-clone makers use the Microsoft drivers from Windows(R) for Workgroups 3.11 and the Microsoft(R) MS-DOS(R) utility as the primary tests for compatibility with the NS16550A. This over-simplistic criteria means that if a different operating system is used, problems could appear due to subtle differences between the clones and genuine components. National Semiconductor has made available a program named COMTEST that performs compatibility tests independent of any OS drivers. It should be remembered that the purpose of this type of program is to demonstrate the flaws in the products of the competition, so the program will report major as well as extremely subtle differences in behavior in the part being tested. In a series of tests performed by the author of this document in 1994, components made by National Semiconductor, TI, StarTech, and CMD as well as megacells and emulations embedded in internal modems were tested with COMTEST. A difference count for some of these components is listed below. Since these tests were performed in 1994, they may not reflect the current performance of the given product from a vendor. It should be noted that COMTEST normally aborts when an excessive number or certain types of problems have been detected. As part of this testing, COMTEST was modified so that it would not abort no matter how many differences were encountered. [.informaltable] [cols="1,1,1", frame="none", options="header"] |=== | Vendor | Part Number | Errors (aka "differences" reported) |National |(PC16550DV) |0 |National |(NS16550AFN) |0 |National |(NS16C552V) |0 |TI |(TL16550AFN) |3 |CMD |(16C550PE) |19 |StarTech |(ST16C550J) |23 |Rockwell |Reference modem with internal 16550 or an emulation (RC144DPi/C3000-25) |117 |Sierra |Modem with an internal 16550 (SC11951/SC11351) |91 |=== [NOTE] ==== To date, the author of this document has not found any non-National parts that report zero differences using the COMTEST program. It should also be noted that National has had five versions of the 16550 over the years and the newest parts behave a bit differently than the classic NS16550AFN that is considered the benchmark for functionality. COMTEST appears to turn a blind eye to the differences within the National product line and reports no errors on the National parts (except for the original 16550) even when there are official erratas that describe bugs in the A, B and C revisions of the parts, so this bias in COMTEST must be taken into account. ==== It is important to understand that a simple count of differences from COMTEST does not reveal a lot about what differences are important and which are not. For example, about half of the differences reported in the two modems listed above that have internal UARTs were caused by the clone UARTs not supporting five- and six-bit character modes. The real 16550, 16450, and 8250 UARTs all support these modes and COMTEST checks the functionality of these modes so over fifty differences are reported. However, almost no modern modem supports five- or six-bit characters, particularly those with error-correction and compression capabilities. This means that the differences related to five- and six-bit character modes can be discounted. Many of the differences COMTEST reports have to do with timing. In many of the clone designs, when the host reads from one port, the status bits in some other port may not update in the same amount of time (some faster, some slower) as a _real_ NS16550AFN and COMTEST looks for these differences. This means that the number of differences can be misleading in that one device may only have one or two differences but they are extremely serious, and some other device that updates the status registers faster or slower than the reference part (that would probably never affect the operation of a properly written driver) could have dozens of differences reported. COMTEST can be used as a screening tool to alert the administrator to the presence of potentially incompatible components that might cause problems or have to be handled as a special case. If you run COMTEST on a 16550 that is in a modem or a modem is attached to the serial port, you need to first issue a ATE0&W command to the modem so that the modem will not echo any of the test characters. If you forget to do this, COMTEST will report at least this one difference: [source,shell] .... Error (6)...Timeout interrupt failed: IIR = c1 LSR = 61 .... === 8250/16450/16550 Registers The 8250/16450/16550 UART occupies eight contiguous I/O port addresses. In the IBM PC, there are two defined locations for these eight ports and they are known collectively as [.filename]#COM1# and [.filename]#COM2#. The makers of PC-clones and add-on cards have created two additional areas known as [.filename]#COM3# and [.filename]#COM4#, but these extra COM ports conflict with other hardware on some systems. The most common conflict is with video adapters that provide IBM 8514 emulation. [.filename]#COM1# is located from 0x3f8 to 0x3ff and normally uses IRQ 4. [.filename]#COM2# is located from 0x2f8 to 0x2ff and normally uses IRQ 3. [.filename]#COM3# is located from 0x3e8 to 0x3ef and has no standardized IRQ. [.filename]#COM4# is located from 0x2e8 to 0x2ef and has no standardized IRQ. A description of the I/O ports of the 8250/16450/16550 UART is provided below. [.informaltable] [cols="10%,10%,80%", frame="none", options="header"] |=== | I/O Port | Access Allowed | Description |+0x00 |write (DLAB==0) | Transmit Holding Register (THR). Information written to this port are treated as data words and will be transmitted by the UART. |+0x00 |read (DLAB==0) | Receive Buffer Register (RBR). Any data words received by the UART form the serial link are accessed by the host by reading this port. |+0x00 |write/read (DLAB==1) | Divisor Latch LSB (DLL) This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 0 thru 7 of the divisor. |+0x01 |write/read (DLAB==1) | Divisor Latch MSB (DLH) This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 8 thru 15 of the divisor. |+0x01 |write/read (DLAB==0) |Interrupt Enable Register (IER) + The 8250/16450/16550 UART classifies events into one of four categories. Each category can be configured to generate an interrupt when any of the events occurs. The 8250/16450/16550 UART generates a single external interrupt signal regardless of how many events in the enabled categories have occurred. It is up to the host processor to respond to the interrupt and then poll the enabled interrupt categories (usually all categories have interrupts enabled) to determine the true cause(s) of the interrupt. + Bit 7 -> Reserved, always 0. + Bit 6 -> Reserved, always 0. + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> Enable Modem Status Interrupt (EDSSI). Setting this bit to "1" allows the UART to generate an interrupt when a change occurs on one or more of the status lines. + Bit 2 -> Enable Receiver Line Status Interrupt (ELSI) Setting this bit to "1" causes the UART to generate an interrupt when the an error (or a BREAK signal) has been detected in the incoming data. + Bit 1 -> Enable Transmitter Holding Register Empty Interrupt (ETBEI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has room for one or more additional characters that are to be transmitted. + Bit 0 -> Enable Received Data Available Interrupt (ERBFI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has received enough characters to exceed the trigger level of the FIFO, or the FIFO timer has expired (stale data), or a single character has been received when the FIFO is disabled. |+0x02 |write |FIFO Control Register (FCR) (This port does not exist on the 8250 and 16450 UART.) + Bit 7 -> Receiver Trigger Bit #1 + Bit 6 -> Receiver Trigger Bit #0 + These two bits control at what point the receiver is to generate an interrupt when the FIFO is active. + 7 6 How many words are received before an interrupt is generated + 0 0 1 + 0 1 4 + 1 0 8 + 1 1 14 + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> DMA Mode Select. If Bit 0 is set to "1" (FIFOs enabled), setting this bit changes the operation of the -RXRDY and -TXRDY signals from Mode 0 to Mode 1. + Bit 2 -> Transmit FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being transmitted will be sent intact. This function is useful in aborting transfers. + Bit 1 -> Receiver FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being assembled in the shift register will be received intact. + Bit 0 -> 16550 FIFO Enable. When set, both the transmit and receive FIFOs are enabled. Any contents in the holding register, shift registers or FIFOs are lost when FIFOs are enabled or disabled. + |+0x02 |read |Interrupt Identification Register + Bit 7 -> FIFOs enabled. On the 8250/16450 UART, this bit is zero. + Bit 6 -> FIFOs enabled. On the 8250/16450 UART, this bit is zero. + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> Interrupt ID Bit #2. On the 8250/16450 UART, this bit is zero. + Bit 2 -> Interrupt ID Bit #1 + Bit 1 -> Interrupt ID Bit #0.These three bits combine to report the category of event that caused the interrupt that is in progress. These categories have priorities, so if multiple categories of events occur at the same time, the UART will report the more important events first and the host must resolve the events in the order they are reported. All events that caused the current interrupt must be resolved before any new interrupts will be generated. (This is a limitation of the PC architecture.) + 2 1 0 Priority Description + 0 1 1 First Received Error (OE, PE, BI, or FE) + 0 1 0 Second Received Data Available + 1 1 0 Second Trigger level identification (Stale data in receive buffer) + 0 0 1 Third Transmitter has room for more words (THRE) + 0 0 0 Fourth Modem Status Change (-CTS, -DSR, -RI, or -DCD) + Bit 0 -> Interrupt Pending Bit. If this bit is set to "0", then at least one interrupt is pending. |+0x03 |write/read |Line Control Register (LCR) + Bit 7 -> Divisor Latch Access Bit (DLAB). When set, access to the data transmit/receive register (THR/RBR) and the Interrupt Enable Register (IER) is disabled. Any access to these ports is now redirected to the Divisor Latch Registers. Setting this bit, loading the Divisor Registers, and clearing DLAB should be done with interrupts disabled. + Bit 6 -> Set Break. When set to "1", the transmitter begins to transmit continuous Spacing until this bit is set to "0". This overrides any bits of characters that are being transmitted. + Bit 5 -> Stick Parity. When parity is enabled, setting this bit causes parity to always be "1" or "0", based on the value of Bit 4. Bit 4 -> Even Parity Select (EPS). When parity is enabled and Bit 5 is "0", setting this bit causes even parity to be transmitted and expected. Otherwise, odd parity is used. + Bit 3 -> Parity Enable (PEN). When set to "1", a parity bit is inserted between the last bit of the data and the Stop Bit. The UART will also expect parity to be present in the received data. + Bit 2 -> Number of Stop Bits (STB). If set to "1" and using 5-bit data words, 1.5 Stop Bits are transmitted and expected in each data word. For 6, 7 and 8-bit data words, 2 Stop Bits are transmitted and expected. When this bit is set to "0", one Stop Bit is used on each data word. + Bit 1 -> Word Length Select Bit #1 (WLSB1) + Bit 0 -> Word Length Select Bit #0 (WLSB0) + Together these bits specify the number of bits in each data word. + 1 0 Word Length + 0 0 5 Data Bits + 0 1 6 Data Bits + 1 0 7 Data Bits + 1 1 8 Data Bits + |+0x04 |write/read |Modem Control Register (MCR) + Bit 7 -> Reserved, always 0. + Bit 6 -> Reserved, always 0. + Bit 5 -> Reserved, always 0. + Bit 4 -> Loop-Back Enable. When set to "1", the UART transmitter and receiver are internally connected together to allow diagnostic operations. In addition, the UART modem control outputs are connected to the UART modem control inputs. CTS is connected to RTS, DTR is connected to DSR, OUT1 is connected to RI, and OUT 2 is connected to DCD. + Bit 3 -> OUT 2. An auxiliary output that the host processor may set high or low. In the IBM PC serial adapter (and most clones), OUT 2 is used to tri-state (disable) the interrupt signal from the 8250/16450/16550 UART. + Bit 2 -> OUT 1. An auxiliary output that the host processor may set high or low. This output is not used on the IBM PC serial adapter. + Bit 1 -> Request to Send (RTS). When set to "1", the output of the UART -RTS line is Low (Active). + Bit 0 -> Data Terminal Ready (DTR). When set to "1", the output of the UART -DTR line is Low (Active). + |+0x05 |write/read |Line Status Register (LSR) + Bit 7 -> Error in Receiver FIFO. On the 8250/16450 UART, this bit is zero. This bit is set to "1" when any of the bytes in the FIFO have one or more of the following error conditions: PE, FE, or BI. + Bit 6 -> Transmitter Empty (TEMT). When set to "1", there are no words remaining in the transmit FIFO or the transmit shift register. The transmitter is completely idle. + Bit 5 -> Transmitter Holding Register Empty (THRE). When set to "1", the FIFO (or holding register) now has room for at least one additional word to transmit. The transmitter may still be transmitting when this bit is set to "1". + Bit 4 -> Break Interrupt (BI). The receiver has detected a Break signal. + Bit 3 -> Framing Error (FE). A Start Bit was detected but the Stop Bit did not appear at the expected time. The received word is probably garbled. + Bit 2 -> Parity Error (PE). The parity bit was incorrect for the word received. + Bit 1 -> Overrun Error (OE). A new word was received and there was no room in the receive buffer. The newly-arrived word in the shift register is discarded. On 8250/16450 UARTs, the word in the holding register is discarded and the newly- arrived word is put in the holding register. + Bit 0 -> Data Ready (DR) One or more words are in the receive FIFO that the host may read. A word must be completely received and moved from the shift register into the FIFO (or holding register for 8250/16450 designs) before this bit is set. |+0x06 |write/read |Modem Status Register (MSR) + Bit 7 -> Data Carrier Detect (DCD). Reflects the state of the DCD line on the UART. + Bit 6 -> Ring Indicator (RI). Reflects the state of the RI line on the UART. + Bit 5 -> Data Set Ready (DSR). Reflects the state of the DSR line on the UART. + Bit 4 -> Clear To Send (CTS). Reflects the state of the CTS line on the UART. + Bit 3 -> Delta Data Carrier Detect (DDCD). Set to "1" if the -DCD line has changed state one more time since the last time the MSR was read by the host. + Bit 2 -> Trailing Edge Ring Indicator (TERI). Set to "1" if the -RI line has had a low to high transition since the last time the MSR was read by the host. + Bit 1 -> Delta Data Set Ready (DDSR). Set to "1" if the -DSR line has changed state one more time since the last time the MSR was read by the host. + Bit 0 -> Delta Clear To Send (DCTS). Set to "1" if the -CTS line has changed state one more time since the last time the MSR was read by the host. + |+0x07 |write/read |Scratch Register (SCR). This register performs no function in the UART. Any value can be written by the host to this location and read by the host later on. |=== === Beyond the 16550A UART Although National Semiconductor has not offered any components compatible with the 16550 that provide additional features, various other vendors have. Some of these components are described below. It should be understood that to effectively utilize these improvements, drivers may have to be provided by the chip vendor since most of the popular operating systems do not support features beyond those provided by the 16550. ST16650:: By default this part is similar to the NS16550A, but an extended 32-byte send and receive buffer can be optionally enabled. Made by StarTech. TIL16660:: By default this part behaves similar to the NS16550A, but an extended 64-byte send and receive buffer can be optionally enabled. Made by Texas Instruments. Hayes ESP:: This proprietary plug-in card contains a 2048-byte send and receive buffer, and supports data rates to 230.4Kbit/sec. Made by Hayes. In addition to these "dumb" UARTs, many vendors produce intelligent serial communication boards. This type of design usually provides a microprocessor that interfaces with several UARTs, processes and buffers the data, and then alerts the main PC processor when necessary. As the UARTs are not directly accessed by the PC processor in this type of communication system, it is not necessary for the vendor to use UARTs that are compatible with the 8250, 16450, or the 16550 UART. This leaves the designer free to components that may have better performance characteristics. [[sio]] == Configuring the [.filename]#sio# driver The [.filename]#sio# driver provides support for NS8250-, NS16450-, NS16550 and NS16550A-based EIA RS-232C (CCITT V.24) communications interfaces. Several multiport cards are supported as well. See the man:sio[4] manual page for detailed technical documentation. === Digi International (DigiBoard) PC/8 _Contributed by `{awebster}`. 26 August 1995._ Here is a config snippet from a machine with a Digi International PC/8 with 16550. It has 8 modems connected to these 8 lines, and they work just great. Do not forget to add `options COM_MULTIPORT` or it will not work very well! [.programlisting] .... device sio4 at isa? port 0x100 flags 0xb05 device sio5 at isa? port 0x108 flags 0xb05 device sio6 at isa? port 0x110 flags 0xb05 device sio7 at isa? port 0x118 flags 0xb05 device sio8 at isa? port 0x120 flags 0xb05 device sio9 at isa? port 0x128 flags 0xb05 device sio10 at isa? port 0x130 flags 0xb05 device sio11 at isa? port 0x138 flags 0xb05 irq 9 .... The trick in setting this up is that the MSB of the flags represent the last SIO port, in this case 11 so flags are 0xb05. === Boca 16 _Contributed by `{whiteside}`. 26 August 1995._ The procedures to make a Boca 16 port board with FreeBSD are pretty straightforward, but you will need a couple things to make it work: . You either need the kernel sources installed so you can recompile the necessary options or you will need someone else to compile it for you. The 2.0.5 default kernel does _not_ come with multiport support enabled and you will need to add a device entry for each port anyways. . Two, you will need to know the interrupt and IO setting for your Boca Board so you can set these options properly in the kernel. One important note - the actual UART chips for the Boca 16 are in the connector box, not on the internal board itself. So if you have it unplugged, probes of those ports will fail. I have never tested booting with the box unplugged and plugging it back in, and I suggest you do not either. If you do not already have a custom kernel configuration file set up, refer to link:{handbook}#kernelconfig[Kernel Configuration] chapter of the FreeBSD Handbook for general procedures. The following are the specifics for the Boca 16 board and assume you are using the kernel name MYKERNEL and editing with vi. [.procedure] ==== . Add the line + [.programlisting] .... options COM_MULTIPORT .... to the config file. . Where the current `device sio__n__` lines are, you will need to add 16 more devices. The following example is for a Boca Board with an interrupt of 3, and a base IO address 100h. The IO address for Each port is +8 hexadecimal from the previous port, thus the 100h, 108h, 110h... addresses. + [.programlisting] .... device sio1 at isa? port 0x100 flags 0x1005 device sio2 at isa? port 0x108 flags 0x1005 device sio3 at isa? port 0x110 flags 0x1005 device sio4 at isa? port 0x118 flags 0x1005 ... device sio15 at isa? port 0x170 flags 0x1005 device sio16 at isa? port 0x178 flags 0x1005 irq 3 .... + The flags entry _must_ be changed from this example unless you are using the exact same sio assignments. Flags are set according to 0x``__MYY__`` where _M_ indicates the minor number of the master port (the last port on a Boca 16) and _YY_ indicates if FIFO is enabled or disabled(enabled), IRQ sharing is used(yes) and if there is an AST/4 compatible IRQ control register(no). In this example, + [.programlisting] .... flags 0x1005 .... indicates that the master port is sio16. If I added another board and assigned sio17 through sio28, the flags for all 16 ports on _that_ board would be 0x1C05, where 1C indicates the minor number of the master port. Do not change the 05 setting. . Save and complete the kernel configuration, recompile, install and reboot. Presuming you have successfully installed the recompiled kernel and have it set to the correct address and IRQ, your boot message should indicate the successful probe of the Boca ports as follows: (obviously the sio numbers, IO and IRQ could be different) + [source,shell] .... sio1 at 0x100-0x107 flags 0x1005 on isa sio1: type 16550A (multiport) sio2 at 0x108-0x10f flags 0x1005 on isa sio2: type 16550A (multiport) sio3 at 0x110-0x117 flags 0x1005 on isa sio3: type 16550A (multiport) sio4 at 0x118-0x11f flags 0x1005 on isa sio4: type 16550A (multiport) sio5 at 0x120-0x127 flags 0x1005 on isa sio5: type 16550A (multiport) sio6 at 0x128-0x12f flags 0x1005 on isa sio6: type 16550A (multiport) sio7 at 0x130-0x137 flags 0x1005 on isa sio7: type 16550A (multiport) sio8 at 0x138-0x13f flags 0x1005 on isa sio8: type 16550A (multiport) sio9 at 0x140-0x147 flags 0x1005 on isa sio9: type 16550A (multiport) sio10 at 0x148-0x14f flags 0x1005 on isa sio10: type 16550A (multiport) sio11 at 0x150-0x157 flags 0x1005 on isa sio11: type 16550A (multiport) sio12 at 0x158-0x15f flags 0x1005 on isa sio12: type 16550A (multiport) sio13 at 0x160-0x167 flags 0x1005 on isa sio13: type 16550A (multiport) sio14 at 0x168-0x16f flags 0x1005 on isa sio14: type 16550A (multiport) sio15 at 0x170-0x177 flags 0x1005 on isa sio15: type 16550A (multiport) sio16 at 0x178-0x17f irq 3 flags 0x1005 on isa sio16: type 16550A (multiport master) .... + If the messages go by too fast to see, + [source,shell] .... # dmesg | more .... will show you the boot messages. . Next, appropriate entries in [.filename]#/dev# for the devices must be made using the [.filename]#/dev/MAKEDEV# script. This step can be omitted if you are running FreeBSD 5.X with a kernel that has man:devfs[5] support compiled in. + If you do need to create the [.filename]#/dev# entries, run the following as `root`: + [source,shell] .... # cd /dev # ./MAKEDEV tty1 # ./MAKEDEV cua1 (everything in between) # ./MAKEDEV ttyg # ./MAKEDEV cuag .... + If you do not want or need call-out devices for some reason, you can dispense with making the [.filename]#cua*# devices. . If you want a quick and sloppy way to make sure the devices are working, you can simply plug a modem into each port and (as root) + [source,shell] .... # echo at > ttyd* .... for each device you have made. You _should_ see the RX lights flash for each working port. ==== === Support for Cheap Multi-UART Cards _Contributed by Helge Oldach_ mailto:hmo@sep.hamburg.com[hmo@sep.hamburg.com], September 1999 Ever wondered about FreeBSD support for your 20$ multi-I/O card with two (or more) COM ports, sharing IRQs? Here is how: Usually the only option to support these kind of boards is to use a distinct IRQ for each port. For example, if your CPU board has an on-board [.filename]#COM1# port (aka [.filename]#sio0#-I/O address 0x3F8 and IRQ 4) and you have an extension board with two UARTs, you will commonly need to configure them as [.filename]#COM2# (aka [.filename]#sio1#-I/O address 0x2F8 and IRQ 3), and the third port (aka [.filename]#sio2#) as I/O 0x3E8 and IRQ 5. Obviously this is a waste of IRQ resources, as it should be basically possible to run both extension board ports using a single IRQ with the `COM_MULTIPORT` configuration described in the previous sections. Such cheap I/O boards commonly have a 4 by 3 jumper matrix for the COM ports, similar to the following: [.programlisting] .... o o o * Port A | o * o * Port B | o * o o IRQ 2 3 4 5 .... Shown here is port A wired for IRQ 5 and port B wired for IRQ 3. The IRQ columns on your specific board may vary-other boards may supply jumpers for IRQs 3, 4, 5, and 7 instead. One could conclude that wiring both ports for IRQ 3 using a handcrafted wire-made jumper covering all three connection points in the IRQ 3 column would solve the issue, but no. You cannot duplicate IRQ 3 because the output drivers of each UART are wired in a "totem pole" fashion, so if one of the UARTs drives IRQ 3, the output signal will not be what you would expect. Depending on the implementation of the extension board or your motherboard, the IRQ 3 line will continuously stay up, or always stay low. You need to decouple the IRQ drivers for the two UARTs, so that the IRQ line of the board only goes up if (and only if) one of the UARTs asserts a IRQ, and stays low otherwise. The solution was proposed by Joerg Wunsch mailto:j@ida.interface-business.de[j@ida.interface-business.de]: To solder up a wired-or consisting of two diodes (Germanium or Schottky-types strongly preferred) and a 1 kOhm resistor. Here is the schematic, starting from the 4 by 3 jumper field above: [.programlisting] .... Diode +---------->|-------+ / | o * o o | 1 kOhm Port A +----|######|-------+ o * o o | | Port B `-------------------+ ==+== o * o o | Ground \ | +--------->|-------+ IRQ 2 3 4 5 Diode .... The cathodes of the diodes are connected to a common point, together with a 1 kOhm pull-down resistor. It is essential to connect the resistor to ground to avoid floating of the IRQ line on the bus. Now we are ready to configure a kernel. Staying with this example, we would configure: [.programlisting] .... # standard on-board COM1 port device sio0 at isa? port "IO_COM1" flags 0x10 # patched-up multi-I/O extension board options COM_MULTIPORT device sio1 at isa? port "IO_COM2" flags 0x205 device sio2 at isa? port "IO_COM3" flags 0x205 irq 3 .... Note that the `flags` setting for [.filename]#sio1# and [.filename]#sio2# is truly essential; refer to man:sio[4] for details. (Generally, the `2` in the "flags" attribute refers to [.filename]#sio#`2` which holds the IRQ, and you surely want a `5` low nibble.) With kernel verbose mode turned on this should yield something similar to this: [source,shell] .... sio0: irq maps: 0x1 0x11 0x1 0x1 sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A sio1: irq maps: 0x1 0x9 0x1 0x1 sio1 at 0x2f8-0x2ff flags 0x205 on isa sio1: type 16550A (multiport) sio2: irq maps: 0x1 0x9 0x1 0x1 sio2 at 0x3e8-0x3ef irq 3 flags 0x205 on isa sio2: type 16550A (multiport master) .... Though [.filename]#/sys/i386/isa/sio.c# is somewhat cryptic with its use of the "irq maps" array above, the basic idea is that you observe `0x1` in the first, third, and fourth place. This means that the corresponding IRQ was set upon output and cleared after, which is just what we would expect. If your kernel does not display this behavior, most likely there is something wrong with your wiring. [[cy]] == Configuring the [.filename]#cy# driver _Contributed by Alex Nash. 6 June 1996._ The Cyclades multiport cards are based on the [.filename]#cy# driver instead of the usual [.filename]#sio# driver used by other multiport cards. Configuration is a simple matter of: [.procedure] ==== . Add the [.filename]#cy# device to your kernel configuration (note that your irq and iomem settings may differ). + [.programlisting] .... device cy0 at isa? irq 10 iomem 0xd4000 iosiz 0x2000 .... . Rebuild and install the new kernel. . Make the device nodes by typing (the following example assumes an 8-port board): + [source,shell] .... # cd /dev # for i in 0 1 2 3 4 5 6 7;do ./MAKEDEV cuac$i ttyc$i;done .... . If appropriate, add dialup entries to [.filename]#/etc/ttys# by duplicating serial device (`ttyd`) entries and using `ttyc` in place of `ttyd`. For example: + [.programlisting] .... ttyc0 "/usr/libexec/getty std.38400" unknown on insecure ttyc1 "/usr/libexec/getty std.38400" unknown on insecure ttyc2 "/usr/libexec/getty std.38400" unknown on insecure ... ttyc7 "/usr/libexec/getty std.38400" unknown on insecure .... . Reboot with the new kernel. ==== == Configuring the [.filename]#si# driver _Contributed by `{nsayer}`. 25 March 1998._ The Specialix SI/XIO and SX multiport cards use the [.filename]#si# driver. A single machine can have up to 4 host cards. The following host cards are supported: * ISA SI/XIO host card (2 versions) * EISA SI/XIO host card * PCI SI/XIO host card * ISA SX host card * PCI SX host card Although the SX and SI/XIO host cards look markedly different, their functionality are basically the same. The host cards do not use I/O locations, but instead require a 32K chunk of memory. The factory configuration for ISA cards places this at `0xd0000-0xd7fff`. They also require an IRQ. PCI cards will, of course, auto-configure themselves. You can attach up to 4 external modules to each host card. The external modules contain either 4 or 8 serial ports. They come in the following varieties: * SI 4 or 8 port modules. Up to 57600 bps on each port supported. * XIO 8 port modules. Up to 115200 bps on each port supported. One type of XIO module has 7 serial and 1 parallel port. * SXDC 8 port modules. Up to 921600 bps on each port supported. Like XIO, a module is available with one parallel port as well. To configure an ISA host card, add the following line to your kernel configuration file, changing the numbers as appropriate: [.programlisting] .... device si0 at isa? iomem 0xd0000 irq 11 .... Valid IRQ numbers are 9, 10, 11, 12 and 15 for SX ISA host cards and 11, 12 and 15 for SI/XIO ISA host cards. To configure an EISA or PCI host card, use this line: [.programlisting] .... device si0 .... After adding the configuration entry, rebuild and install your new kernel. [NOTE] ==== The following step, is not necessary if you are using man:devfs[5] in FreeBSD 5._X_. ==== After rebooting with the new kernel, you need to make the device nodes in [.filename]#/dev#. The [.filename]#MAKEDEV# script will take care of this for you. Count how many total ports you have and type: [source,shell] .... # cd /dev # ./MAKEDEV ttyAnn cuaAnn .... (where _nn_ is the number of ports) If you want login prompts to appear on these ports, you will need to add lines like this to [.filename]#/etc/ttys#: [.programlisting] .... ttyA01 "/usr/libexec/getty std.9600" vt100 on insecure .... Change the terminal type as appropriate. For modems, `dialup` or `unknown` is fine. diff --git a/documentation/content/en/articles/vinum/_index.adoc b/documentation/content/en/articles/vinum/_index.adoc index d6a662b7c4..6b2db7cf8f 100644 --- a/documentation/content/en/articles/vinum/_index.adoc +++ b/documentation/content/en/articles/vinum/_index.adoc @@ -1,699 +1,700 @@ --- title: The vinum Volume Manager authors: - author: Greg Lehey description: The vinum Volume Manager in FreeBSD tags: ["vinum", "Volume Manager", "FreeBSD"] --- = The vinum Volume Manager :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: -include::shared/en/urls.adoc[] ifeval::["{backend}" == "html5"] +include::shared/en/urls.adoc[] :imagesdir: ../../../images/articles/vinum/ endif::[] ifeval::["{backend}" == "pdf"] +include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/articles/vinum/ endif::[] ifeval::["{backend}" == "epub3"] :imagesdir: ../../../../static/images/articles/vinum/ endif::[] ''' toc::[] [[vinum-synopsis]] == Synopsis No matter the type of disks, there are always potential problems. The disks can be too small, too slow, or too unreliable to meet the system's requirements. While disks are getting bigger, so are data storage requirements. Often a file system is needed that is bigger than a disk's capacity. Various solutions to these problems have been proposed and implemented. One method is through the use of multiple, and sometimes redundant, disks. In addition to supporting various cards and controllers for hardware Redundant Array of Independent Disks RAID systems, the base FreeBSD system includes the [.filename]#vinum# volume manager, a block device driver that implements virtual disk drives and addresses these three problems. [.filename]#vinum# provides more flexibility, performance, and reliability than traditional disk storage and implements `RAID`-0, `RAID`-1, and `RAID`-5 models, both individually and in combination. This chapter provides an overview of potential problems with traditional disk storage, and an introduction to the [.filename]#vinum# volume manager. [NOTE] ==== Starting with FreeBSD 5, [.filename]#vinum# has been rewritten in order to fit into the link:{handbook}#geom[GEOM architecture], while retaining the original ideas, terminology, and on-disk metadata. This rewrite is called _gvinum_ (for _GEOM vinum_). While this chapter uses the term [.filename]#vinum#, any command invocations should be performed with `gvinum`. The name of the kernel module has changed from the original [.filename]#vinum.ko# to [.filename]#geom_vinum.ko#, and all device nodes reside under [.filename]#/dev/gvinum# instead of [.filename]#/dev/vinum#. As of FreeBSD 6, the original [.filename]#vinum# implementation is no longer available in the code base. ==== [[vinum-access-bottlenecks]] == Access Bottlenecks Modern systems frequently need to access data in a highly concurrent manner. For example, large FTP or HTTP servers can maintain thousands of concurrent sessions and have multiple 100 Mbit/s connections to the outside world, well beyond the sustained transfer rate of most disks. Current disk drives can transfer data sequentially at up to 70 MB/s, but this value is of little importance in an environment where many independent processes access a drive, and where they may achieve only a fraction of these values. In such cases, it is more interesting to view the problem from the viewpoint of the disk subsystem. The important parameter is the load that a transfer places on the subsystem, or the time for which a transfer occupies the drives involved in the transfer. In any disk transfer, the drive must first position the heads, wait for the first sector to pass under the read head, and then perform the transfer. These actions can be considered to be atomic as it does not make any sense to interrupt them. [[vinum-latency]] Consider a typical transfer of about 10 kB: the current generation of high-performance disks can position the heads in an average of 3.5 ms. The fastest drives spin at 15,000 rpm, so the average rotational latency (half a revolution) is 2 ms. At 70 MB/s, the transfer itself takes about 150 μs, almost nothing compared to the positioning time. In such a case, the effective transfer rate drops to a little over 1 MB/s and is clearly highly dependent on the transfer size. The traditional and obvious solution to this bottleneck is "more spindles": rather than using one large disk, use several smaller disks with the same aggregate storage space. Each disk is capable of positioning and transferring independently, so the effective throughput increases by a factor close to the number of disks used. The actual throughput improvement is smaller than the number of disks involved. Although each drive is capable of transferring in parallel, there is no way to ensure that the requests are evenly distributed across the drives. Inevitably the load on one drive will be higher than on another. The evenness of the load on the disks is strongly dependent on the way the data is shared across the drives. In the following discussion, it is convenient to think of the disk storage as a large number of data sectors which are addressable by number, rather like the pages in a book. The most obvious method is to divide the virtual disk into groups of consecutive sectors the size of the individual physical disks and store them in this manner, rather like taking a large book and tearing it into smaller sections. This method is called _concatenation_ and has the advantage that the disks are not required to have any specific size relationships. It works well when the access to the virtual disk is spread evenly about its address space. When access is concentrated on a smaller area, the improvement is less marked. <> illustrates the sequence in which storage units are allocated in a concatenated organization. [[vinum-concat]] .Concatenated Organization image::vinum-concat.png[] An alternative mapping is to divide the address space into smaller, equal-sized components and store them sequentially on different devices. For example, the first 256 sectors may be stored on the first disk, the next 256 sectors on the next disk and so on. After filling the last disk, the process repeats until the disks are full. This mapping is called _striping_ or RAID-0. `RAID` offers various forms of fault tolerance, though RAID-0 is somewhat misleading as it provides no redundancy. Striping requires somewhat more effort to locate the data, and it can cause additional I/O load where a transfer is spread over multiple disks, but it can also provide a more constant load across the disks. <> illustrates the sequence in which storage units are allocated in a striped organization. [[vinum-striped]] .Striped Organization image::vinum-striped.png[] [[vinum-data-integrity]] == Data Integrity The final problem with disks is that they are unreliable. Although reliability has increased tremendously over the last few years, disk drives are still the most likely core component of a server to fail. When they do, the results can be catastrophic and replacing a failed disk drive and restoring data can result in server downtime. One approach to this problem is _mirroring_, or `RAID-1`, which keeps two copies of the data on different physical hardware. Any write to the volume writes to both disks; a read can be satisfied from either, so if one drive fails, the data is still available on the other drive. Mirroring has two problems: * It requires twice as much disk storage as a non-redundant solution. * Writes must be performed to both drives, so they take up twice the bandwidth of a non-mirrored volume. Reads do not suffer from a performance penalty and can even be faster. An alternative solution is _parity_, implemented in `RAID` levels 2, 3, 4 and 5. Of these, `RAID-5` is the most interesting. As implemented in [.filename]#vinum#, it is a variant on a striped organization which dedicates one block of each stripe to parity one of the other blocks. As implemented by [.filename]#vinum#, a `RAID-5` plex is similar to a striped plex, except that it implements `RAID-5` by including a parity block in each stripe. As required by `RAID-5`, the location of this parity block changes from one stripe to the next. The numbers in the data blocks indicate the relative block numbers. [[vinum-raid5-org]] .`RAID`-5 Organization image::vinum-raid5-org.png[] Compared to mirroring, `RAID-5` has the advantage of requiring significantly less storage space. Read access is similar to that of striped organizations, but write access is significantly slower, approximately 25% of the read performance. If one drive fails, the array can continue to operate in degraded mode where a read from one of the remaining accessible drives continues normally, but a read from the failed drive is recalculated from the corresponding block from all the remaining drives. [[vinum-objects]] == [.filename]#vinum# Objects In order to address these problems, [.filename]#vinum# implements a four-level hierarchy of objects: * The most visible object is the virtual disk, called a _volume_. Volumes have essentially the same properties as a UNIX(R) disk drive, though there are some minor differences. For one, they have no size limitations. * Volumes are composed of _plexes_, each of which represent the total address space of a volume. This level in the hierarchy provides redundancy. Think of plexes as individual disks in a mirrored array, each containing the same data. * Since [.filename]#vinum# exists within the UNIX(R) disk storage framework, it would be possible to use UNIX(R) partitions as the building block for multi-disk plexes. In fact, this turns out to be too inflexible as UNIX(R) disks can have only a limited number of partitions. Instead, [.filename]#vinum# subdivides a single UNIX(R) partition, the _drive_, into contiguous areas called _subdisks_, which are used as building blocks for plexes. * Subdisks reside on [.filename]#vinum#_drives_, currently UNIX(R) partitions. [.filename]#vinum# drives can contain any number of subdisks. With the exception of a small area at the beginning of the drive, which is used for storing configuration and state information, the entire drive is available for data storage. The following sections describe the way these objects provide the functionality required of [.filename]#vinum#. === Volume Size Considerations Plexes can include multiple subdisks spread over all drives in the [.filename]#vinum# configuration. As a result, the size of an individual drive does not limit the size of a plex or a volume. === Redundant Data Storage [.filename]#vinum# implements mirroring by attaching multiple plexes to a volume. Each plex is a representation of the data in a volume. A volume may contain between one and eight plexes. Although a plex represents the complete data of a volume, it is possible for parts of the representation to be physically missing, either by design (by not defining a subdisk for parts of the plex) or by accident (as a result of the failure of a drive). As long as at least one plex can provide the data for the complete address range of the volume, the volume is fully functional. === Which Plex Organization? [.filename]#vinum# implements both concatenation and striping at the plex level: * A _concatenated plex_ uses the address space of each subdisk in turn. Concatenated plexes are the most flexible as they can contain any number of subdisks, and the subdisks may be of different length. The plex may be extended by adding additional subdisks. They require less CPU time than striped plexes, though the difference in CPU overhead is not measurable. On the other hand, they are most susceptible to hot spots, where one disk is very active and others are idle. * A _striped plex_ stripes the data across each subdisk. The subdisks must all be the same size and there must be at least two subdisks in order to distinguish it from a concatenated plex. The greatest advantage of striped plexes is that they reduce hot spots. By choosing an optimum sized stripe, about 256 kB, the load can be evened out on the component drives. Extending a plex by adding new subdisks is so complicated that [.filename]#vinum# does not implement it. <> summarizes the advantages and disadvantages of each plex organization. [[vinum-comparison]] .[.filename]#vinum# Plex Organizations [cols="1,1,1,1,1", frame="none", options="header"] |=== | Plex type | Minimum subdisks | Can add subdisks | Must be equal size | Application |concatenated |1 |yes |no |Large data storage with maximum placement flexibility and moderate performance |striped |2 |no |yes |High performance in combination with highly concurrent access |=== [[vinum-examples]] == Some Examples [.filename]#vinum# maintains a _configuration database_ which describes the objects known to an individual system. Initially, the user creates the configuration database from one or more configuration files using man:gvinum[8]. [.filename]#vinum# stores a copy of its configuration database on each disk _device_ under its control. This database is updated on each state change, so that a restart accurately restores the state of each [.filename]#vinum# object. === The Configuration File The configuration file describes individual [.filename]#vinum# objects. The definition of a simple volume might be: [.programlisting] .... drive a device /dev/da3h volume myvol plex org concat sd length 512m drive a .... This file describes four [.filename]#vinum# objects: * The _drive_ line describes a disk partition (_drive_) and its location relative to the underlying hardware. It is given the symbolic name _a_. This separation of symbolic names from device names allows disks to be moved from one location to another without confusion. * The _volume_ line describes a volume. The only required attribute is the name, in this case _myvol_. * The _plex_ line defines a plex. The only required parameter is the organization, in this case _concat_. No name is necessary as the system automatically generates a name from the volume name by adding the suffix _.px_, where _x_ is the number of the plex in the volume. Thus this plex will be called _myvol.p0_. * The _sd_ line describes a subdisk. The minimum specifications are the name of a drive on which to store it, and the length of the subdisk. No name is necessary as the system automatically assigns names derived from the plex name by adding the suffix _.sx_, where _x_ is the number of the subdisk in the plex. Thus [.filename]#vinum# gives this subdisk the name _myvol.p0.s0_. After processing this file, man:gvinum[8] produces the following output: [.programlisting] .... # gvinum -> create config1 Configuration summary Drives: 1 (4 configured) Volumes: 1 (4 configured) Plexes: 1 (8 configured) Subdisks: 1 (16 configured) D a State: up Device /dev/da3h Avail: 2061/2573 MB (80%) V myvol State: up Plexes: 1 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB .... This output shows the brief listing format of man:gvinum[8]. It is represented graphically in <>. [[vinum-simple-vol]] .A Simple [.filename]#vinum# Volume image::vinum-simple-vol.png[] This figure, and the ones which follow, represent a volume, which contains the plexes, which in turn contains the subdisks. In this example, the volume contains one plex, and the plex contains one subdisk. This particular volume has no specific advantage over a conventional disk partition. It contains a single plex, so it is not redundant. The plex contains a single subdisk, so there is no difference in storage allocation from a conventional disk partition. The following sections illustrate various more interesting configuration methods. === Increased Resilience: Mirroring The resilience of a volume can be increased by mirroring. When laying out a mirrored volume, it is important to ensure that the subdisks of each plex are on different drives, so that a drive failure will not take down both plexes. The following configuration mirrors a volume: [.programlisting] .... drive b device /dev/da4h volume mirror plex org concat sd length 512m drive a plex org concat sd length 512m drive b .... In this example, it was not necessary to specify a definition of drive _a_ again, since [.filename]#vinum# keeps track of all objects in its configuration database. After processing this definition, the configuration looks like: [.programlisting] .... Drives: 2 (4 configured) Volumes: 2 (4 configured) Plexes: 3 (8 configured) Subdisks: 3 (16 configured) D a State: up Device /dev/da3h Avail: 1549/2573 MB (60%) D b State: up Device /dev/da4h Avail: 2061/2573 MB (80%) V myvol State: up Plexes: 1 Size: 512 MB V mirror State: up Plexes: 2 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p1 C State: initializing Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p1.s0 State: empty PO: 0 B Size: 512 MB .... <> shows the structure graphically. [[vinum-mirrored-vol]] .A Mirrored [.filename]#vinum# Volume image::vinum-mirrored-vol.png[] In this example, each plex contains the full 512 MB of address space. As in the previous example, each plex contains only a single subdisk. === Optimizing Performance The mirrored volume in the previous example is more resistant to failure than an unmirrored volume, but its performance is less as each write to the volume requires a write to both drives, using up a greater proportion of the total disk bandwidth. Performance considerations demand a different approach: instead of mirroring, the data is striped across as many disk drives as possible. The following configuration shows a volume with a plex striped across four disk drives: [.programlisting] .... drive c device /dev/da5h drive d device /dev/da6h volume stripe plex org striped 512k sd length 128m drive a sd length 128m drive b sd length 128m drive c sd length 128m drive d .... As before, it is not necessary to define the drives which are already known to [.filename]#vinum#. After processing this definition, the configuration looks like: [.programlisting] .... Drives: 4 (4 configured) Volumes: 3 (4 configured) Plexes: 4 (8 configured) Subdisks: 7 (16 configured) D a State: up Device /dev/da3h Avail: 1421/2573 MB (55%) D b State: up Device /dev/da4h Avail: 1933/2573 MB (75%) D c State: up Device /dev/da5h Avail: 2445/2573 MB (95%) D d State: up Device /dev/da6h Avail: 2445/2573 MB (95%) V myvol State: up Plexes: 1 Size: 512 MB V mirror State: up Plexes: 2 Size: 512 MB V striped State: up Plexes: 1 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p1 C State: initializing Subdisks: 1 Size: 512 MB P striped.p1 State: up Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p1.s0 State: empty PO: 0 B Size: 512 MB S striped.p0.s0 State: up PO: 0 B Size: 128 MB S striped.p0.s1 State: up PO: 512 kB Size: 128 MB S striped.p0.s2 State: up PO: 1024 kB Size: 128 MB S striped.p0.s3 State: up PO: 1536 kB Size: 128 MB .... [[vinum-striped-vol]] .A Striped [.filename]#vinum# Volume image::vinum-striped-vol.png[] This volume is represented in <>. The darkness of the stripes indicates the position within the plex address space, where the lightest stripes come first and the darkest last. === Resilience and Performance [[vinum-resilience]]With sufficient hardware, it is possible to build volumes which show both increased resilience and increased performance compared to standard UNIX(R) partitions. A typical configuration file might be: [.programlisting] .... volume raid10 plex org striped 512k sd length 102480k drive a sd length 102480k drive b sd length 102480k drive c sd length 102480k drive d sd length 102480k drive e plex org striped 512k sd length 102480k drive c sd length 102480k drive d sd length 102480k drive e sd length 102480k drive a sd length 102480k drive b .... The subdisks of the second plex are offset by two drives from those of the first plex. This helps to ensure that writes do not go to the same subdisks even if a transfer goes over two drives. <> represents the structure of this volume. [[vinum-raid10-vol]] .A Mirrored, Striped [.filename]#vinum# Volume image::vinum-raid10-vol.png[] [[vinum-object-naming]] == Object Naming [.filename]#vinum# assigns default names to plexes and subdisks, although they may be overridden. Overriding the default names is not recommended as it does not bring a significant advantage and it can cause confusion. Names may contain any non-blank character, but it is recommended to restrict them to letters, digits and the underscore characters. The names of volumes, plexes, and subdisks may be up to 64 characters long, and the names of drives may be up to 32 characters long. [.filename]#vinum# objects are assigned device nodes in the hierarchy [.filename]#/dev/gvinum#. The configuration shown above would cause [.filename]#vinum# to create the following device nodes: * Device entries for each volume. These are the main devices used by [.filename]#vinum#. The configuration above would include the devices [.filename]#/dev/gvinum/myvol#, [.filename]#/dev/gvinum/mirror#, [.filename]#/dev/gvinum/striped#, [.filename]#/dev/gvinum/raid5# and [.filename]#/dev/gvinum/raid10#. * All volumes get direct entries under [.filename]#/dev/gvinum/#. * The directories [.filename]#/dev/gvinum/plex#, and [.filename]#/dev/gvinum/sd#, which contain device nodes for each plex and for each subdisk, respectively. For example, consider the following configuration file: [.programlisting] .... drive drive1 device /dev/sd1h drive drive2 device /dev/sd2h drive drive3 device /dev/sd3h drive drive4 device /dev/sd4h volume s64 setupstate plex org striped 64k sd length 100m drive drive1 sd length 100m drive drive2 sd length 100m drive drive3 sd length 100m drive drive4 .... After processing this file, man:gvinum[8] creates the following structure in [.filename]#/dev/gvinum#: [.programlisting] .... drwxr-xr-x 2 root wheel 512 Apr 13 16:46 plex crwxr-xr-- 1 root wheel 91, 2 Apr 13 16:46 s64 drwxr-xr-x 2 root wheel 512 Apr 13 16:46 sd /dev/vinum/plex: total 0 crwxr-xr-- 1 root wheel 25, 0x10000002 Apr 13 16:46 s64.p0 /dev/vinum/sd: total 0 crwxr-xr-- 1 root wheel 91, 0x20000002 Apr 13 16:46 s64.p0.s0 crwxr-xr-- 1 root wheel 91, 0x20100002 Apr 13 16:46 s64.p0.s1 crwxr-xr-- 1 root wheel 91, 0x20200002 Apr 13 16:46 s64.p0.s2 crwxr-xr-- 1 root wheel 91, 0x20300002 Apr 13 16:46 s64.p0.s3 .... Although it is recommended that plexes and subdisks should not be allocated specific names, [.filename]#vinum# drives must be named. This makes it possible to move a drive to a different location and still recognize it automatically. Drive names may be up to 32 characters long. === Creating File Systems Volumes appear to the system to be identical to disks, with one exception. Unlike UNIX(R) drives, [.filename]#vinum# does not partition volumes, which thus do not contain a partition table. This has required modification to some disk utilities, notably man:newfs[8], so that it does not try to interpret the last letter of a [.filename]#vinum# volume name as a partition identifier. For example, a disk drive may have a name like [.filename]#/dev/ad0a# or [.filename]#/dev/da2h#. These names represent the first partition ([.filename]#a#) on the first (0) IDE disk ([.filename]#ad#) and the eighth partition ([.filename]#h#) on the third (2) SCSI disk ([.filename]#da#) respectively. By contrast, a [.filename]#vinum# volume might be called [.filename]#/dev/gvinum/concat#, which has no relationship with a partition name. In order to create a file system on this volume, use man:newfs[8]: [source,shell] .... # newfs /dev/gvinum/concat .... [[vinum-config]] == Configuring [.filename]#vinum# The [.filename]#GENERIC# kernel does not contain [.filename]#vinum#. It is possible to build a custom kernel which includes [.filename]#vinum#, but this is not recommended. The standard way to start [.filename]#vinum# is as a kernel module. man:kldload[8] is not needed because when man:gvinum[8] starts, it checks whether the module has been loaded, and if it is not, it loads it automatically. === Startup [.filename]#vinum# stores configuration information on the disk slices in essentially the same form as in the configuration files. When reading from the configuration database, [.filename]#vinum# recognizes a number of keywords which are not allowed in the configuration files. For example, a disk configuration might contain the following text: [.programlisting] .... volume myvol state up volume bigraid state down plex name myvol.p0 state up org concat vol myvol plex name myvol.p1 state up org concat vol myvol plex name myvol.p2 state init org striped 512b vol myvol plex name bigraid.p0 state initializing org raid5 512b vol bigraid sd name myvol.p0.s0 drive a plex myvol.p0 state up len 1048576b driveoffset 265b plexoffset 0b sd name myvol.p0.s1 drive b plex myvol.p0 state up len 1048576b driveoffset 265b plexoffset 1048576b sd name myvol.p1.s0 drive c plex myvol.p1 state up len 1048576b driveoffset 265b plexoffset 0b sd name myvol.p1.s1 drive d plex myvol.p1 state up len 1048576b driveoffset 265b plexoffset 1048576b sd name myvol.p2.s0 drive a plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 0b sd name myvol.p2.s1 drive b plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 524288b sd name myvol.p2.s2 drive c plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 1048576b sd name myvol.p2.s3 drive d plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 1572864b sd name bigraid.p0.s0 drive a plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 0b sd name bigraid.p0.s1 drive b plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 4194304b sd name bigraid.p0.s2 drive c plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 8388608b sd name bigraid.p0.s3 drive d plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 12582912b sd name bigraid.p0.s4 drive e plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 16777216b .... The obvious differences here are the presence of explicit location information and naming, both of which are allowed but discouraged, and the information on the states. [.filename]#vinum# does not store information about drives in the configuration information. It finds the drives by scanning the configured disk drives for partitions with a [.filename]#vinum# label. This enables [.filename]#vinum# to identify drives correctly even if they have been assigned different UNIX(R) drive IDs. [[vinum-rc-startup]] ==== Automatic Startup _Gvinum_ always features an automatic startup once the kernel module is loaded, via man:loader.conf[5]. To load the _Gvinum_ module at boot time, add `geom_vinum_load="YES"` to [.filename]#/boot/loader.conf#. When [.filename]#vinum# is started with `gvinum start`, [.filename]#vinum# reads the configuration database from one of the [.filename]#vinum# drives. Under normal circumstances, each drive contains an identical copy of the configuration database, so it does not matter which drive is read. After a crash, however, [.filename]#vinum# must determine which drive was updated most recently and read the configuration from this drive. It then updates the configuration, if necessary, from progressively older drives. [[vinum-root]] == Using [.filename]#vinum# for the Root File System For a machine that has fully-mirrored file systems using [.filename]#vinum#, it is desirable to also mirror the root file system. Setting up such a configuration is less trivial than mirroring an arbitrary file system because: * The root file system must be available very early during the boot process, so the [.filename]#vinum# infrastructure must already be available at this time. * The volume containing the root file system also contains the system bootstrap and the kernel. These must be read using the host system's native utilities, such as the BIOS, which often cannot be taught about the details of [.filename]#vinum#. In the following sections, the term "root volume" is generally used to describe the [.filename]#vinum# volume that contains the root file system. === Starting up [.filename]#vinum# Early Enough for the Root File System [.filename]#vinum# must be available early in the system boot as man:loader[8] must be able to load the vinum kernel module before starting the kernel. This can be accomplished by putting this line in [.filename]#/boot/loader.conf#: [.programlisting] .... geom_vinum_load="YES" .... === Making a [.filename]#vinum#-based Root Volume Accessible to the Bootstrap The current FreeBSD bootstrap is only 7.5 KB of code and does not understand the internal [.filename]#vinum# structures. This means that it cannot parse the [.filename]#vinum# configuration data or figure out the elements of a boot volume. Thus, some workarounds are necessary to provide the bootstrap code with the illusion of a standard `a` partition that contains the root file system. For this to be possible, the following requirements must be met for the root volume: * The root volume must not be a stripe or `RAID`-5. * The root volume must not contain more than one concatenated subdisk per plex. Note that it is desirable and possible to use multiple plexes, each containing one replica of the root file system. The bootstrap process will only use one replica for finding the bootstrap and all boot files, until the kernel mounts the root file system. Each single subdisk within these plexes needs its own `a` partition illusion, for the respective device to be bootable. It is not strictly needed that each of these faked `a` partitions is located at the same offset within its device, compared with other devices containing plexes of the root volume. However, it is probably a good idea to create the [.filename]#vinum# volumes that way so the resulting mirrored devices are symmetric, to avoid confusion. In order to set up these `a` partitions for each device containing part of the root volume, the following is required: [.procedure] ==== . The location, offset from the beginning of the device, and size of this device's subdisk that is part of the root volume needs to be examined, using the command: + [source,shell] .... # gvinum l -rv root .... + [.filename]#vinum# offsets and sizes are measured in bytes. They must be divided by 512 in order to obtain the block numbers that are to be used by `bsdlabel`. . Run this command for each device that participates in the root volume: + [source,shell] .... # bsdlabel -e devname .... + _devname_ must be either the name of the disk, like [.filename]#da0# for disks without a slice table, or the name of the slice, like [.filename]#ad0s1#. + If there is already an `a` partition on the device from a pre-[.filename]#vinum# root file system, it should be renamed to something else so that it remains accessible (just in case), but will no longer be used by default to bootstrap the system. A currently mounted root file system cannot be renamed, so this must be executed either when being booted from a "Fixit" media, or in a two-step process where, in a mirror, the disk that is not been currently booted is manipulated first. + The offset of the [.filename]#vinum# partition on this device (if any) must be added to the offset of the respective root volume subdisk on this device. The resulting value will become the `offset` value for the new `a` partition. The `size` value for this partition can be taken verbatim from the calculation above. The `fstype` should be `4.2BSD`. The `fsize`, `bsize`, and `cpg` values should be chosen to match the actual file system, though they are fairly unimportant within this context. + That way, a new `a` partition will be established that overlaps the [.filename]#vinum# partition on this device. `bsdlabel` will only allow for this overlap if the [.filename]#vinum# partition has properly been marked using the `vinum` fstype. . A faked `a` partition now exists on each device that has one replica of the root volume. It is highly recommendable to verify the result using a command like: + [source,shell] .... # fsck -n /dev/devnamea .... ==== It should be remembered that all files containing control information must be relative to the root file system in the [.filename]#vinum# volume which, when setting up a new [.filename]#vinum# root volume, might not match the root file system that is currently active. So in particular, [.filename]#/etc/fstab# and [.filename]#/boot/loader.conf# need to be taken care of. At next reboot, the bootstrap should figure out the appropriate control information from the new [.filename]#vinum#-based root file system, and act accordingly. At the end of the kernel initialization process, after all devices have been announced, the prominent notice that shows the success of this setup is a message like: [source,shell] .... Mounting root from ufs:/dev/gvinum/root .... === Example of a [.filename]#vinum#-based Root Setup After the [.filename]#vinum# root volume has been set up, the output of `gvinum l -rv root` could look like: [source,shell] .... ... Subdisk root.p0.s0: Size: 125829120 bytes (120 MB) State: up Plex root.p0 at offset 0 (0 B) Drive disk0 (/dev/da0h) at offset 135680 (132 kB) Subdisk root.p1.s0: Size: 125829120 bytes (120 MB) State: up Plex root.p1 at offset 0 (0 B) Drive disk1 (/dev/da1h) at offset 135680 (132 kB) .... The values to note are `135680` for the offset, relative to partition [.filename]#/dev/da0h#. This translates to 265 512-byte disk blocks in `bsdlabel`'s terms. Likewise, the size of this root volume is 245760 512-byte blocks. [.filename]#/dev/da1h#, containing the second replica of this root volume, has a symmetric setup. The bsdlabel for these devices might look like: [source,shell] .... ... 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 245760 281 4.2BSD 2048 16384 0 # (Cyl. 0*- 15*) c: 71771688 0 unused 0 0 # (Cyl. 0 - 4467*) h: 71771672 16 vinum # (Cyl. 0*- 4467*) .... It can be observed that the `size` parameter for the faked `a` partition matches the value outlined above, while the `offset` parameter is the sum of the offset within the [.filename]#vinum# partition `h`, and the offset of this partition within the device or slice. This is a typical setup that is necessary to avoid the problem described in <>. The entire `a` partition is completely within the `h` partition containing all the [.filename]#vinum# data for this device. In the above example, the entire device is dedicated to [.filename]#vinum# and there is no leftover pre-[.filename]#vinum# root partition. === Troubleshooting The following list contains a few known pitfalls and solutions. ==== System Bootstrap Loads, but System Does Not Boot If for any reason the system does not continue to boot, the bootstrap can be interrupted by pressing kbd:[space] at the 10-seconds warning. The loader variable `vinum.autostart` can be examined by typing `show` and manipulated using `set` or `unset`. If the [.filename]#vinum# kernel module was not yet in the list of modules to load automatically, type `load geom_vinum`. When ready, the boot process can be continued by typing `boot -as` which `-as` requests the kernel to ask for the root file system to mount (`-a`) and make the boot process stop in single-user mode (`-s`), where the root file system is mounted read-only. That way, even if only one plex of a multi-plex volume has been mounted, no data inconsistency between plexes is being risked. At the prompt asking for a root file system to mount, any device that contains a valid root file system can be entered. If [.filename]#/etc/fstab# is set up correctly, the default should be something like `ufs:/dev/gvinum/root`. A typical alternate choice would be something like `ufs:da0d` which could be a hypothetical partition containing the pre-[.filename]#vinum# root file system. Care should be taken if one of the alias `a` partitions is entered here, that it actually references the subdisks of the [.filename]#vinum# root device, because in a mirrored setup, this would only mount one piece of a mirrored root device. If this file system is to be mounted read-write later on, it is necessary to remove the other plex(es) of the [.filename]#vinum# root volume since these plexes would otherwise carry inconsistent data. ==== Only Primary Bootstrap Loads If [.filename]#/boot/loader# fails to load, but the primary bootstrap still loads (visible by a single dash in the left column of the screen right after the boot process starts), an attempt can be made to interrupt the primary bootstrap by pressing kbd:[space]. This will make the bootstrap stop in link:{handbook}#boot-boot1[stage two]. An attempt can be made here to boot off an alternate partition, like the partition containing the previous root file system that has been moved away from `a`. [[vinum-root-panic]] ==== Nothing Boots, the Bootstrap Panics This situation will happen if the bootstrap had been destroyed by the [.filename]#vinum# installation. Unfortunately, [.filename]#vinum# accidentally leaves only 4 KB at the beginning of its partition free before starting to write its [.filename]#vinum# header information. However, the stage one and two bootstraps plus the bsdlabel require 8 KB. So if a [.filename]#vinum# partition was started at offset 0 within a slice or disk that was meant to be bootable, the [.filename]#vinum# setup will trash the bootstrap. Similarly, if the above situation has been recovered, by booting from a "Fixit" media, and the bootstrap has been re-installed using `bsdlabel -B` as described in link:{handbook}#boot-boot1[stage two], the bootstrap will trash the [.filename]#vinum# header, and [.filename]#vinum# will no longer find its disk(s). Though no actual [.filename]#vinum# configuration data or data in [.filename]#vinum# volumes will be trashed, and it would be possible to recover all the data by entering exactly the same [.filename]#vinum# configuration data again, the situation is hard to fix. It is necessary to move the entire [.filename]#vinum# partition by at least 4 KB, in order to have the [.filename]#vinum# header and the system bootstrap no longer collide.