diff --git a/documentation/content/en/books/arch-handbook/boot/_index.adoc b/documentation/content/en/books/arch-handbook/boot/_index.adoc index ebed0609ca..c280b5fe12 100644 --- a/documentation/content/en/books/arch-handbook/boot/_index.adoc +++ b/documentation/content/en/books/arch-handbook/boot/_index.adoc @@ -1,1337 +1,1337 @@ --- title: Chapter 1. Bootstrapping and Kernel Initialization prev: books/arch-handbook/parti next: books/arch-handbook/locking description: Bootstrapping and Kernel Initialization tags: ["boot", "BIOS", "kernel", "MBR", "FreeBSD"] showBookMenu: true weight: 2 path: "/books/arch-handbook/" --- [[boot]] = Bootstrapping and Kernel Initialization :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :sectnumoffset: 1 :partnums: :source-highlighter: rouge :experimental: :images-path: books/arch-handbook/ ifdef::env-beastie[] ifdef::backend-html5[] :imagesdir: ../../../../images/{images-path} endif::[] ifndef::book[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] toc::[] endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] toc::[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [[boot-synopsis]] == Synopsis -This chapter is an overview of the boot and system initialization processes, starting from the BIOS (firmware) POST, to the first user process creation. Since the initial steps of system startup are very architecture dependent, the IA-32 architecture is used as an example. +This chapter is an overview of the boot and system initialization processes, starting from the BIOS (firmware) POST, to the first user process creation. Since the initial steps of system startup are very architecture dependent, the IA-32 architecture is used as an example. But the AMD64 and ARM64 architectures are much more important and compelling examples and should be explained in the near future according to the topic of this document. The FreeBSD boot process can be surprisingly complex. After control is passed from the BIOS, a considerable amount of low-level configuration must be done before the kernel can be loaded and executed. This setup must be done in a simple and flexible manner, allowing the user a great deal of customization possibilities. [[boot-overview]] == Overview -The boot process is an extremely machine-dependent activity. Not only must code be written for every computer architecture, but there may also be multiple types of booting on the same architecture. For example, a directory listing of [.filename]#/usr/src/sys/boot# reveals a great amount of architecture-dependent code. There is a directory for each of the various supported architectures. In the x86-specific [.filename]#i386# directory, there are subdirectories for different boot standards like [.filename]#mbr# (Master Boot Record), [.filename]#gpt# (GUID Partition Table), and [.filename]#efi# (Extensible Firmware Interface). Each boot standard has its own conventions and data structures. The example that follows shows booting an x86 computer from an MBR hard drive with the FreeBSD [.filename]#boot0# multi-boot loader stored in the very first sector. That boot code starts the FreeBSD three-stage boot process. +The boot process is an extremely machine-dependent activity. Not only must code be written for every computer architecture, but there may also be multiple types of booting on the same architecture. For example, a directory listing of [.filename]#stand# reveals a great amount of architecture-dependent code. There is a directory for each of the various supported architectures. FreeBSD supports the CSM boot standard (Compatibility Support Module). So CSM is supported (with both GPT and MBR partitioning support) and UEFI booting (GPT is totally supported, MBR is mostly supported). It also supports loading files from ext2fs, MSDOS, UFS and ZFS. FreeBSD also supports the boot environment feature of ZFS which allows the HOST OS to communicate details about what to boot that go beyond a simple partition as was possible in the past. But UEFI is more relevant than the CMS these days. The example that follows shows booting an x86 computer from an MBR-partitioned hard drive with the FreeBSD [.filename]#boot0# multi-boot loader stored in the very first sector. That boot code starts the FreeBSD three-stage boot process. The key to understanding this process is that it is a series of stages of increasing complexity. These stages are [.filename]#boot1#, [.filename]#boot2#, and [.filename]#loader# (see man:boot[8] for more detail). The boot system executes each stage in sequence. The last stage, [.filename]#loader#, is responsible for loading the FreeBSD kernel. Each stage is examined in the following sections. Here is an example of the output generated by the different boot stages. Actual output may differ from machine to machine: [.informaltable] [cols="20%,80%", frame="none"] |=== |*FreeBSD Component* |*Output (may vary)* |`boot0` a| [source,bash] .... F1 FreeBSD F2 BSD F5 Disk 2 .... |`boot2` footnote:[This prompt will appear if the user presses a key just after selecting an OS to boot at the boot0 stage.] a| [source,bash] .... ->>FreeBSD/i386 BOOT -Default: 1:ad(1,a)/boot/loader +>>FreeBSD/x86 BOOT +Default: 0:ad(0p4)/boot/loader boot: .... |[.filename]#loader# a| [source,bash] .... BTX loader 1.00 BTX version is 1.02 Consoles: internal video/keyboard BIOS drive C: is disk0 BIOS 639kB/2096064kB available memory FreeBSD/x86 bootstrap loader, Revision 1.1 Console internal video/keyboard -(root@snap.freebsd.org, Thu Jan 16 22:18:05 UTC 2014) +(root@releng1.nyi.freebsd.org, Fri Apr 9 04:04:45 UTC 2021) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0xed9008 data=0x117d28+0x176650 syms=[0x8+0x137988+0x8+0x1515f8] .... |kernel a| [source,bash] .... -Copyright (c) 1992-2013 The FreeBSD Project. +Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. -FreeBSD 10.0-RELEASE 0 r260789: Thu Jan 16 22:34:59 UTC 2014 - root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 -FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 +FreeBSD 13.0-RELEASE 0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:04:45 UTC 2021 + root@releng1.nyi.freebsd.org:/usr/obj/usr/src/i386.i386/sys/GENERIC i386 +FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) .... |=== [[boot-bios]] == The BIOS When the computer powers on, the processor's registers are set to some predefined values. One of the registers is the _instruction pointer_ register, and its value after a power on is well defined: it is a 32-bit value of `0xfffffff0`. The instruction pointer register (also known as the Program Counter) points to code to be executed by the processor. Another important register is the `cr0` 32-bit control register, and its value just after a reboot is `0`. One of ``cr0``'s bits, the PE (Protection Enabled) bit, indicates whether the processor is running in 32-bit protected mode or 16-bit real mode. Since this bit is cleared at boot time, the processor boots in 16-bit real mode. Real mode means, among other things, that linear and physical addresses are identical. The reason for the processor not to start immediately in 32-bit protected mode is backwards compatibility. In particular, the boot process relies on the services provided by the BIOS, and the BIOS itself works in legacy, 16-bit code. The value of `0xfffffff0` is slightly less than 4 GB, so unless the machine has 4 GB of physical memory, it cannot point to a valid memory address. The computer's hardware translates this address so that it points to a BIOS memory block. The BIOS (Basic Input Output System) is a chip on the motherboard that has a relatively small amount of read-only memory (ROM). This memory contains various low-level routines that are specific to the hardware supplied with the motherboard. The processor will first jump to the address 0xfffffff0, which really resides in the BIOS's memory. Usually this address contains a jump instruction to the BIOS's POST routines. The POST (Power On Self Test) is a set of routines including the memory check, system bus check, and other low-level initialization so the CPU can set up the computer properly. The important step of this stage is determining the boot device. Modern BIOS implementations permit the selection of a boot device, allowing booting from a floppy, CD-ROM, hard disk, or other devices. The very last thing in the POST is the `INT 0x19` instruction. The `INT 0x19` handler reads 512 bytes from the first sector of boot device into the memory at address `0x7c00`. The term _first sector_ originates from hard drive architecture, where the magnetic plate is divided into a number of cylindrical tracks. Tracks are numbered, and every track is divided into a number (usually 64) of sectors. Track numbers start at 0, but sector numbers start from 1. Track 0 is the outermost on the magnetic plate, and sector 1, the first sector, has a special purpose. It is also called the MBR, or Master Boot Record. The remaining sectors on the first track are never used. This sector is our boot-sequence starting point. As we will see, this sector contains a copy of our [.filename]#boot0# program. A jump is made by the BIOS to address `0x7c00` so it starts executing. [[boot-boot0]] == The Master Boot Record (`boot0`) After control is received from the BIOS at memory address `0x7c00`, [.filename]#boot0# starts executing. It is the first piece of code under FreeBSD control. The task of [.filename]#boot0# is quite simple: scan the partition table and let the user choose which partition to boot from. The Partition Table is a special, standard data structure embedded in the MBR (hence embedded in [.filename]#boot0#) describing the four standard PC "partitions". [.filename]#boot0# resides in the filesystem as [.filename]#/boot/boot0#. It is a small 512-byte file, and it is exactly what FreeBSD's installation procedure wrote to the hard disk's MBR if you chose the "bootmanager" option at installation time. Indeed, [.filename]#boot0#_is_ the MBR. -As mentioned previously, the `INT 0x19` instruction causes the `INT 0x19` handler to load an MBR ([.filename]#boot0#) into memory at address `0x7c00`. The source file for [.filename]#boot0# can be found in [.filename]#sys/boot/i386/boot0/boot0.S# - which is an awesome piece of code written by Robert Nordier. +As mentioned previously, we're calling the BIOS `INT 0x19` to load the MBR ([.filename]#boot0#) into memory at address `0x7c00`. The source file for [.filename]#boot0# can be found in [.filename]#stand/i386/boot0/boot0.S# - which is an awesome piece of code written by Robert Nordier. A special structure starting from offset `0x1be` in the MBR is called the _partition table_. It has four records of 16 bytes each, called _partition records_, which represent how the hard disk is partitioned, or, in FreeBSD's terminology, sliced. One byte of those 16 says whether a partition (slice) is bootable or not. Exactly one record must have that flag set, otherwise [.filename]#boot0#'s code will refuse to proceed. A partition record has the following fields: * the 1-byte filesystem type * the 1-byte bootable flag * the 6 byte descriptor in CHS format * the 8 byte descriptor in LBA format A partition record descriptor contains information about where exactly the partition resides on the drive. Both descriptors, LBA and CHS, describe the same information, but in different ways: LBA (Logical Block Addressing) has the starting sector for the partition and the partition's length, while CHS (Cylinder Head Sector) has coordinates for the first and last sectors of the partition. The partition table ends with the special signature `0xaa55`. The MBR must fit into 512 bytes, a single disk sector. This program uses low-level "tricks" like taking advantage of the side effects of certain instructions and reusing register values from previous operations to make the most out of the fewest possible instructions. Care must also be taken when handling the partition table, which is embedded in the MBR itself. For these reasons, be very careful when modifying [.filename]#boot0.S#. Note that the [.filename]#boot0.S# source file is assembled "as is": instructions are translated one by one to binary, with no additional information (no ELF file format, for example). This kind of low-level control is achieved at link time through special control flags passed to the linker. For example, the text section of the program is set to be located at address `0x600`. In practice this means that [.filename]#boot0# must be loaded to memory address `0x600` in order to function properly. -It is worth looking at the [.filename]#Makefile# for [.filename]#boot0# ([.filename]#sys/boot/i386/boot0/Makefile#), as it defines some of the run-time behavior of [.filename]#boot0#. For instance, if a terminal connected to the serial port (COM1) is used for I/O, the macro `SIO` must be defined (`-DSIO`). `-DPXE` enables boot through PXE by pressing kbd:[F6]. Additionally, the program defines a set of _flags_ that allow further modification of its behavior. All of this is illustrated in the [.filename]#Makefile#. For example, look at the linker directives which command the linker to start the text section at address `0x600`, and to build the output file "as is" (strip out any file formatting): +It is worth looking at the [.filename]#Makefile# for [.filename]#boot0# ([.filename]#stand/i386/boot0/Makefile#), as it defines some of the run-time behavior of [.filename]#boot0#. For instance, if a terminal connected to the serial port (COM1) is used for I/O, the macro `SIO` must be defined (`-DSIO`). `-DPXE` enables boot through PXE by pressing kbd:[F6]. Additionally, the program defines a set of _flags_ that allow further modification of its behavior. All of this is illustrated in the [.filename]#Makefile#. For example, look at the linker directives which command the linker to start the text section at address `0x600`, and to build the output file "as is" (strip out any file formatting): [.programlisting] .... BOOT_BOOT0_ORG?=0x600 - LDFLAGS=-e start -Ttext ${BOOT_BOOT0_ORG} \ - -Wl,-N,-S,--oformat,binary + ORG=${BOOT_BOOT0_ORG} .... -.[.filename]#sys/boot/i386/boot0/Makefile# [[boot-boot0-makefile-as-is]] +.[.filename]#stand/i386/boot0/Makefile# [[boot-boot0-makefile-as-is]] Let us now start our study of the MBR, or [.filename]#boot0#, starting where execution begins. [NOTE] ==== Some modifications have been made to some instructions in favor of better exposition. For example, some macros are expanded, and some macro tests are omitted when the result of the test is known. This applies to all of the code examples shown. ==== [.programlisting] .... start: cld # String ops inc xorw %ax,%ax # Zero movw %ax,%es # Address movw %ax,%ds # data movw %ax,%ss # Set up - movw 0x7c00,%sp # stack + movw $LOAD,%sp # stack .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-entrypoint]] -This first block of code is the entry point of the program. It is where the BIOS transfers control. First, it makes sure that the string operations autoincrement its pointer operands (the `cld` instruction) footnote:[When in doubt, we refer the reader to the official Intel manuals, which describe the exact semantics for each instruction: .]. Then, as it makes no assumption about the state of the segment registers, it initializes them. Finally, it sets the stack pointer register (`%sp`) to address `0x7c00`, so we have a working stack. +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-entrypoint]] +This first block of code is the entry point of the program. It is where the BIOS transfers control. First, it makes sure that the string operations autoincrement its pointer operands (the `cld` instruction) footnote:[When in doubt, we refer the reader to the official Intel manuals, which describe the exact semantics for each instruction: .]. Then, as it makes no assumption about the state of the segment registers, it initializes them. Finally, it sets the stack pointer register (`%sp`) to ($LOAD = address `0x7c00`), so we have a working stack. The next block is responsible for the relocation and subsequent jump to the relocated code. [.programlisting] .... - movw $0x7c00,%si # Source - movw $0x600,%di # Destination - movw $512,%cx # Word count + movw %sp,%si # Source + movw $start,%di # Destination + movw $0x100,%cx # Word count rep # Relocate - movsb # code + movsw # code movw %di,%bp # Address variables - movb $16,%cl # Words to clear + movb $0x8,%cl # Words to clear rep # Zero - stosb # them + stosw # them incb -0xe(%di) # Set the S field to 1 - jmp main-0x7c00+0x600 # Jump to relocated code + jmp main-LOAD+ORIGIN # Jump to relocated code .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-relocation]] -As [.filename]#boot0# is loaded by the BIOS to address `0x7C00`, it copies itself to address `0x600` and then transfers control there (recall that it was linked to execute at address `0x600`). The source address, `0x7c00`, is copied to register `%si`. The destination address, `0x600`, to register `%di`. The number of bytes to copy, `512` (the program's size), is copied to register `%cx`. Next, the `rep` instruction repeats the instruction that follows, that is, `movsb`, the number of times dictated by the `%cx` register. The `movsb` instruction copies the byte pointed to by `%si` to the address pointed to by `%di`. This is repeated another 511 times. On each repetition, both the source and destination registers, `%si` and `%di`, are incremented by one. Thus, upon completion of the 512-byte copy, `%di` has the value `0x600`+`512`= `0x800`, and `%si` has the value `0x7c00`+`512`= `0x7e00`; we have thus completed the code _relocation_. +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-relocation]] +As [.filename]#boot0# is loaded by the BIOS to address `0x7C00`, it copies itself to address `0x600` and then transfers control there (recall that it was linked to execute at address `0x600`). The source address, `0x7c00`, is copied to register `%si`. The destination address, `0x600`, to register `%di`. The number of words to copy, `256` (the program's size = 512 bytes), is copied to register `%cx`. Next, the `rep` instruction repeats the instruction that follows, that is, `movsw`, the number of times dictated by the `%cx` register. The `movsw` instruction copies the word pointed to by `%si` to the address pointed to by `%di`. This is repeated another 255 times. On each repetition, both the source and destination registers, `%si` and `%di`, are incremented by one. Thus, upon completion of the 256-word (512-byte) copy, `%di` has the value `0x600`+`512`= `0x800`, and `%si` has the value `0x7c00`+`512`= `0x7e00`; we have thus completed the code _relocation_. Since the last update of this document, the copy instructions have changed in the code, so instead of the movsb and stosb, movsw and stosw have been introduced, which copy 2 bytes(1 word) in one iteration. -Next, the destination register `%di` is copied to `%bp`. `%bp` gets the value `0x800`. The value `16` is copied to `%cl` in preparation for a new string operation (like our previous `movsb`). Now, `stosb` is executed 16 times. This instruction copies a `0` value to the address pointed to by the destination register (`%di`, which is `0x800`), and increments it. This is repeated another 15 times, so `%di` ends up with value `0x810`. Effectively, this clears the address range `0x800`-`0x80f`. This range is used as a (fake) partition table for writing the MBR back to disk. Finally, the sector field for the CHS addressing of this fake partition is given the value 1 and a jump is made to the main function from the relocated code. Note that until this jump to the relocated code, any reference to an absolute address was avoided. +Next, the destination register `%di` is copied to `%bp`. `%bp` gets the value `0x800`. The value `8` is copied to `%cl` in preparation for a new string operation (like our previous `movsw`). Now, `stosw` is executed 8 times. This instruction copies a `0` value to the address pointed to by the destination register (`%di`, which is `0x800`), and increments it. This is repeated another 7 times, so `%di` ends up with value `0x810`. Effectively, this clears the address range `0x800`-`0x80f`. This range is used as a (fake) partition table for writing the MBR back to disk. Finally, the sector field for the CHS addressing of this fake partition is given the value 1 and a jump is made to the main function from the relocated code. Note that until this jump to the relocated code, any reference to an absolute address was avoided. The following code block tests whether the drive number provided by the BIOS should be used, or the one stored in [.filename]#boot0#. [.programlisting] .... main: - testb $SETDRV,-69(%bp) # Set drive number? + testb $SETDRV,_FLAGS(%bp) # Set drive number? +#ifndef CHECK_DRIVE /* disable drive checks */ + jz save_curdrive # no, use the default +#else jnz disable_update # Yes testb %dl,%dl # Drive number valid? js save_curdrive # Possibly (0x80 set) +#endif .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-drivenumber]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-drivenumber]] This code tests the `SETDRV` bit (`0x20`) in the _flags_ variable. Recall that register `%bp` points to address location `0x800`, so the test is done to the _flags_ variable at address `0x800`-`69`= `0x7bb`. This is an example of the type of modifications that can be done to [.filename]#boot0#. The `SETDRV` flag is not set by default, but it can be set in the [.filename]#Makefile#. When set, the drive number stored in the MBR is used instead of the one provided by the BIOS. We assume the defaults, and that the BIOS provided a valid drive number, so we jump to `save_curdrive`. The next block saves the drive number provided by the BIOS, and calls `putn` to print a new line on the screen. [.programlisting] .... save_curdrive: movb %dl, (%bp) # Save drive number pushw %dx # Also in the stack #ifdef TEST /* test code, print internal bios drive */ rolb $1, %dl movw $drive, %si call putkey #endif callw putn # Print a newline .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-savedrivenumber]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-savedrivenumber]] Note that we assume `TEST` is not defined, so the conditional code in it is not assembled and will not appear in our executable [.filename]#boot0#. Our next block implements the actual scanning of the partition table. It prints to the screen the partition type for each of the four entries in the partition table. It compares each type with a list of well-known operating system file systems. Examples of recognized partition types are NTFS (Windows(R), ID 0x7), `ext2fs` (Linux(R), ID 0x83), and, of course, `ffs`/`ufs2` (FreeBSD, ID 0xa5). The implementation is fairly simple. [.programlisting] .... movw $(partbl+0x4),%bx # Partition table (+4) xorw %dx,%dx # Item number read_entry: movb %ch,-0x4(%bx) # Zero active flag (ch == 0) btw %dx,_FLAGS(%bp) # Entry enabled? jnc next_entry # No movb (%bx),%al # Load type test %al, %al # skip empty partition jz next_entry movw $bootable_ids,%di # Lookup tables movb $(TLEN+1),%cl # Number of entries repne # Locate scasb # type addw $(TLEN-1), %di # Adjust movb (%di),%cl # Partition addw %cx,%di # description callw putx # Display it next_entry: incw %dx # Next item addb $0x10,%bl # Next entry jnc read_entry # Till done .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-partition-scan]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-partition-scan]] It is important to note that the active flag for each entry is cleared, so after the scanning, _no_ partition entry is active in our memory copy of [.filename]#boot0#. Later, the active flag will be set for the selected partition. This ensures that only one active partition exists if the user chooses to write the changes back to disk. The next block tests for other drives. At startup, the BIOS writes the number of drives present in the computer to address `0x475`. If there are any other drives present, [.filename]#boot0# prints the current drive to screen. The user may command [.filename]#boot0# to scan partitions on another drive later. [.programlisting] .... popw %ax # Drive number - subb $0x79,%al # Does next - cmpb 0x475,%al # drive exist? (from BIOS?) + subb $0x80-0x1,%al # Does next + cmpb NHRDRV,%al # drive exist? (from BIOS?) jb print_drive # Yes decw %ax # Already drive 0? jz print_prompt # Yes .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-test-drives]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-test-drives]] We make the assumption that a single drive is present, so the jump to `print_drive` is not performed. We also assume nothing strange happened, so we jump to `print_prompt`. This next block just prints out a prompt followed by the default option: [.programlisting] .... print_prompt: movw $prompt,%si # Display callw putstr # prompt movb _OPT(%bp),%dl # Display decw %si # default callw putkey # key jmp start_input # Skip beep .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-prompt]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-prompt]] Finally, a jump is performed to `start_input`, where the BIOS services are used to start a timer and for reading user input from the keyboard; if the timer expires, the default option will be selected: [.programlisting] .... start_input: xorb %ah,%ah # BIOS: Get int $0x1a # system time movw %dx,%di # Ticks when addw _TICKS(%bp),%di # timeout read_key: movb $0x1,%ah # BIOS: Check int $0x16 # for keypress jnz got_key # Have input xorb %ah,%ah # BIOS: int 0x1a, 00 int $0x1a # get system time cmpw %di,%dx # Timeout? jb read_key # No .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-start-input]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-start-input]] An interrupt is requested with number `0x1a` and argument `0` in register `%ah`. The BIOS has a predefined set of services, requested by applications as software-generated interrupts through the `int` instruction and receiving arguments in registers (in this case, `%ah`). Here, particularly, we are requesting the number of clock ticks since last midnight; this value is computed by the BIOS through the RTC (Real Time Clock). This clock can be programmed to work at frequencies ranging from 2 Hz to 8192 Hz. The BIOS sets it to 18.2 Hz at startup. When the request is satisfied, a 32-bit result is returned by the BIOS in registers `%cx` and `%dx` (lower bytes in `%dx`). This result (the `%dx` part) is copied to register `%di`, and the value of the `TICKS` variable is added to `%di`. This variable resides in [.filename]#boot0# at offset `_TICKS` (a negative value) from register `%bp` (which, recall, points to `0x800`). The default value of this variable is `0xb6` (182 in decimal). Now, the idea is that [.filename]#boot0# constantly requests the time from the BIOS, and when the value returned in register `%dx` is greater than the value stored in `%di`, the time is up and the default selection will be made. Since the RTC ticks 18.2 times per second, this condition will be met after 10 seconds (this default behavior can be changed in the [.filename]#Makefile#). Until this time has passed, [.filename]#boot0# continually asks the BIOS for any user input; this is done through `int 0x16`, argument `1` in `%ah`. Whether a key was pressed or the time expired, subsequent code validates the selection. Based on the selection, the register `%si` is set to point to the appropriate partition entry in the partition table. This new selection overrides the previous default one. Indeed, it becomes the new default. Finally, the ACTIVE flag of the selected partition is set. If it was enabled at compile time, the in-memory version of [.filename]#boot0# with these modified values is written back to the MBR on disk. We leave the details of this implementation to the reader. We now end our study with the last code block from the [.filename]#boot0# program: [.programlisting] .... - movw $0x7c00,%bx # Address for read + movw $LOAD,%bx # Address for read movb $0x2,%ah # Read sector callw intx13 # from disk jc beep # If error - cmpw $0xaa55,0x1fe(%bx) # Bootable? + cmpw $MAGIC,0x1fe(%bx) # Bootable? jne beep # No pushw %si # Save ptr to selected part. callw putn # Leave some space popw %si # Restore, next stage uses it jmp *%bx # Invoke bootstrap .... -.[.filename]#sys/boot/i386/boot0/boot0.S# [[boot-boot0-check-bootable]] +.[.filename]#stand/i386/boot0/boot0.S# [[boot-boot0-check-bootable]] Recall that `%si` points to the selected partition entry. This entry tells us where the partition begins on disk. We assume, of course, that the partition selected is actually a FreeBSD slice. [NOTE] ==== From now on, we will favor the use of the technically more accurate term "slice" rather than "partition". ==== The transfer buffer is set to `0x7c00` (register `%bx`), and a read for the first sector of the FreeBSD slice is requested by calling `intx13`. We assume that everything went okay, so a jump to `beep` is not performed. In particular, the new sector read must end with the magic sequence `0xaa55`. Finally, the value at `%si` (the pointer to the selected partition table) is preserved for use by the next stage, and a jump is performed to address `0x7c00`, where execution of our next stage (the just-read block) is started. [[boot-boot1]] == `boot1` Stage So far we have gone through the following sequence: * The BIOS did some early hardware initialization, including the POST. The MBR ([.filename]#boot0#) was loaded from absolute disk sector one to address `0x7c00`. Execution control was passed to that location. * [.filename]#boot0# relocated itself to the location it was linked to execute (`0x600`), followed by a jump to continue execution at the appropriate place. Finally, [.filename]#boot0# loaded the first disk sector from the FreeBSD slice to address `0x7c00`. Execution control was passed to that location. [.filename]#boot1# is the next step in the boot-loading sequence. It is the first of three boot stages. Note that we have been dealing exclusively with disk sectors. Indeed, the BIOS loads the absolute first sector, while [.filename]#boot0# loads the first sector of the FreeBSD slice. Both loads are to address `0x7c00`. We can conceptually think of these disk sectors as containing the files [.filename]#boot0# and [.filename]#boot1#, respectively, but in reality this is not entirely true for [.filename]#boot1#. Strictly speaking, unlike [.filename]#boot0#, [.filename]#boot1# is not part of the boot blocks footnote:[There is a file /boot/boot1, but it is not the written to the beginning of the FreeBSD slice. Instead, it is concatenated with boot2 to form boot, which is written to the beginning of the FreeBSD slice and read at boot time.]. Instead, a single, full-blown file, [.filename]#boot# ([.filename]#/boot/boot#), is what ultimately is written to disk. This file is a combination of [.filename]#boot1#, [.filename]#boot2# and the `Boot Extender` (or BTX). This single file is greater in size than a single sector (greater than 512 bytes). Fortunately, [.filename]#boot1# occupies _exactly_ the first 512 bytes of this single file, so when [.filename]#boot0# loads the first sector of the FreeBSD slice (512 bytes), it is actually loading [.filename]#boot1# and transferring control to it. The main task of [.filename]#boot1# is to load the next boot stage. This next stage is somewhat more complex. It is composed of a server called the "Boot Extender", or BTX, and a client, called [.filename]#boot2#. As we will see, the last boot stage, [.filename]#loader#, is also a client of the BTX server. Let us now look in detail at what exactly is done by [.filename]#boot1#, starting like we did for [.filename]#boot0#, at its entry point: [.programlisting] .... start: jmp main .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-entry]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-entry]] The entry point at `start` simply jumps past a special data area to the label `main`, which in turn looks like this: [.programlisting] .... main: cld # String ops inc xor %cx,%cx # Zero mov %cx,%es # Address mov %cx,%ds # data mov %cx,%ss # Set up mov $start,%sp # stack mov %sp,%si # Source - mov $0x700,%di # Destination + mov $MEM_REL,%di # Destination incb %ch # Word count rep # Copy movsw # code .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-main]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-main]] Just like [.filename]#boot0#, this code relocates [.filename]#boot1#, this time to memory address `0x700`. However, unlike [.filename]#boot0#, it does not jump there. [.filename]#boot1# is linked to execute at address `0x7c00`, effectively where it was loaded in the first place. The reason for this relocation will be discussed shortly. Next comes a loop that looks for the FreeBSD slice. Although [.filename]#boot0# loaded [.filename]#boot1# from the FreeBSD slice, no information was passed to it about this footnote:[Actually we did pass a pointer to the slice entry in register %si. However, boot1 does not assume that it was loaded by boot0 (perhaps some other MBR loaded it, and did not pass this information), so it assumes nothing.], so [.filename]#boot1# must rescan the partition table to find where the FreeBSD slice starts. Therefore it rereads the MBR: [.programlisting] .... mov $part4,%si # Partition cmpb $0x80,%dl # Hard drive? jb main.4 # No movb $0x1,%dh # Block count callw nread # Read MBR .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-find-freebsd]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-find-freebsd]] In the code above, register `%dl` maintains information about the boot device. This is passed on by the BIOS and preserved by the MBR. Numbers `0x80` and greater tells us that we are dealing with a hard drive, so a call is made to `nread`, where the MBR is read. Arguments to `nread` are passed through `%si` and `%dh`. The memory address at label `part4` is copied to `%si`. This memory address holds a "fake partition" to be used by `nread`. The following is the data in the fake partition: [.programlisting] .... part4: .byte 0x80, 0x00, 0x01, 0x00 .byte 0xa5, 0xfe, 0xff, 0xff .byte 0x00, 0x00, 0x00, 0x00 .byte 0x50, 0xc3, 0x00, 0x00 .... -.[.filename]#sys/boot/i386/boot2/Makefile# [[boot-boot2-make-fake-partition]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot2-make-fake-partition]] In particular, the LBA for this fake partition is hardcoded to zero. This is used as an argument to the BIOS for reading absolute sector one from the hard drive. Alternatively, CHS addressing could be used. In this case, the fake partition holds cylinder 0, head 0 and sector 1, which is equivalent to absolute sector one. Let us now proceed to take a look at `nread`: [.programlisting] .... nread: - mov $0x8c00,%bx # Transfer buffer + mov $MEM_BUF,%bx # Transfer buffer mov 0x8(%si),%ax # Get mov 0xa(%si),%cx # LBA push %cs # Read from callw xread.1 # disk jnc return # If success, return .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-nread]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-nread]] Recall that `%si` points to the fake partition. The word footnote:[In the context of 16-bit real mode, a word is 2 bytes.] at offset `0x8` is copied to register `%ax` and word at offset `0xa` to `%cx`. They are interpreted by the BIOS as the lower 4-byte value denoting the LBA to be read (the upper four bytes are assumed to be zero). Register `%bx` holds the memory address where the MBR will be loaded. The instruction pushing `%cs` onto the stack is very interesting. In this context, it accomplishes nothing. However, as we will see shortly, [.filename]#boot2#, in conjunction with the BTX server, also uses `xread.1`. This mechanism will be discussed in the next section. The code at `xread.1` further calls the `read` function, which actually calls the BIOS asking for the disk sector: [.programlisting] .... xread.1: pushl $0x0 # absolute push %cx # block push %ax # number push %es # Address of push %bx # transfer buffer xor %ax,%ax # Number of movb %dh,%al # blocks to push %ax # transfer push $0x10 # Size of packet mov %sp,%bp # Packet pointer callw read # Read from disk lea 0x10(%bp),%sp # Clear stack lret # To far caller .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-xread1]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-xread1]] Note the long return instruction at the end of this block. This instruction pops out the `%cs` register pushed by `nread`, and returns. Finally, `nread` also returns. With the MBR loaded to memory, the actual loop for searching the FreeBSD slice begins: [.programlisting] .... mov $0x1,%cx # Two passes main.1: - mov $0x8dbe,%si # Partition table + mov $MEM_BUF+PRT_OFF,%si # Partition table movb $0x1,%dh # Partition main.2: - cmpb $0xa5,0x4(%si) # Our partition type? + cmpb $PRT_BSD,0x4(%si) # Our partition type? jne main.3 # No jcxz main.5 # If second pass testb $0x80,(%si) # Active? jnz main.5 # Yes main.3: add $0x10,%si # Next entry incb %dh # Partition - cmpb $0x5,%dh # In table? + cmpb $0x1+PRT_NUM,%dh # In table? jb main.2 # Yes dec %cx # Do two jcxz main.1 # passes .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-find-part]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-find-part]] If a FreeBSD slice is identified, execution continues at `main.5`. Note that when a FreeBSD slice is found `%si` points to the appropriate entry in the partition table, and `%dh` holds the partition number. We assume that a FreeBSD slice is found, so we continue execution at `main.5`: [.programlisting] .... main.5: - mov %dx,0x900 # Save args - movb $0x10,%dh # Sector count + mov %dx,MEM_ARG # Save args + movb $NSECT,%dh # Sector count callw nread # Read disk - mov $0x9000,%bx # BTX + mov $MEM_BTX,%bx # BTX mov 0xa(%bx),%si # Get BTX length and set add %bx,%si # %si to start of boot2.bin - mov $0xc000,%di # Client page 2 - mov $0xa200,%cx # Byte + mov $MEM_USR+SIZ_PAG*2,%di # Client page 2 + mov $MEM_BTX+(NSECT-1)*SIZ_SEC,%cx # Byte sub %si,%cx # count rep # Relocate movsb # client .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-main5]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-main5]] Recall that at this point, register `%si` points to the FreeBSD slice entry in the MBR partition table, so a call to `nread` will effectively read sectors at the beginning of this partition. The argument passed on register `%dh` tells `nread` to read 16 disk sectors. Recall that the first 512 bytes, or the first sector of the FreeBSD slice, coincides with the [.filename]#boot1# program. Also recall that the file written to the beginning of the FreeBSD slice is not [.filename]#/boot/boot1#, but [.filename]#/boot/boot#. Let us look at the size of these files in the filesystem: [source,bash] .... -r--r--r-- 1 root wheel 512B Jan 8 00:15 /boot/boot0 -r--r--r-- 1 root wheel 512B Jan 8 00:15 /boot/boot1 -r--r--r-- 1 root wheel 7.5K Jan 8 00:15 /boot/boot2 -r--r--r-- 1 root wheel 8.0K Jan 8 00:15 /boot/boot .... Both [.filename]#boot0# and [.filename]#boot1# are 512 bytes each, so they fit _exactly_ in one disk sector. [.filename]#boot2# is much bigger, holding both the BTX server and the [.filename]#boot2# client. Finally, a file called simply [.filename]#boot# is 512 bytes larger than [.filename]#boot2#. This file is a concatenation of [.filename]#boot1# and [.filename]#boot2#. As already noted, [.filename]#boot0# is the file written to the absolute first disk sector (the MBR), and [.filename]#boot# is the file written to the first sector of the FreeBSD slice; [.filename]#boot1# and [.filename]#boot2# are _not_ written to disk. The command used to concatenate [.filename]#boot1# and [.filename]#boot2# into a single [.filename]#boot# is merely `cat boot1 boot2 > boot`. So [.filename]#boot1# occupies exactly the first 512 bytes of [.filename]#boot# and, because [.filename]#boot# is written to the first sector of the FreeBSD slice, [.filename]#boot1# fits exactly in this first sector. When `nread` reads the first 16 sectors of the FreeBSD slice, it effectively reads the entire [.filename]#boot# file footnote:[512*16=8192 bytes, exactly the size of boot]. We will see more details about how [.filename]#boot# is formed from [.filename]#boot1# and [.filename]#boot2# in the next section. Recall that `nread` uses memory address `0x8c00` as the transfer buffer to hold the sectors read. This address is conveniently chosen. Indeed, because [.filename]#boot1# belongs to the first 512 bytes, it ends up in the address range `0x8c00`-`0x8dff`. The 512 bytes that follows (range `0x8e00`-`0x8fff`) is used to store the _bsdlabel_ footnote:[Historically known as disklabel. If you ever wondered where FreeBSD stored this information, it is in this region. See man:bsdlabel[8]]. Starting at address `0x9000` is the beginning of the BTX server, and immediately following is the [.filename]#boot2# client. The BTX server acts as a kernel, and executes in protected mode in the most privileged level. In contrast, the BTX clients ([.filename]#boot2#, for example), execute in user mode. We will see how this is accomplished in the next section. The code after the call to `nread` locates the beginning of [.filename]#boot2# in the memory buffer, and copies it to memory address `0xc000`. This is because the BTX server arranges [.filename]#boot2# to execute in a segment starting at `0xa000`. We explore this in detail in the following section. The last code block of [.filename]#boot1# enables access to memory above 1MB footnote:[This is necessary for legacy reasons. Interested readers should see .] and concludes with a jump to the starting point of the BTX server: [.programlisting] .... seta20: cli # Disable interrupts seta20.1: dec %cx # Timeout? jz seta20.3 # Yes inb $0x64,%al # Get status testb $0x2,%al # Busy? jnz seta20.1 # Yes movb $0xd1,%al # Command: Write outb %al,$0x64 # output port seta20.2: inb $0x64,%al # Get status testb $0x2,%al # Busy? jnz seta20.2 # Yes movb $0xdf,%al # Enable outb %al,$0x60 # A20 seta20.3: sti # Enable interrupts jmp 0x9010 # Start BTX .... -.[.filename]#sys/boot/i386/boot2/boot1.S# [[boot-boot1-seta20]] +.[.filename]#stand/i386/boot2/boot1.S# [[boot-boot1-seta20]] Note that right before the jump, interrupts are enabled. [[btx-server]] == The BTX Server Next in our boot sequence is the BTX Server. Let us quickly remember how we got here: * The BIOS loads the absolute sector one (the MBR, or [.filename]#boot0#), to address `0x7c00` and jumps there. * [.filename]#boot0# relocates itself to `0x600`, the address it was linked to execute, and jumps over there. It then reads the first sector of the FreeBSD slice (which consists of [.filename]#boot1#) into address `0x7c00` and jumps over there. * [.filename]#boot1# loads the first 16 sectors of the FreeBSD slice into address `0x8c00`. This 16 sectors, or 8192 bytes, is the whole file [.filename]#boot#. The file is a concatenation of [.filename]#boot1# and [.filename]#boot2#. [.filename]#boot2#, in turn, contains the BTX server and the [.filename]#boot2# client. Finally, a jump is made to address `0x9010`, the entry point of the BTX server. -Before studying the BTX Server in detail, let us further review how the single, all-in-one [.filename]#boot# file is created. The way [.filename]#boot# is built is defined in its [.filename]#Makefile# ([.filename]#/usr/src/sys/boot/i386/boot2/Makefile#). Let us look at the rule that creates the [.filename]#boot# file: +Before studying the BTX Server in detail, let us further review how the single, all-in-one [.filename]#boot# file is created. The way [.filename]#boot# is built is defined in its [.filename]#Makefile# ([.filename]#stand/i386/boot2/Makefile#). Let us look at the rule that creates the [.filename]#boot# file: [.programlisting] .... boot: boot1 boot2 cat boot1 boot2 > boot .... -.[.filename]#sys/boot/i386/boot2/Makefile# [[boot-boot1-make-boot]] +.[.filename]#stand/i386/boot2/Makefile# [[boot-boot1-make-boot]] This tells us that [.filename]#boot1# and [.filename]#boot2# are needed, and the rule simply concatenates them to produce a single file called [.filename]#boot#. The rules for creating [.filename]#boot1# are also quite simple: [.programlisting] .... boot1: boot1.out - objcopy -S -O binary boot1.out boot1 + ${OBJCOPY} -S -O binary boot1.out ${.TARGET} boot1.out: boot1.o - ld -e start -Ttext 0x7c00 -o boot1.out boot1.o + ${LD} ${LD_FLAGS} -e start --defsym ORG=${ORG1} -T ${LDSCRIPT} -o ${.TARGET} boot1.o .... -.[.filename]#sys/boot/i386/boot2/Makefile# [[boot-boot1-make-boot1]] +.[.filename]#stand/i386/boot2/Makefile# [[boot-boot1-make-boot1]] To apply the rule for creating [.filename]#boot1#, [.filename]#boot1.out# must be resolved. This, in turn, depends on the existence of [.filename]#boot1.o#. This last file is simply the result of assembling our familiar [.filename]#boot1.S#, without linking. Now, the rule for creating [.filename]#boot1.out# is applied. This tells us that [.filename]#boot1.o# should be linked with `start` as its entry point, and starting at address `0x7c00`. Finally, [.filename]#boot1# is created from [.filename]#boot1.out# applying the appropriate rule. This rule is the [.filename]#objcopy# command applied to [.filename]#boot1.out#. Note the flags passed to [.filename]#objcopy#: `-S` tells it to strip all relocation and symbolic information; `-O binary` indicates the output format, that is, a simple, unformatted binary file. Having [.filename]#boot1#, let us take a look at how [.filename]#boot2# is constructed: [.programlisting] .... boot2: boot2.ld - @set -- `ls -l boot2.ld`; x=$$((7680-$$5)); \ + @set -- `ls -l ${.ALLSRC}`; x=$$((${BOOT2SIZE}-$$5)); \ echo "$$x bytes available"; test $$x -ge 0 - dd if=boot2.ld of=boot2 obs=7680 conv=osync + ${DD} if=${.ALLSRC} of=${.TARGET} bs=${BOOT2SIZE} conv=sync - boot2.ld: boot2.ldr boot2.bin ../btx/btx/btx - btxld -v -E 0x2000 -f bin -b ../btx/btx/btx -l boot2.ldr \ - -o boot2.ld -P 1 boot2.bin + boot2.ld: boot2.ldr boot2.bin ${BTXKERN} + btxld -v -E ${ORG2} -f bin -b ${BTXKERN} -l boot2.ldr \ + -o ${.TARGET} -P 1 boot2.bin boot2.ldr: - dd if=/dev/zero of=boot2.ldr bs=512 count=1 + ${DD} if=/dev/zero of=${.TARGET} bs=512 count=1 boot2.bin: boot2.out - objcopy -S -O binary boot2.out boot2.bin + ${OBJCOPY} -S -O binary boot2.out ${.TARGET} - boot2.out: ../btx/lib/crt0.o boot2.o sio.o - ld -Ttext 0x2000 -o boot2.out - - boot2.o: boot2.s - ${CC} ${ACFLAGS} -c boot2.s - - boot2.s: boot2.c boot2.h ${.CURDIR}/../../common/ufsread.c - ${CC} ${CFLAGS} -S -o boot2.s.tmp ${.CURDIR}/boot2.c - sed -e '/align/d' -e '/nop/d' "MISSING" boot2.s.tmp > boot2.s - rm -f boot2.s.tmp + boot2.out: ${BTXCRT} boot2.o sio.o ashldi3.o + ${LD} ${LD_FLAGS} --defsym ORG=${ORG2} -T ${LDSCRIPT} -o ${.TARGET} ${.ALLSRC} boot2.h: boot1.out ${NM} -t d ${.ALLSRC} | awk '/([0-9])+ T xread/ \ { x = $$1 - ORG1; \ printf("#define XREADORG %#x\n", REL1 + x) }' \ ORG1=`printf "%d" ${ORG1}` \ REL1=`printf "%d" ${REL1}` > ${.TARGET} .... -.[.filename]#sys/boot/i386/boot2/Makefile# [[boot-boot1-make-boot2]] +.[.filename]#stand/i386/boot2/Makefile# [[boot-boot1-make-boot2]] The mechanism for building [.filename]#boot2# is far more elaborate. Let us point out the most relevant facts. The dependency list is as follows: [.programlisting] .... boot2: boot2.ld - boot2.ld: boot2.ldr boot2.bin ${BTXDIR}/btx/btx + boot2.ld: boot2.ldr boot2.bin ${BTXDIR} boot2.bin: boot2.out - boot2.out: ${BTXDIR}/lib/crt0.o boot2.o sio.o - boot2.o: boot2.s - boot2.s: boot2.c boot2.h ${.CURDIR}/../../common/ufsread.c + boot2.out: ${BTXDIR} boot2.o sio.o ashldi3.o boot2.h: boot1.out .... -.[.filename]#sys/boot/i386/boot2/Makefile# [[boot-boot1-make-boot2-more]] +.[.filename]#stand/i386/boot2/Makefile# [[boot-boot1-make-boot2-more]] Note that initially there is no header file [.filename]#boot2.h#, but its creation depends on [.filename]#boot1.out#, which we already have. The rule for its creation is a bit terse, but the important thing is that the output, [.filename]#boot2.h#, is something like this: [.programlisting] .... #define XREADORG 0x725 .... -.[.filename]#sys/boot/i386/boot2/boot2.h# [[boot-boot1-make-boot2h]] +.[.filename]#stand/i386/boot2/boot2.h# [[boot-boot1-make-boot2h]] Recall that [.filename]#boot1# was relocated (i.e., copied from `0x7c00` to `0x700`). This relocation will now make sense, because as we will see, the BTX server reclaims some memory, including the space where [.filename]#boot1# was originally loaded. However, the BTX server needs access to [.filename]#boot1#'s `xread` function; this function, according to the output of [.filename]#boot2.h#, is at location `0x725`. Indeed, the BTX server uses the `xread` function from [.filename]#boot1#'s relocated code. This function is now accessible from within the [.filename]#boot2# client. -We next build [.filename]#boot2.s# from files [.filename]#boot2.h#, [.filename]#boot2.c# and [.filename]#/usr/src/sys/boot/common/ufsread.c#. The rule for this is to compile the code in [.filename]#boot2.c# (which includes [.filename]#boot2.h# and [.filename]#ufsread.c#) into assembly code. Having [.filename]#boot2.s#, the next rule assembles [.filename]#boot2.s#, creating the object file [.filename]#boot2.o#. The next rule directs the linker to link various files ([.filename]#crt0.o#, [.filename]#boot2.o# and [.filename]#sio.o#). Note that the output file, [.filename]#boot2.out#, is linked to execute at address `0x2000`. Recall that [.filename]#boot2# will be executed in user mode, within a special user segment set up by the BTX server. This segment starts at `0xa000`. Also, remember that the [.filename]#boot2# portion of [.filename]#boot# was copied to address `0xc000`, that is, offset `0x2000` from the start of the user segment, so [.filename]#boot2# will work properly when we transfer control to it. Next, [.filename]#boot2.bin# is created from [.filename]#boot2.out# by stripping its symbols and format information; boot2.bin is a _raw_ binary. Now, note that a file [.filename]#boot2.ldr# is created as a 512-byte file full of zeros. This space is reserved for the bsdlabel. +The next rule directs the linker to link various files ([.filename]#ashldi3.o#, [.filename]#boot2.o# and [.filename]#sio.o#). Note that the output file, [.filename]#boot2.out#, is linked to execute at address `0x2000` (${ORG2}). Recall that [.filename]#boot2# will be executed in user mode, within a special user segment set up by the BTX server. This segment starts at `0xa000`. Also, remember that the [.filename]#boot2# portion of [.filename]#boot# was copied to address `0xc000`, that is, offset `0x2000` from the start of the user segment, so [.filename]#boot2# will work properly when we transfer control to it. Next, [.filename]#boot2.bin# is created from [.filename]#boot2.out# by stripping its symbols and format information; boot2.bin is a _raw_ binary. Now, note that a file [.filename]#boot2.ldr# is created as a 512-byte file full of zeros. This space is reserved for the bsdlabel. -Now that we have files [.filename]#boot1#, [.filename]#boot2.bin# and [.filename]#boot2.ldr#, only the BTX server is missing before creating the all-in-one [.filename]#boot# file. The BTX server is located in [.filename]#/usr/src/sys/boot/i386/btx/btx#; it has its own [.filename]#Makefile# with its own set of rules for building. The important thing to notice is that it is also compiled as a _raw_ binary, and that it is linked to execute at address `0x9000`. The details can be found in [.filename]#/usr/src/sys/boot/i386/btx/btx/Makefile#. +Now that we have files [.filename]#boot1#, [.filename]#boot2.bin# and [.filename]#boot2.ldr#, only the BTX server is missing before creating the all-in-one [.filename]#boot# file. The BTX server is located in [.filename]#stand/i386/btx/btx#; it has its own [.filename]#Makefile# with its own set of rules for building. The important thing to notice is that it is also compiled as a _raw_ binary, and that it is linked to execute at address `0x9000`. The details can be found in [.filename]#stand/i386/btx/btx/Makefile#. Having the files that comprise the [.filename]#boot# program, the final step is to _merge_ them. This is done by a special program called [.filename]#btxld# (source located in [.filename]#/usr/src/usr.sbin/btxld#). Some arguments to this program include the name of the output file ([.filename]#boot#), its entry point (`0x2000`) and its file format (raw binary). The various files are finally merged by this utility into the file [.filename]#boot#, which consists of [.filename]#boot1#, [.filename]#boot2#, the `bsdlabel` and the BTX server. This file, which takes exactly 16 sectors, or 8192 bytes, is what is actually written to the beginning of the FreeBSD slice during installation. Let us now proceed to study the BTX server program. The BTX server prepares a simple environment and switches from 16-bit real mode to 32-bit protected mode, right before passing control to the client. This includes initializing and updating the following data structures: * Modifies the `Interrupt Vector Table (IVT)`. The IVT provides exception and interrupt handlers for Real-Mode code. * The `Interrupt Descriptor Table (IDT)` is created. Entries are provided for processor exceptions, hardware interrupts, two system calls and V86 interface. The IDT provides exception and interrupt handlers for Protected-Mode code. * A `Task-State Segment (TSS)` is created. This is necessary because the processor works in the _least_ privileged level when executing the client ([.filename]#boot2#), but in the _most_ privileged level when executing the BTX server. * The GDT (Global Descriptor Table) is set up. Entries (descriptors) are provided for supervisor code and data, user code and data, and real-mode code and data. footnote:[Real-mode code and data are necessary when switching back to real mode from protected mode, as suggested by the Intel manuals.] Let us now start studying the actual implementation. Recall that [.filename]#boot1# made a jump to address `0x9010`, the BTX server's entry point. Before studying program execution there, note that the BTX server has a special header at address range `0x9000-0x900f`, right before its entry point. This header is defined as follows: [.programlisting] .... start: # Start of code /* * BTX header. */ btx_hdr: .byte 0xeb # Machine ID .byte 0xe # Header size .ascii "BTX" # Magic .byte 0x1 # Major version .byte 0x2 # Minor version .byte BTX_FLAGS # Flags .word PAG_CNT-MEM_ORG>>0xc # Paging control .word break-start # Text size .long 0x0 # Entry address .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-header]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-header]] Note the first two bytes are `0xeb` and `0xe`. In the IA-32 architecture, these two bytes are interpreted as a relative jump past the header into the entry point, so in theory, [.filename]#boot1# could jump here (address `0x9000`) instead of address `0x9010`. Note that the last field in the BTX header is a pointer to the client's ([.filename]#boot2#) entry point. This field is patched at link time. Immediately following the header is the BTX server's entry point: [.programlisting] .... /* * Initialization routine. */ init: cli # Disable interrupts xor %ax,%ax # Zero/segment mov %ax,%ss # Set up - mov $0x1800,%sp # stack + mov $MEM_ESP0,%sp # stack mov %ax,%es # Address mov %ax,%ds # data pushl $0x2 # Clear popfl # flags .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-init]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-init]] This code disables interrupts, sets up a working stack (starting at address `0x1800`) and clears the flags in the EFLAGS register. Note that the `popfl` instruction pops out a doubleword (4 bytes) from the stack and places it in the EFLAGS register. As the value actually popped is `2`, the EFLAGS register is effectively cleared (IA-32 requires that bit 2 of the EFLAGS register always be 1). Our next code block clears (sets to `0`) the memory range `0x5e00-0x8fff`. This range is where the various data structures will be created: [.programlisting] .... /* * Initialize memory. */ - mov $0x5e00,%di # Memory to initialize - mov $(0x9000-0x5e00)/2,%cx # Words to zero + mov $MEM_IDT,%di # Memory to initialize + mov $(MEM_ORG-MEM_IDT)/2,%cx # Words to zero rep # Zero-fill stosw # memory .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-clear-mem]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-clear-mem]] Recall that [.filename]#boot1# was originally loaded to address `0x7c00`, so, with this memory initialization, that copy effectively disappeared. However, also recall that [.filename]#boot1# was relocated to `0x700`, so _that_ copy is still in memory, and the BTX server will make use of it. Next, the real-mode IVT (Interrupt Vector Table is updated. The IVT is an array of segment/offset pairs for exception and interrupt handlers. The BIOS normally maps hardware interrupts to interrupt vectors `0x8` to `0xf` and `0x70` to `0x77` but, as will be seen, the 8259A Programmable Interrupt Controller, the chip controlling the actual mapping of hardware interrupts to interrupt vectors, is programmed to remap these interrupt vectors from `0x8-0xf` to `0x20-0x27` and from `0x70-0x77` to `0x28-0x2f`. Thus, interrupt handlers are provided for interrupt vectors `0x20-0x2f`. The reason the BIOS-provided handlers are not used directly is because they work in 16-bit real mode, but not 32-bit protected mode. Processor mode will be switched to 32-bit protected mode shortly. However, the BTX server sets up a mechanism to effectively use the handlers provided by the BIOS: [.programlisting] .... /* * Update real mode IDT for reflecting hardware interrupts. */ mov $intr20,%bx # Address first handler mov $0x10,%cx # Number of handlers mov $0x20*4,%di # First real mode IDT entry init.0: mov %bx,(%di) # Store IP inc %di # Address next inc %di # entry stosw # Store CS add $4,%bx # Next handler loop init.0 # Next IRQ .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-ivt]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-ivt]] The next block creates the IDT (Interrupt Descriptor Table). The IDT is analogous, in protected mode, to the IVT in real mode. That is, the IDT describes the various exception and interrupt handlers used when the processor is executing in protected mode. In essence, it also consists of an array of segment/offset pairs, although the structure is somewhat more complex, because segments in protected mode are different than in real mode, and various protection mechanisms apply: [.programlisting] .... /* * Create IDT. */ - mov $0x5e00,%di # IDT's address + mov $MEM_IDT,%di # IDT's address mov $idtctl,%si # Control string init.1: lodsb # Get entry cbw # count xchg %ax,%cx # as word jcxz init.4 # If done lodsb # Get segment xchg %ax,%dx # P:DPL:type lodsw # Get control xchg %ax,%bx # set lodsw # Get handler offset mov $SEL_SCODE,%dh # Segment selector init.2: shr %bx # Handle this int? jnc init.3 # No mov %ax,(%di) # Set handler offset mov %dh,0x2(%di) # and selector mov %dl,0x5(%di) # Set P:DPL:type add $0x4,%ax # Next handler init.3: lea 0x8(%di),%di # Next entry loop init.2 # Till set done jmp init.1 # Continue .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-idt]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-idt]] Each entry in the `IDT` is 8 bytes long. Besides the segment/offset information, they also describe the segment type, privilege level, and whether the segment is present in memory or not. The construction is such that interrupt vectors from `0` to `0xf` (exceptions) are handled by function `intx00`; vector `0x10` (also an exception) is handled by `intx10`; hardware interrupts, which are later configured to start at interrupt vector `0x20` all the way to interrupt vector `0x2f`, are handled by function `intx20`. Lastly, interrupt vector `0x30`, which is used for system calls, is handled by `intx30`, and vectors `0x31` and `0x32` are handled by `intx31`. It must be noted that only descriptors for interrupt vectors `0x30`, `0x31` and `0x32` are given privilege level 3, the same privilege level as the [.filename]#boot2# client, which means the client can execute a software-generated interrupt to this vectors through the `int` instruction without failing (this is the way [.filename]#boot2# use the services provided by the BTX server). Also, note that _only_ software-generated interrupts are protected from code executing in lesser privilege levels. Hardware-generated interrupts and processor-generated exceptions are _always_ handled adequately, regardless of the actual privileges involved. The next step is to initialize the TSS (Task-State Segment). The TSS is a hardware feature that helps the operating system or executive software implement multitasking functionality through process abstraction. The IA-32 architecture demands the creation and use of _at least_ one TSS if multitasking facilities are used or different privilege levels are defined. Since the [.filename]#boot2# client is executed in privilege level 3, but the BTX server runs in privilege level 0, a TSS must be defined: [.programlisting] .... /* * Initialize TSS. */ init.4: movb $_ESP0H,TSS_ESP0+1(%di) # Set ESP0 movb $SEL_SDATA,TSS_SS0(%di) # Set SS0 movb $_TSSIO,TSS_MAP(%di) # Set I/O bit map base .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-tss]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-tss]] Note that a value is given for the Privilege Level 0 stack pointer and stack segment in the TSS. This is needed because, if an interrupt or exception is received while executing [.filename]#boot2# in Privilege Level 3, a change to Privilege Level 0 is automatically performed by the processor, so a new working stack is needed. Finally, the I/O Map Base Address field of the TSS is given a value, which is a 16-bit offset from the beginning of the TSS to the I/O Permission Bitmap and the Interrupt Redirection Bitmap. After the IDT and TSS are created, the processor is ready to switch to protected mode. This is done in the next block: [.programlisting] .... /* * Bring up the system. */ mov $0x2820,%bx # Set protected mode callw setpic # IRQ offsets lidt idtdesc # Set IDT lgdt gdtdesc # Set GDT mov %cr0,%eax # Switch to protected inc %ax # mode mov %eax,%cr0 # ljmp $SEL_SCODE,$init.8 # To 32-bit code .code32 init.8: xorl %ecx,%ecx # Zero movb $SEL_SDATA,%cl # To 32-bit movw %cx,%ss # stack .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-prot]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-prot]] First, a call is made to `setpic` to program the 8259A PIC (Programmable Interrupt Controller). This chip is connected to multiple hardware interrupt sources. Upon receiving an interrupt from a device, it signals the processor with the appropriate interrupt vector. This can be customized so that specific interrupts are associated with specific interrupt vectors, as explained before. Next, the IDTR (Interrupt Descriptor Table Register) and GDTR (Global Descriptor Table Register) are loaded with the instructions `lidt` and `lgdt`, respectively. These registers are loaded with the base address and limit address for the IDT and GDT. The following three instructions set the Protection Enable (PE) bit of the `%cr0` register. This effectively switches the processor to 32-bit protected mode. Next, a long jump is made to `init.8` using segment selector SEL_SCODE, which selects the Supervisor Code Segment. The processor is effectively executing in CPL 0, the most privileged level, after this jump. Finally, the Supervisor Data Segment is selected for the stack by assigning the segment selector SEL_SDATA to the `%ss` register. This data segment also has a privilege level of `0`. Our last code block is responsible for loading the TR (Task Register) with the segment selector for the TSS we created earlier, and setting the User Mode environment before passing execution control to the [.filename]#boot2# client. [.programlisting] .... /* * Launch user task. */ movb $SEL_TSS,%cl # Set task ltr %cx # register - movl $0xa000,%edx # User base address + movl $MEM_USR,%edx # User base address movzwl %ss:BDA_MEM,%eax # Get free memory shll $0xa,%eax # To bytes subl $ARGSPACE,%eax # Less arg space subl %edx,%eax # Less base movb $SEL_UDATA,%cl # User data selector pushl %ecx # Set SS pushl %eax # Set ESP push $0x202 # Set flags (IF set) push $SEL_UCODE # Set CS pushl btx_hdr+0xc # Set EIP pushl %ecx # Set GS pushl %ecx # Set FS pushl %ecx # Set DS pushl %ecx # Set ES pushl %edx # Set EAX movb $0x7,%cl # Set remaining init.9: push $0x0 # general loop init.9 # registers +#ifdef BTX_SERIAL + call sio_init # setup the serial console +#endif popa # and initialize popl %es # Initialize popl %ds # user popl %fs # segment popl %gs # registers iret # To user mode .... -.[.filename]#sys/boot/i386/btx/btx/btx.S# [[btx-end]] +.[.filename]#stand/i386/btx/btx/btx.S# [[btx-end]] Note that the client's environment include a stack segment selector and stack pointer (registers `%ss` and `%esp`). Indeed, once the TR is loaded with the appropriate stack segment selector (instruction `ltr`), the stack pointer is calculated and pushed onto the stack along with the stack's segment selector. Next, the value `0x202` is pushed onto the stack; it is the value that the EFLAGS will get when control is passed to the client. Also, the User Mode code segment selector and the client's entry point are pushed. Recall that this entry point is patched in the BTX header at link time. Finally, segment selectors (stored in register `%ecx`) for the segment registers `%gs, %fs, %ds and %es` are pushed onto the stack, along with the value at `%edx` (`0xa000`). Keep in mind the various values that have been pushed onto the stack (they will be popped out shortly). Next, values for the remaining general purpose registers are also pushed onto the stack (note the `loop` that pushes the value `0` seven times). Now, values will be started to be popped out of the stack. First, the `popa` instruction pops out of the stack the latest seven values pushed. They are stored in the general purpose registers in order `%edi, %esi, %ebp, %ebx, %edx, %ecx, %eax`. Then, the various segment selectors pushed are popped into the various segment registers. Five values still remain on the stack. They are popped when the `iret` instruction is executed. This instruction first pops the value that was pushed from the BTX header. This value is a pointer to [.filename]#boot2#'s entry point. It is placed in the register `%eip`, the instruction pointer register. Next, the segment selector for the User Code Segment is popped and copied to register `%cs`. Remember that this segment's privilege level is 3, the least privileged level. This means that we must provide values for the stack of this privilege level. This is why the processor, besides further popping the value for the EFLAGS register, does two more pops out of the stack. These values go to the stack pointer (`%esp`) and the stack segment (`%ss`). Now, execution continues at ``boot0``'s entry point. It is important to note how the User Code Segment is defined. This segment's _base address_ is set to `0xa000`. This means that code memory addresses are _relative_ to address 0xa000; if code being executed is fetched from address `0x2000`, the _actual_ memory addressed is `0xa000+0x2000=0xc000`. [[boot2]] == boot2 Stage `boot2` defines an important structure, `struct bootinfo`. This structure is initialized by `boot2` and passed to the loader, and then further to the kernel. Some nodes of this structures are set by `boot2`, the rest by the loader. This structure, among other information, contains the kernel filename, BIOS harddisk geometry, BIOS drive number for boot device, physical memory available, `envp` pointer etc. The definition for it is: [.programlisting] .... /usr/include/machine/bootinfo.h: struct bootinfo { u_int32_t bi_version; u_int32_t bi_kernelname; /* represents a char * */ u_int32_t bi_nfs_diskless; /* struct nfs_diskless * */ /* End of fields that are always present. */ #define bi_endcommon bi_n_bios_used u_int32_t bi_n_bios_used; u_int32_t bi_bios_geom[N_BIOS_GEOM]; u_int32_t bi_size; u_int8_t bi_memsizes_valid; u_int8_t bi_bios_dev; /* bootdev BIOS unit number */ u_int8_t bi_pad[2]; u_int32_t bi_basemem; u_int32_t bi_extmem; u_int32_t bi_symtab; /* struct symtab * */ u_int32_t bi_esymtab; /* struct symtab * */ /* Items below only from advanced bootloader */ u_int32_t bi_kernend; /* end of kernel space */ u_int32_t bi_envp; /* environment */ u_int32_t bi_modulep; /* preloaded modules */ }; .... `boot2` enters into an infinite loop waiting for user input, then calls `load()`. If the user does not press anything, the loop breaks by a timeout, so `load()` will load the default file ([.filename]#/boot/loader#). Functions `ino_t lookup(char *filename)` and `int xfsread(ino_t inode, void *buf, size_t nbyte)` are used to read the content of a file into memory. [.filename]#/boot/loader# is an ELF binary, but where the ELF header is prepended with [.filename]#a.out#'s `struct exec` structure. `load()` scans the loader's ELF header, loading the content of [.filename]#/boot/loader# into memory, and passing the execution to the loader's entry: [.programlisting] .... -sys/boot/i386/boot2/boot2.c: +stand/i386/boot2/boot2.c: __exec((caddr_t)addr, RB_BOOTINFO | (opts & RBX_MASK), - MAKEBOOTDEV(dev_maj[dsk.type], 0, dsk.slice, dsk.unit, dsk.part), + MAKEBOOTDEV(dev_maj[dsk.type], dsk.slice, dsk.unit, dsk.part), 0, 0, 0, VTOP(&bootinfo)); .... [[boot-loader]] == loader Stage loader is a BTX client as well. I will not describe it here in detail, there is a comprehensive man page written by Mike Smith, man:loader[8]. The underlying mechanisms and BTX were discussed above. The main task for the loader is to boot the kernel. When the kernel is loaded into memory, it is being called by the loader: [.programlisting] .... -sys/boot/common/boot.c: +stand/common/boot.c: /* Call the exec handler from the loader matching the kernel */ - module_formats[km->m_loader]->l_exec(km); + file_formats[fp->f_loader]->l_exec(fp); .... [[boot-kernel]] == Kernel Initialization -Let us take a look at the command that links the kernel. This will help identify the exact location where the loader passes execution to the kernel. This location is the kernel's actual entry point. +Let us take a look at the command that links the kernel. This will help identify the exact location where the loader passes execution to the kernel. This location is the kernel's actual entry point. This command is now excluded from [.filename]#sys/conf/Makefile.i386#. The content that interests us can be found in [.filename]#/usr/obj/usr/src/i386.i386/sys/GENERIC/#. [.programlisting] .... -sys/conf/Makefile.i386: -ld -elf -Bdynamic -T /usr/src/sys/conf/ldscript.i386 -export-dynamic \ --dynamic-linker /red/herring -o kernel -X locore.o \ +/usr/obj/usr/src/i386.i386/sys/GENERIC/kernel.meta: +ld -m elf_i386_fbsd -Bdynamic -T /usr/src/sys/conf/ldscript.i386 --build-id=sha1 --no-warn-mismatch \ +--warn-common --export-dynamic --dynamic-linker /red/herring -X -o kernel locore.o .... A few interesting things can be seen here. First, the kernel is an ELF dynamically linked binary, but the dynamic linker for kernel is [.filename]#/red/herring#, which is definitely a bogus file. Second, taking a look at the file [.filename]#sys/conf/ldscript.i386# gives an idea about what ld options are used when compiling a kernel. Reading through the first few lines, the string [.programlisting] .... sys/conf/ldscript.i386: ENTRY(btext) .... says that a kernel's entry point is the symbol `btext`. This symbol is defined in [.filename]#locore.s#: [.programlisting] .... sys/i386/i386/locore.s: .text /********************************************************************** * * This is where the bootblocks start us, set the ball rolling... * */ NON_GPROF_ENTRY(btext) .... First, the register EFLAGS is set to a predefined value of 0x00000002. Then all the segment registers are initialized: [.programlisting] .... sys/i386/i386/locore.s: /* Don't trust what the BIOS gives for eflags. */ pushl $PSL_KERNEL popfl /* * Don't trust what the BIOS gives for %fs and %gs. Trust the bootstrap * to set %cs, %ds, %es and %ss. */ mov %ds, %ax mov %ax, %fs mov %ax, %gs .... -btext calls the routines `recover_bootinfo()`, `identify_cpu()`, `create_pagetables()`, which are also defined in [.filename]#locore.s#. Here is a description of what they do: +btext calls the routines `recover_bootinfo()`, `identify_cpu()`, which are also defined in [.filename]#locore.s#. Here is a description of what they do: [.informaltable] [cols="1,1", frame="none"] |=== |`recover_bootinfo` |This routine parses the parameters to the kernel passed from the bootstrap. The kernel may have been booted in 3 ways: by the loader, described above, by the old disk boot blocks, or by the old diskless boot procedure. This function determines the booting method, and stores the `struct bootinfo` structure into the kernel memory. |`identify_cpu` -|This functions tries to find out what CPU it is running on, storing the value found in a variable `_cpu`. - -|`create_pagetables` -|This function allocates and fills out a Page Table Directory at the top of the kernel memory area. +|This function tries to find out what CPU it is running on, storing the value found in a variable `_cpu`. |=== The next steps are enabling VME, if the CPU supports it: [.programlisting] .... - testl $CPUID_VME, R(_cpu_feature) - jz 1f - movl %cr4, %eax - orl $CR4_VME, %eax - movl %eax, %cr4 +sys/i386/i386/mpboot.s: + testl $CPUID_VME,%edx + jz 3f + orl $CR4_VME,%eax +3: movl %eax,%cr4 .... Then, enabling paging: [.programlisting] .... +sys/i386/i386/mpboot.s: /* Now enable paging */ - movl R(_IdlePTD), %eax + movl IdlePTD_nopae, %eax movl %eax,%cr3 /* load ptd addr into mmu */ movl %cr0,%eax /* get control word */ orl $CR0_PE|CR0_PG,%eax /* enable paging */ movl %eax,%cr0 /* and let's page NOW! */ .... The next three lines of code are because the paging was set, so the jump is needed to continue the execution in virtualized address space: [.programlisting] .... - pushl $begin /* jump to high virtualized address */ +sys/i386/i386/mpboot.s: + pushl $mp_begin /* jump to high mem */ ret /* now running relocated at KERNBASE where the system is linked to run */ -begin: +mp_begin: /* now running relocated at KERNBASE */ .... The function `init386()` is called with a pointer to the first free physical page, after that `mi_startup()`. `init386` is an architecture dependent initialization function, and `mi_startup()` is an architecture independent one (the 'mi_' prefix stands for Machine Independent). The kernel never returns from `mi_startup()`, and by calling it, the kernel finishes booting: [.programlisting] .... sys/i386/i386/locore.s: - movl physfree, %esi - pushl %esi /* value of first for init386(first) */ - call _init386 /* wire 386 chip for unix operation */ - call _mi_startup /* autoconfiguration, mountroot etc */ - hlt /* never returns to here */ + pushl physfree /* value of first for init386(first) */ + call init386 /* wire 386 chip for unix operation */ + addl $4,%esp + movl %eax,%esp /* Switch to true top of stack. */ + call mi_startup /* autoconfiguration, mountroot etc */ + /* NOTREACHED */ .... === `init386()` `init386()` is defined in [.filename]#sys/i386/i386/machdep.c# and performs low-level initialization specific to the i386 chip. The switch to protected mode was performed by the loader. The loader has created the very first task, in which the kernel continues to operate. Before looking at the code, consider the tasks the processor must complete to initialize protected mode execution: * Initialize the kernel tunable parameters, passed from the bootstrapping program. * Prepare the GDT. * Prepare the IDT. * Initialize the system console. * Initialize the DDB, if it is compiled into kernel. * Initialize the TSS. * Prepare the LDT. -* Set up proc0's pcb. +* Set up thread0's pcb. `init386()` initializes the tunable parameters passed from bootstrap by setting the environment pointer (envp) and calling `init_param1()`. The envp pointer has been passed from loader in the `bootinfo` structure: [.programlisting] .... sys/i386/i386/machdep.c: - kern_envp = (caddr_t)bootinfo.bi_envp + KERNBASE; - /* Init basic tunables, hz etc */ init_param1(); .... `init_param1()` is defined in [.filename]#sys/kern/subr_param.c#. That file has a number of sysctls, and two functions, `init_param1()` and `init_param2()`, that are called from `init386()`: [.programlisting] .... sys/kern/subr_param.c: - hz = HZ; + hz = -1; TUNABLE_INT_FETCH("kern.hz", &hz); + if (hz == -1) + hz = vm_guest > VM_GUEST_NO ? HZ_VM : HZ; .... TUNABLE__FETCH is used to fetch the value from the environment: [.programlisting] .... /usr/src/sys/sys/kernel.h: #define TUNABLE_INT_FETCH(path, var) getenv_int((path), (var)) .... Sysctl `kern.hz` is the system clock tick. Additionally, these sysctls are set by `init_param1()`: `kern.maxswzone, kern.maxbcache, kern.maxtsiz, kern.dfldsiz, kern.maxdsiz, kern.dflssiz, kern.maxssiz, kern.sgrowsiz`. Then `init386()` prepares the Global Descriptors Table (GDT). Every task on an x86 is running in its own virtual address space, and this space is addressed by a segment:offset pair. Say, for instance, the current instruction to be executed by the processor lies at CS:EIP, then the linear virtual address for that instruction would be "the virtual address of code segment CS" + EIP. For convenience, segments begin at virtual address 0 and end at a 4Gb boundary. Therefore, the instruction's linear virtual address for this example would just be the value of EIP. Segment registers such as CS, DS etc are the selectors, i.e., indexes, into GDT (to be more precise, an index is not a selector itself, but the INDEX field of a selector). FreeBSD's GDT holds descriptors for 15 selectors per CPU: [.programlisting] .... sys/i386/i386/machdep.c: -union descriptor gdt[NGDT * MAXCPU]; /* global descriptor table */ +union descriptor gdt0[NGDT]; /* initial global descriptor table */ +union descriptor *gdt = gdt0; /* global descriptor table */ -sys/i386/include/segments.h: +sys/x86/include/segments.h: /* * Entries in the Global Descriptor Table (GDT) */ #define GNULL_SEL 0 /* Null Descriptor */ -#define GCODE_SEL 1 /* Kernel Code Descriptor */ -#define GDATA_SEL 2 /* Kernel Data Descriptor */ -#define GPRIV_SEL 3 /* SMP Per-Processor Private Data */ -#define GPROC0_SEL 4 /* Task state process slot zero and up */ -#define GLDT_SEL 5 /* LDT - eventually one per process */ -#define GUSERLDT_SEL 6 /* User LDT */ -#define GTGATE_SEL 7 /* Process task switch gate */ +#define GPRIV_SEL 1 /* SMP Per-Processor Private Data */ +#define GUFS_SEL 2 /* User %fs Descriptor (order critical: 1) */ +#define GUGS_SEL 3 /* User %gs Descriptor (order critical: 2) */ +#define GCODE_SEL 4 /* Kernel Code Descriptor (order critical: 1) */ +#define GDATA_SEL 5 /* Kernel Data Descriptor (order critical: 2) */ +#define GUCODE_SEL 6 /* User Code Descriptor (order critical: 3) */ +#define GUDATA_SEL 7 /* User Data Descriptor (order critical: 4) */ #define GBIOSLOWMEM_SEL 8 /* BIOS low memory access (must be entry 8) */ -#define GPANIC_SEL 9 /* Task state to consider panic from */ -#define GBIOSCODE32_SEL 10 /* BIOS interface (32bit Code) */ -#define GBIOSCODE16_SEL 11 /* BIOS interface (16bit Code) */ -#define GBIOSDATA_SEL 12 /* BIOS interface (Data) */ -#define GBIOSUTIL_SEL 13 /* BIOS interface (Utility) */ -#define GBIOSARGS_SEL 14 /* BIOS interface (Arguments) */ +#define GPROC0_SEL 9 /* Task state process slot zero and up */ +#define GLDT_SEL 10 /* Default User LDT */ +#define GUSERLDT_SEL 11 /* User LDT */ +#define GPANIC_SEL 12 /* Task state to consider panic from */ +#define GBIOSCODE32_SEL 13 /* BIOS interface (32bit Code) */ +#define GBIOSCODE16_SEL 14 /* BIOS interface (16bit Code) */ +#define GBIOSDATA_SEL 15 /* BIOS interface (Data) */ +#define GBIOSUTIL_SEL 16 /* BIOS interface (Utility) */ +#define GBIOSARGS_SEL 17 /* BIOS interface (Arguments) */ +#define GNDIS_SEL 18 /* For the NDIS layer */ +#define NGDT 19 .... -Note that those #defines are not selectors themselves, but just a field INDEX of a selector, so they are exactly the indices of the GDT. for example, an actual selector for the kernel code (GCODE_SEL) has the value 0x08. +Note that those #defines are not selectors themselves, but just a field INDEX of a selector, so they are exactly the indices of the GDT. for example, an actual selector for the kernel code (GCODE_SEL) has the value 0x20. The next step is to initialize the Interrupt Descriptor Table (IDT). This table is referenced by the processor when a software or hardware interrupt occurs. For example, to make a system call, user application issues the `INT 0x80` instruction. This is a software interrupt, so the processor's hardware looks up a record with index 0x80 in the IDT. This record points to the routine that handles this interrupt, in this particular case, this will be the kernel's syscall gate. The IDT may have a maximum of 256 (0x100) records. The kernel allocates NIDT records for the IDT, where NIDT is the maximum (256): [.programlisting] .... sys/i386/i386/machdep.c: static struct gate_descriptor idt0[NIDT]; struct gate_descriptor *idt = &idt0[0]; /* interrupt descriptor table */ .... For each interrupt, an appropriate handler is set. The syscall gate for `INT 0x80` is set as well: [.programlisting] .... sys/i386/i386/machdep.c: - setidt(0x80, &IDTVEC(int0x80_syscall), - SDT_SYS386TGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); + setidt(IDT_SYSCALL, &IDTVEC(int0x80_syscall), + SDT_SYS386IGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); .... So when a userland application issues the `INT 0x80` instruction, control will transfer to the function `_Xint0x80_syscall`, which is in the kernel code segment and will be executed with supervisor privileges. Console and DDB are then initialized: [.programlisting] .... sys/i386/i386/machdep.c: cninit(); /* skipped */ -#ifdef DDB - kdb_init(); + kdb_init(); +#ifdef KDB if (boothowto & RB_KDB) - Debugger("Boot flags requested debugger"); + kdb_enter(KDB_WHY_BOOTFLAGS, "Boot flags requested debugger"); #endif .... The Task State Segment is another x86 protected mode structure, the TSS is used by the hardware to store task information when a task switch occurs. The Local Descriptors Table is used to reference userland code and data. Several selectors are defined to point to the LDT, they are the system call gates and the user code and data selectors: [.programlisting] .... -/usr/include/machine/segments.h: +sys/x86/include/segments.h: #define LSYS5CALLS_SEL 0 /* forced by intel BCS */ #define LSYS5SIGR_SEL 1 -#define L43BSDCALLS_SEL 2 /* notyet */ #define LUCODE_SEL 3 -#define LSOL26CALLS_SEL 4 /* Solaris >= 2.6 system call gate */ #define LUDATA_SEL 5 -/* separate stack, es,fs,gs sels ? */ -/* #define LPOSIXCALLS_SEL 5*/ /* notyet */ -#define LBSDICALLS_SEL 16 /* BSDI system call gate */ -#define NLDT (LBSDICALLS_SEL + 1) +#define NLDT (LUDATA_SEL + 1) .... -Next, proc0's Process Control Block (`struct pcb`) structure is initialized. proc0 is a `struct proc` structure that describes a kernel process. It is always present while the kernel is running, therefore it is declared as global: +Next, proc0's Process Control Block (`struct pcb`) structure is initialized. proc0 is a `struct proc` structure that describes a kernel process. It is always present while the kernel is running, therefore it is linked with thread0: [.programlisting] .... -sys/kern/kern_init.c: - struct proc proc0; +sys/i386/i386/machdep.c: +register_t +init386(int first) +{ + /* ... skipped ... */ + + proc_linkup0(&proc0, &thread0); + /* ... skipped ... */ +} .... The structure `struct pcb` is a part of a proc structure. It is defined in [.filename]#/usr/include/machine/pcb.h# and has a process's information specific to the i386 architecture, such as registers values. === `mi_startup()` This function performs a bubble sort of all the system initialization objects and then calls the entry of each object one by one: [.programlisting] .... sys/kern/init_main.c: - for (sipp = sysinit; *sipp; sipp++) { + for (sipp = sysinit; sipp < sysinit_end; sipp++) { /* ... skipped ... */ /* Call function */ (*((*sipp)->func))((*sipp)->udata); /* ... skipped ... */ } .... Although the sysinit framework is described in the link:/books/developers-handbook[Developers' Handbook], I will discuss the internals of it. Every system initialization object (sysinit object) is created by calling a SYSINIT() macro. Let us take as example an `announce` sysinit object. This object prints the copyright message: [.programlisting] .... sys/kern/init_main.c: static void print_caddr_t(void *data __unused) { printf("%s", (char *)data); } -SYSINIT(announce, SI_SUB_COPYRIGHT, SI_ORDER_FIRST, print_caddr_t, copyright) +/* ... skipped ... */ +SYSINIT(announce, SI_SUB_COPYRIGHT, SI_ORDER_FIRST, print_caddr_t, copyright); .... -The subsystem ID for this object is SI_SUB_COPYRIGHT (0x0800001), which comes right after the SI_SUB_CONSOLE (0x0800000). So, the copyright message will be printed out first, just after the console initialization. +The subsystem ID for this object is SI_SUB_COPYRIGHT (0x0800001). So, the copyright message will be printed out first, just after the console initialization. Let us take a look at what exactly the macro `SYSINIT()` does. It expands to a `C_SYSINIT()` macro. The `C_SYSINIT()` macro then expands to a static `struct sysinit` structure declaration with another `DATA_SET` macro call: [.programlisting] .... /usr/include/sys/kernel.h: #define C_SYSINIT(uniquifier, subsystem, order, func, ident) \ static struct sysinit uniquifier ## _sys_init = { \ subsystem, \ - order, \ func, \ ident \ }; \ DATA_SET(sysinit_set,uniquifier ## + order, \ func, \ (ident) \ }; \ DATA_WSET(sysinit_set,uniquifier ## _sys_init); #define SYSINIT(uniquifier, subsystem, order, func, ident) \ C_SYSINIT(uniquifier, subsystem, order, \ - (sysinit_cfunc_t)(sysinit_nfunc_t)func, (void *)ident) + (sysinit_cfunc_t)(sysinit_nfunc_t)func, (void *)(ident)) .... -The `DATA_SET()` macro expands to a `MAKE_SET()`, and that macro is the point where all the sysinit magic is hidden: +The `DATA_SET()` macro expands to a `_MAKE_SET()`, and that macro is the point where all the sysinit magic is hidden: [.programlisting] .... /usr/include/linker_set.h: -#define MAKE_SET(set, sym) \ - static void const * const __set_##set##_sym_##sym = sym; \ - __asm(".section .set." #set ",\"aw\""); \ - __asm(".long " #sym); \ - __asm(".previous") -#endif -#define TEXT_SET(set, sym) MAKE_SET(set, sym) -#define DATA_SET(set, sym) MAKE_SET(set, sym) +#define TEXT_SET(set, sym) _MAKE_SET(set, sym) +#define DATA_SET(set, sym) _MAKE_SET(set, sym) .... -In our case, the following declaration will occur: - -[.programlisting] -.... -static struct sysinit announce_sys_init = { - SI_SUB_COPYRIGHT, - SI_ORDER_FIRST, - (sysinit_cfunc_t)(sysinit_nfunc_t) print_caddr_t, - (void *) copyright -}; - -static void const *const __set_sysinit_set_sym_announce_sys_init = - announce_sys_init; -__asm(".section .set.sysinit_set" ",\"aw\""); -__asm(".long " "announce_sys_init"); -__asm(".previous"); -.... - -The first `__asm` instruction will create an ELF section within the kernel's executable. This will happen at kernel link time. The section will have the name `.set.sysinit_set`. The content of this section is one 32-bit value, the address of announce_sys_init structure, and that is what the second `__asm` is. The third `__asm` instruction marks the end of a section. If a directive with the same section name occurred before, the content, i.e., the 32-bit value, will be appended to the existing section, so forming an array of 32-bit pointers. - +After executing these macros, various sections were made in the kernel, including`set.sysinit_set`. Running objdump on a kernel binary, you may notice the presence of such small sections: [source,bash] .... -% objdump -h /kernel - 7 .set.cons_set 00000014 c03164c0 c03164c0 002154c0 2**2 - CONTENTS, ALLOC, LOAD, DATA - 8 .set.kbddriver_set 00000010 c03164d4 c03164d4 002154d4 2**2 - CONTENTS, ALLOC, LOAD, DATA - 9 .set.scrndr_set 00000024 c03164e4 c03164e4 002154e4 2**2 - CONTENTS, ALLOC, LOAD, DATA - 10 .set.scterm_set 0000000c c0316508 c0316508 00215508 2**2 - CONTENTS, ALLOC, LOAD, DATA - 11 .set.sysctl_set 0000097c c0316514 c0316514 00215514 2**2 - CONTENTS, ALLOC, LOAD, DATA - 12 .set.sysinit_set 00000664 c0316e90 c0316e90 00215e90 2**2 - CONTENTS, ALLOC, LOAD, DATA +% llvm-objdump -h /kernel +Sections: +Idx Name Size VMA Type + 10 set_sysctl_set 000021d4 01827078 DATA + 16 set_kbddriver_set 00000010 0182a4d0 DATA + 20 set_scterm_set 0000000c 0182c75c DATA + 21 set_cons_set 00000014 0182c768 DATA + 33 set_scrndr_set 00000024 0182c828 DATA + 41 set_sysinit_set 000014d8 018fabb0 DATA .... -This screen dump shows that the size of .set.sysinit_set section is 0x664 bytes, so `0x664/sizeof(void *)` sysinit objects are compiled into the kernel. The other sections such as `.set.sysctl_set` represent other linker sets. +This screen dump shows that the size of set.sysinit_set section is 0x14d8 bytes, so `0x14d8/sizeof(void *)` sysinit objects are compiled into the kernel. The other sections such as `set.sysctl_set` represent other linker sets. -By defining a variable of type `struct linker_set` the content of `.set.sysinit_set` section will be "collected" into that variable: +By defining a variable of type `struct sysinit` the content of `set.sysinit_set` section will be "collected" into that variable: [.programlisting] .... sys/kern/init_main.c: - extern struct linker_set sysinit_set; /* XXX */ + SET_DECLARE(sysinit_set, struct sysinit); .... -The `struct linker_set` is defined as follows: +The `struct sysinit` is defined as follows: [.programlisting] .... -/usr/include/linker_set.h: - struct linker_set { - int ls_length; - void *ls_items[1]; /* really ls_length of them, trailing NULL */ +sys/sys/kernel.h: + struct sysinit { + enum sysinit_sub_id subsystem; /* subsystem identifier*/ + enum sysinit_elem_order order; /* init order within subsystem*/ + sysinit_cfunc_t func; /* function */ + const void *udata; /* multiplexer/argument */ }; .... -The first node will be equal to the number of a sysinit objects, and the second node will be a NULL-terminated array of pointers to them. - Returning to the `mi_startup()` discussion, it is must be clear now, how the sysinit objects are being organized. The `mi_startup()` function sorts them and calls each. The very last object is the system scheduler: [.programlisting] .... /usr/include/sys/kernel.h: enum sysinit_sub_id { SI_SUB_DUMMY = 0x0000000, /* not executed; for linker*/ SI_SUB_DONE = 0x0000001, /* processed*/ - SI_SUB_CONSOLE = 0x0800000, /* console*/ + SI_SUB_TUNABLES = 0x0700000, /* establish tunable values */ SI_SUB_COPYRIGHT = 0x0800001, /* first use of console*/ ... - SI_SUB_RUN_SCHEDULER = 0xfffffff /* scheduler: no return*/ + SI_SUB_LAST = 0xfffffff /* final initialization */ }; .... -The system scheduler sysinit object is defined in the file [.filename]#sys/vm/vm_glue.c#, and the entry point for that object is `scheduler()`. That function is actually an infinite loop, and it represents a process with PID 0, the swapper process. The proc0 structure, mentioned before, is used to describe it. +The system scheduler sysinit object is defined in the file [.filename]#sys/vm/vm_glue.c#, and the entry point for that object is `scheduler()`. That function is actually an infinite loop, and it represents a process with PID 0, the swapper process. The thread0 structure, mentioned before, is used to describe it. The first user process, called _init_, is created by the sysinit object `init`: [.programlisting] .... sys/kern/init_main.c: static void create_init(const void *udata __unused) { + struct fork_req fr; + struct ucred *newcred, *oldcred; + struct thread *td; int error; - int s; - s = splhigh(); - error = fork1(proc0, RFFDG | RFPROC, initproc); + bzero(&fr, sizeof(fr)); + fr.fr_flags = RFFDG | RFPROC | RFSTOPPED; + fr.fr_procp = &initproc; + error = fork1(&thread0, &fr); if (error) panic("cannot fork init: %d\n", error); - initproc-p_flag |= P_INMEM | P_SYSTEM; - cpu_set_fork_handler(initproc, start_init, NULL); - remrunqueue(initproc); - splx(s); + KASSERT(initproc->p_pid == 1, ("create_init: initproc->p_pid != 1")); + /* divorce init's credentials from the kernel's */ + newcred = crget(); + sx_xlock(&proctree_lock); + PROC_LOCK(initproc); + initproc->p_flag |= P_SYSTEM | P_INMEM; + initproc->p_treeflag |= P_TREE_REAPER; + oldcred = initproc->p_ucred; + crcopy(newcred, oldcred); +#ifdef MAC + mac_cred_create_init(newcred); +#endif +#ifdef AUDIT + audit_cred_proc1(newcred); +#endif + proc_set_cred(initproc, newcred); + td = FIRST_THREAD_IN_PROC(initproc); + crcowfree(td); + td->td_realucred = crcowget(initproc->p_ucred); + td->td_ucred = td->td_realucred; + PROC_UNLOCK(initproc); + sx_xunlock(&proctree_lock); + crfree(oldcred); + cpu_fork_kthread_handler(FIRST_THREAD_IN_PROC(initproc), start_init, NULL); } -SYSINIT(init,SI_SUB_CREATE_INIT, SI_ORDER_FIRST, create_init, NULL) +SYSINIT(init, SI_SUB_CREATE_INIT, SI_ORDER_FIRST, create_init, NULL); .... -The `create_init()` allocates a new process by calling `fork1()`, but does not mark it runnable. When this new process is scheduled for execution by the scheduler, the `start_init()` will be called. That function is defined in [.filename]#init_main.c#. It tries to load and exec the [.filename]#init# binary, probing [.filename]#/sbin/init# first, then [.filename]#/sbin/oinit#, [.filename]#/sbin/init.bak#, and finally [.filename]#/stand/sysinstall#: +The function `create_init()` allocates a new process by calling `fork1()`, but does not mark it runnable. When this new process is scheduled for execution by the scheduler, the `start_init()` will be called. That function is defined in [.filename]#init_main.c#. It tries to load and exec the [.filename]#init# binary, probing [.filename]#/sbin/init# first, then [.filename]#/sbin/oinit#, [.filename]#/sbin/init.bak#, and finally [.filename]#/rescue/init#: [.programlisting] .... sys/kern/init_main.c: static char init_path[MAXPATHLEN] = #ifdef INIT_PATH __XSTRING(INIT_PATH); #else - "/sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall"; + "/sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init"; #endif ....