diff --git a/documentation/config/_default/config.toml b/documentation/config/_default/config.toml --- a/documentation/config/_default/config.toml +++ b/documentation/config/_default/config.toml @@ -12,6 +12,7 @@ ignoreFiles = [ "chapter.adoc$", "contrib-386bsd.adoc$", "contrib-additional.adoc$", "contrib-committers.adoc$", "contrib-corealumni.adoc$", "contrib-develalumni.adoc$", "contrib-develinmemoriam.adoc$", "contrib-portmgralumni.adoc$", "\\.po$" ] enableRobotsTXT = true enableGitInfo = true +timeout = 600000 [params] websiteURL = "https://www.FreeBSD.org" diff --git a/documentation/content/en/books/handbook/linuxemu/_index.adoc b/documentation/content/en/books/handbook/linuxemu/_index.adoc --- a/documentation/content/en/books/handbook/linuxemu/_index.adoc +++ b/documentation/content/en/books/handbook/linuxemu/_index.adoc @@ -51,11 +51,14 @@ [[linuxemu-synopsis]] == Synopsis -FreeBSD provides optional binary compatibility with Linux(R), allowing users to install and run unmodified Linux binaries. -It is available for the i386, amd64, and arm64 architectures. - +FreeBSD provides *optional* binary compatibility with Linux(R), commonly referred to as Linuxulator, allowing users to install and run unmodified Linux binaries. +It is available for the x86 (both 32 and 64 bit) and AArch64 architectures. Some Linux-specific operating system features are not yet supported; this mostly happens with functionality specific to hardware or related to system management, such as cgroups or namespaces. +Before reading this chapter, you should: + +* Know how to install crossref:ports[ports,additional third-party software]. + After reading this chapter, you will know: * How to enable Linux binary compatibility on a FreeBSD system. @@ -63,100 +66,258 @@ * How to install Linux applications on a FreeBSD system. * The implementation details of Linux compatibility in FreeBSD. -Before reading this chapter, you should: - -* Know how to install crossref:ports[ports,additional third-party software]. - [[linuxemu-lbc-install]] == Configuring Linux Binary Compatibility -By default, Linux binary compatibility is not enabled. -To enable it at boot time, add this line to [.filename]#/etc/rc.conf#: +By default, man:linux[4] binary compatibility is not enabled. +This is enough for statically linked Linux binaries to work. +They can be started in the same way native FreeBSD binaries can; they behave almost exactly like native processes and can be traced and debugged the usual way. +ยบ +To enable the Linux ABI at boot time, execute the following command: [.programlisting] .... -linux_enable="YES" +# sysrc linux_enable="YES" .... -Once enabled, it can be started without rebooting by running: +Once enabled, it can be started without rebooting executing the following command: + [source,shell] .... # service linux start .... -The [.filename]#/etc/rc.d/linux# script will load necessary kernel modules and mount filesystems expected by Linux applications under [.filename]#/compat/linux#. -This is enough for statically linked Linux binaries to work. -They can be started in the same way native FreeBSD binaries can; they behave almost exactly like native processes and can be traced and debugged the usual way. +The Linux service will load necessary kernel modules and mount filesystems expected by Linux applications under [.filename]#/compat/linux#. -Linux binaries linked dynamically (which is the vast majority) also require Linux shared libraries to be installed - they can run on top of the FreeBSD kernel, but they cannot use FreeBSD libraries; this is similar to how 32-bit binaries cannot use native 64-bit libraries. -There are several ways of providing those libraries: one can copy them over from an existing Linux installation using the same architecture, install them from FreeBSD packages, or install using man:debootstrap[8] (from package:sysutils/debootstrap[]), and others. +The current content of [.filename]#/compat/linux# can be checked executing the following command: -[[linuxemu-packages]] -== CentOS Base System from FreeBSD Packages +[source,shell] +.... +# ls -l /compat/linux/ +.... -[NOTE] +The output should be similar to the following: + +[.programlisting] +.... +total 1 +dr-xr-xr-x 13 root wheel 512 Apr 11 19:12 dev +dr-xr-xr-x 1 root wheel 0 Apr 11 21:03 proc +dr-xr-xr-x 1 root wheel 0 Apr 11 21:03 sys +.... + +[[linux-userlands]] +== Linux userlands + +Linux software requires more than just an ABI to work. +In order to run Linux software a Linux userland must be installed first. + +[TIP] ==== -This method is not yet available for arm64. +If all that is wanted is to install some software already included in the ports tree like Sublime Text 4, this can be installed and man:pkg[8] will automatically install the required Linux userland. + +For example, to install Sublime Text 4, along with all the Linux libraries it depends on, run this command: + +[source,shell] +.... +# pkg install linux-sublime-text4 +.... ==== -The easiest way to install Linux libraries is to install package:emulators/linux_base-c7[] package or port, which places the CentOS 7-derived base system into [.filename]#/compat/linux#: +[[linuxemu-packages]] +=== CentOS Base System from FreeBSD Packages + +To install the CentOS userland execute the following command: [source,shell] .... # pkg install linux_base-c7 .... -FreeBSD provides packages for some Linux binary applications. -For example, to install Sublime Text 4, along with all the Linux libraries it depends on, run this command: +package:emulators/linux_base-c7[] will place the base system derived from CentOS 7 into [.filename]#/compat/linux#. + +After installing the package, the contents of [.filename]#/compat/linux# can be verified by running the following command to check that the CentOS userland has been installed: + [source,shell] .... -# pkg install linux-sublime-text4 +# ls -l /compat/linux/ +.... + +The output should be similar to the following: + +[.programlisting] +.... +total 30 +lrwxr-xr-x 1 root wheel 7 Apr 11 2018 bin -> usr/bin +drwxr-xr-x 13 root wheel 512 Apr 11 21:10 dev +drwxr-xr-x 25 root wheel 64 Apr 11 21:10 etc +lrwxr-xr-x 1 root wheel 7 Apr 11 2018 lib -> usr/lib +lrwxr-xr-x 1 root wheel 9 Apr 11 2018 lib64 -> usr/lib64 +drwxr-xr-x 2 root wheel 2 Apr 11 21:10 opt +dr-xr-xr-x 1 root wheel 0 Apr 11 21:25 proc +lrwxr-xr-x 1 root wheel 8 Feb 18 02:10 run -> /var/run +lrwxr-xr-x 1 root wheel 8 Apr 11 2018 sbin -> usr/sbin +drwxr-xr-x 2 root wheel 2 Apr 11 21:10 srv +dr-xr-xr-x 1 root wheel 0 Apr 11 21:25 sys +drwxr-xr-x 8 root wheel 9 Apr 11 21:10 usr +drwxr-xr-x 16 root wheel 17 Apr 11 21:10 var .... [[linuxemu-debootstrap]] -== Debian / Ubuntu Base System with man:debootstrap[8] +=== Debian / Ubuntu Base System with debootstrap An alternative way of providing Linux shared libraries is by using package:sysutils/debootstrap[]. This has the advantage of providing a full Debian or Ubuntu distribution. -To use it, follow the instructions at FreeBSD Wiki: https://wiki.freebsd.org/LinuxJails[FreeBSD Wiki - Linux Jails]. -After debootstrapping, man:chroot[8] into the newly created directory and install software in a way typical for the Linux distribution inside, for example: +To install debootstrap execute the following command: [source,shell] .... -# chroot /compat/ubuntu /bin/bash -root@hostname:/# apt update +# pkg install debootstrap .... -It is possible to debootstrap into [.filename]#/compat/linux#, but it is discouraged to avoid collisions with files installed from FreeBSD ports and packages. +man:debootstrap[8] needs man:linux[4] ABI enabled and man:linprocfs[5], man:fdescfs[5] and man:linsysfs[5] modules to be loaded on the system before starting the process. + +To enable the Linux ABI at boot time, execute the following command: + +[.programlisting] +.... +# sysrc linux_enable="YES" +.... + +Once enabled, it can be started without rebooting executing the following command: + +[source,shell] +.... +# service linux start +.... + +Then execute the following command to install Ubuntu or Debian in [.filename]#/compat/ubuntu#: + +[source,shell] +.... +# debootstrap focal /compat/ubuntu +.... + +[NOTE] +==== +While it is technically possible to install into [.filename]#/compat/linux# instead, it's discouraged due to possible clashes with CentOS-based packages. Instead, derive the directory name from the distribution or version name, e.g., [.filename]#/compat/ubuntu#. -If the bootstrapped instance is intended to provide Linux shared libraries without having to explicitly use chroot or jails, one can point the kernel at it by updating the `compat.linux.emul_path` sysctl and adding a line like this to [.filename]#/etc/sysctl.conf#: +==== + +The output should be similar to the following: + +[.programlisting] +.... +I: Retrieving InRelease +I: Checking Release signature +I: Valid Release signature (key id F6ECB3762474EDA9D21B7022871920D1991BC93C) +I: Retrieving Packages +I: Validating Packages +I: Resolving dependencies of required packages... +I: Resolving dependencies of base packages... +I: Checking component main on http://archive.ubuntu.com/ubuntu... +[...] +I: Configuring console-setup... +I: Configuring kbd... +I: Configuring ubuntu-minimal... +I: Configuring libc-bin... +I: Configuring ca-certificates... +I: Base system installed successfully. +.... + +Then set up mounts in [.filename]#/etc/fstab#. + +[TIP] +==== +If the contents of the home directory should be shared and be able to run X11 applications, [.filename]#/home# and [.filename]#/tmp# should be mounted in the linux compat area using man:nullfs[5] for loopback. + +The following example can be added to [.filename]#/etc/fstab#: [.programlisting] .... -compat.linux.emul_path="/compat/ubuntu" +# Device Mountpoint FStype Options Dump Pass# +devfs /compat/ubuntu/dev devfs rw,late 0 0 +tmpfs /compat/ubuntu/dev/shm tmpfs rw,late,size=1g,mode=1777 0 0 +fdescfs /compat/ubuntu/dev/fd fdescfs rw,late,linrdlnk 0 0 +linprocfs /compat/ubuntu/proc linprocfs rw,late 0 0 +linsysfs /compat/ubuntu/sys linsysfs rw,late 0 0 +/tmp /compat/ubuntu/tmp nullfs rw,late 0 0 +/home /compat/ubuntu/home nullfs rw,late 0 0 .... -This sysctl controls the kernel's path translation mechanism; see man:linux[4] for details. -Please note that changing it might cause trouble for Linux applications installed from FreeBSD packages; one reason is that many of those applications are still 32-bit, while Ubuntu seems to be deprecating 32-bit library support. +Then execute man:mount[8]: + +[source,shell] +.... +# mount -al +.... +==== + +To access the system using man:chroot[8] execute the following command: + +[source,shell] +.... +# chroot /compat/ubuntu /bin/bash +.... + +Then man:uname[1] can be executed to check the Linux environment: + +[source,shell] +.... +# uname -s -r -m +.... + +The output should be similar to the following: + +[.programlisting] +.... +Linux 3.17.0 x86_64 +.... + +Once inside the system, it can be worked on it as if it were a normal Ubuntu installation. +While systemd doesn't work, the man:service[8] command works as usual. + +[TIP] +==== +To add the package repositories missing from defaults edit the file [.filename]#/compat/ubuntu/etc/apt/sources.list#. + +For amd64 the following example can be used: + +[.programlisting] +.... +deb http://archive.ubuntu.com/ubuntu focal main universe restricted multiverse +deb http://security.ubuntu.com/ubuntu/ focal-security universe multiverse restricted main +deb http://archive.ubuntu.com/ubuntu focal-backports universe multiverse restricted main +deb http://archive.ubuntu.com/ubuntu focal-updates universe multiverse restricted main +.... + +For arm64 this other example can be used: + +[.programlisting] +.... +deb http://ports.ubuntu.com/ubuntu-ports bionic main universe restricted multiverse +.... +==== [[linuxemu-advanced]] == Advanced Topics -The Linux compatibility layer is a work in progress. -Consult https://wiki.freebsd.org/Linuxulator[FreeBSD Wiki - Linuxulator] for more information. - A list of all Linux-related man:sysctl[8] knobs can be found in man:linux[4]. Some applications require specific filesystems to be mounted. -This is normally handled by the [.filename]#/etc/rc.d/linux# script, but can be disabled by adding this line to [.filename]#/etc/rc.conf#: + +This is normally handled by the [.filename]#/etc/rc.d/linux# script. +But can be disabled at boot executing the following command: [.programlisting] .... -linux_mounts_enable="NO" +sysrc linux_mounts_enable="NO" .... Filesystems mounted by the rc script will not work for Linux processes inside chroots or jails; if needed, configure them in [.filename]#/etc/fstab#: + +[.programlisting] .... devfs /compat/linux/dev devfs rw,late 0 0 tmpfs /compat/linux/dev/shm tmpfs rw,late,size=1g,mode=1777 0 0 @@ -165,7 +326,7 @@ linsysfs /compat/linux/sys linsysfs rw,late 0 0 .... -Since the Linux binary compatibility layer has gained support for running both 32- and 64-bit Linux binaries (on 64-bit x86 hosts), it is no longer possible to link the emulation functionality statically into a custom kernel. +Since the Linux binary compatibility layer has gained support for running both 32- and 64-bit Linux binaries, it is no longer possible to link the emulation functionality statically into a custom kernel. [[linuxemu-libs-manually]] === Installing Additional Libraries Manually @@ -178,11 +339,18 @@ If a Linux application complains about missing shared libraries after configuring Linux binary compatibility, determine which shared libraries the Linux binary needs and install them manually. From a Linux system using the same CPU architecture, `ldd` can be used to determine which shared libraries the application needs. + For example, to check which shared libraries `linuxdoom` needs, run this command from a Linux system that has Doom installed: [source,shell] .... % ldd linuxdoom +.... + +The output should be similar to the following: + +[.programlisting] +.... libXt.so.3 (DLL Jump 3.1) => /usr/X11/lib/libXt.so.3.1.0 libX11.so.3 (DLL Jump 3.1) => /usr/X11/lib/libX11.so.3.1.0 libc.so.4 (DLL Jump 4.5pl26) => /lib/libc.so.4.6.29 @@ -190,9 +358,10 @@ Then, copy all the files in the last column of the output from the Linux system into [.filename]#/compat/linux# on the FreeBSD system. Once copied, create symbolic links to the names in the first column. + This example will result in the following files on the FreeBSD system: -[source,shell] +[.programlisting] .... /compat/linux/usr/X11/lib/libXt.so.3.1.0 /compat/linux/usr/X11/lib/libXt.so.3 -> libXt.so.3.1.0 @@ -208,7 +377,7 @@ For example, these libraries already exist on the FreeBSD system: -[source,shell] +[.programlisting] .... /compat/linux/lib/libc.so.4.6.27 /compat/linux/lib/libc.so.4 -> libc.so.4.6.27 @@ -216,7 +385,7 @@ and `ldd` indicates that a binary requires a later version: -[source,shell] +[.programlisting] .... libc.so.4 (DLL Jump 4.5pl26) -> libc.so.4.6.29 .... @@ -224,7 +393,7 @@ Since the existing library is only one or two versions out of date in the last digit, the program should still work with the slightly older version. However, it is safe to replace the existing [.filename]#libc.so# with the newer version: -[source,shell] +[.programlisting] .... /compat/linux/lib/libc.so.4.6.29 /compat/linux/lib/libc.so.4 -> libc.so.4.6.29 @@ -236,11 +405,18 @@ === Branding Linux ELF Binaries The FreeBSD kernel uses several methods to determine if the binary to be executed is a Linux one: it checks the brand in the ELF file header, looks for known ELF interpreter paths and checks ELF notes; finally, by default, unbranded ELF executables are assumed to be Linux anyway. + Should all those methods fail, an attempt to execute the binary might result in error message: [source,shell] .... % ./my-linux-elf-binary +.... + +The output should be similar to the following: + +[.programlisting] +.... ELF binary type not known Abort .... @@ -270,7 +446,7 @@ If DNS does not work or this error appears: -[source,shell] +[.programlisting] .... resolv+: "bind" is an invalid keyword resolv+: "hosts" is an invalid keyword @@ -291,51 +467,4 @@ [[linuxemu-misc]] === Miscellaneous -This section describes how Linux binary compatibility works and is based on an email written to {freebsd-chat} by Terry Lambert mailto:tlambert@primenet.com[tlambert@primenet.com] (Message ID: `<199906020108.SAA07001@usr09.primenet.com>`). - -FreeBSD has an abstraction called an "execution class loader". -This is a wedge into the man:execve[2] system call. - -Historically, the UNIX(R) loader examined the magic number (generally the first 4 or 8 bytes of the file) to see if it was a binary known to the system, and if so, invoked the binary loader. - -If it was not the binary type for the system, the man:execve[2] call returned a failure, and the shell attempted to start executing it as shell commands. -The assumption was a default of "whatever the current shell is". - -Later, a hack was made for man:sh[1] to examine the first two characters, and if they were `:\n`, it invoked the man:csh[1] shell instead. - -FreeBSD has a list of loaders, instead of a single loader, with a fallback to the `#!` loader for running shell interpreters or shell scripts. - -For the Linux ABI support, FreeBSD sees the magic number as an ELF binary. -The ELF loader looks for a specialized _brand_, which is a comment section in the ELF image, and which is not present on SVR4/Solaris(TM) ELF binaries. - -For Linux binaries to function, they must be _branded_ as type `Linux` using man:brandelf[1]: - -[source,shell] -.... -# brandelf -t Linux file -.... - -When the ELF loader sees the `Linux` brand, the loader replaces a pointer in the `proc` structure. -All system calls are indexed through this pointer. -In addition, the process is flagged for special handling of the trap vector for the signal trampoline code, and several other (minor) fix-ups that are handled by the Linux kernel module. - -The Linux system call vector contains, among other things, a list of `sysent[]` entries whose addresses reside in the kernel module. - -When a system call is called by the Linux binary, the trap code dereferences the system call function pointer off the `proc` structure, and gets the Linux, not the FreeBSD, system call entry points. - -Linux mode dynamically _reroots_ lookups. -This is, in effect, equivalent to `union` file system mounts. -First, an attempt is made to look up the file in [.filename]#/compat/linux/original-path#. -If that fails, the lookup is done in [.filename]#/original-path#. -This makes sure that binaries that require other binaries can run. -For example, the Linux toolchain can all run under Linux ABI support. -It also means that the Linux binaries can load and execute FreeBSD binaries, if there are no corresponding Linux binaries present, and that a man:uname[1] command can be placed in the [.filename]#/compat/linux# directory tree to ensure that the Linux binaries cannot tell they are not running on Linux. - -In effect, there is a Linux kernel in the FreeBSD kernel. -The various underlying functions that implement all of the services provided by the kernel are identical to both the FreeBSD system call table entries, and the Linux system call table entries: file system operations, virtual memory operations, signal delivery, and System V IPC. -The only difference is that FreeBSD binaries get the FreeBSD _glue_ functions, and Linux binaries get the Linux _glue_ functions. -The FreeBSD _glue_ functions are statically linked into the kernel, and the Linux _glue_ functions can be statically linked, or they can be accessed via a kernel module. - -Technically, this is not really emulation, it is an ABI implementation. -It is sometimes called "Linux emulation" because the implementation was done at a time when there was no other word to describe what was going on. -Saying that FreeBSD ran Linux binaries was not true, since the code was not compiled in. +More information on how binary compatibility works with Linux(R) can be found in the article link:{linux-emulation}[Linux emulation in FreeBSD].