diff --git a/documentation/content/en/books/handbook/virtualization/_index.adoc b/documentation/content/en/books/handbook/virtualization/_index.adoc --- a/documentation/content/en/books/handbook/virtualization/_index.adoc +++ b/documentation/content/en/books/handbook/virtualization/_index.adoc @@ -706,9 +706,8 @@ * When adding the default user, ensure they are a member of the *wheel* group. Once the installation completes, the virtual machine reboots into the newly installed FreeBSD image. -Several additional steps are required to configure the virtual machine: -Login as `root`. Update the system as follows: +Login as `root` and update the system as follows: [source,shell] .... @@ -845,6 +844,449 @@ * `Power Down` simulates an ACPI shutdown signal and the operating system goes through a graceful shutdown. * `Quit` powers off the virtual machine immediately - also not recommended unless necessary. +[[qemu-adding-serial-port-to-guest-vm]] +=== Adding a Serial Port Interface to a Guest VM + +To implement a serial console, a guest VM running FreeBSD needs to insert +[.programlisting] +.... +console="comconsole" +.... +in [.filename]#/boot/loader.conf# to allow the use of the FreeBSD serial console. + +The updated configuration below shows how to implement the serial console on the guest VM. +Run the script under man:sudo[8] to start the VM. +[.programlisting] +.... +# left+serial.sh +echo +echo "NOTE: telnet startup server running on guest VM!" +echo "To start QEMU, start another session and telnet to localhost port 4410" +echo + +sudo /usr/local/bin/qemu-system-x86_64 -monitor none \ + -serial telnet:localhost:4410,server=on,wait=on\ + -cpu qemu64 \ + -vga std \ + -m 4096 \ + -smp 4 \ + -cdrom ../ISO/fbsd.iso \ + -boot order=cd,menu=on \ + -blockdev driver=file,aio=threads,node-name=imgleft,filename=../VM/left.img \ + -blockdev driver=raw,node-name=drive0,file=imgleft \ + -device virtio-blk-pci,drive=drive0,bootindex=1 \ + -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \ + -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \ + -name \"left\" +.... +[[qemu-left-serial-port]] +.Enabling a Serial Port over TCP +image::qemu-freebsd07.png[] + +In Figure xref:qemu-left-serial-port[{counter:figure}], the serial port is redirected to a TCP port on the host system at VM startup and the QEMU monitor waits (`wait=on`) to activate the guest VM until a man:telnet[1] connection occurs on the indicated localhost port. +After receiving a connection from a separate session, the FreeBSD system starts booting and recognizes the console directive in [.filename]#/boot/loader.conf#. +It then starts up a serial console. +The QEMU monitor detects this and directs the necessary character I/O on that serial port to the telnet sesson on the host. +The system boots and once finished, login prompts are enabled on the serial port (`ttyu0`) and on the console (`ttyv0`). + +It is important to note that the this serial redirect over TCP takes place outside the virtual machine. +There is no interaction with any network on the virtual machine and therefore it is not subject to any firewall rules. +Think of it like a dumb terminal sitting on an RS-232 or USB port on a real machine. + +[[qemu-notes-on-serial-console]] +==== Notes on Using the Serial Console + +On the serial console, if the window is resized, execute man:resizewin[1] to update the terminal size. + +It may be desirable (even necessary) to stop syslog message from being sent to the console (both the QEMU console and the serial port). +Consult man:syslog.conf[5] for details on redirecting console messages. + +[NOTE] +==== +Once the [.filename]#/boot.loader.conf# has been updated to permit a serial console, +the guest VM will attempt to boot from the serial port every time. +Ensure that the serial port is enabled as shown in the listing above, or update the [.filename]#/boot/loader.conf# file to not require a serial console. +==== + +[[qemu-user-mode-emulation]] +=== QEMU User Mode Emulation + +QEMU also supports running applications that are precompiled on an architecture different from the host CPU. +For example, it is possible to run a Sparc64 architecture operating system on an x86_64 host. +This is demonstrated in the next section. + +[[qemu-sparc64-user-mode-emulation]] +==== Setting up a SPARC64 Guest VM on an x86_64 Host + +Setting up a new VM with an architecture different from the host involves several steps: + +* Getting the software that will run on the guest VM +* Creating a new disk image for the guest VM +* Setting up a new QEMU script with the new architecture +* Performing the install + +In the following procedure a copy of OpenBSD 6.8 SPARC64 sofware is used for this QEMU User Mode Emulation exercise. + +[NOTE] +==== +Not all versions of OpenBSD Sparc64 work on QEMU. +OpenBSD version 6.8 is known to work and was selected as the example for this section. +==== + +. Download OpenBSD 6.8 Sparc64 from an OpenBSD archive. ++ +On the OpenBSD download sites, only the most current versions are maintained. +It is necessary to search an archive to obtain past releases. ++ +[source,shell] +.... +% cd ~/QEMU/ISO +% fetch https://mirror.planetunix.net/pub/OpenBSD-archive/6.8/sparc64/install68.iso +.... + +. Creating a new disk image for the Sparc64 VM is similar to the "right" VM above. +This case uses the QEMU qcow2 format for the disk: ++ +[source,shell] +.... +% cd ~/QEMU/VM +qemu-img create -f qcow2 -o preallocation=full,lazy_refcounts=on sparc64.qcow2 16G +.... + +. Use the script below for the new Sparc64 architecture. +As with above example, run the script, then start a new session and `telnet` to locahost on the port indicated: ++ +[.programlisting] +.... +echo +echo "NOTE: telnet startup server running on guest VM!" +echo "To start QEMU, start another session and telnet to localhost port 4410" +echo + +sudo /usr/local/bin/qemu-system-sparc64 \ + -serial telnet:localhost:4410,server=on,wait=on \ + -machine sun4u,usb=off \ + -smp 1,sockets=1,cores=1,threads=1 \ + -rtc base=utc \ + -m 1024 \ + -boot d \ + -drive file=../VM/sparc64.qcow2,if=none,id=drive-ide0-0-1,format=qcow2,cache=none \ + -cdrom ../ISO/install68.iso \ + -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-1,id=ide0-0-1 \ + -msg timestamp=on \ + -net nic,model=sunhme -net user \ + -nographic \ + -name \"sparc64\" +.... + +Note the following: + +* The `-boot d` option boots from the QEMU CDROM device which is set as `-cdrom ../ISO/install68.iso`. +* As before, the `telnet` server option is set to wait for a separate connection on port 4410. +Start up another session and use man:telnet[1] to connect to localhost on port 4410. +* The script sets the `-nographic` option meaning there is only serial port I/O. +There is no graphical interface. +* Networking is not set up through the man:tap[4] / man:if_bridge[4] combination. +This example uses a separate method of QEMU networking known as "Serial Line Internet Protocol" (SLIRP), sometimes referred to as "User Mode Networking". +Documentation on this and other QEMU networking methods is here: +link:https://wiki.qemu.org/Documentation/Networking[QEMU Networking Documentation] + +If everything is set correctly, the system will boot as shown in Figure xref:qemu-sparc64-boot-cdrom-installation[{counter:figure}]. +[[qemu-sparc64-boot-cdrom-installation]] +.QEMU Booting OpenBSD 6.8 Sparc64 from CDROM During User Mode Emualation +image::qemu-freebsd08.png[] + +Once the system is installed, modify the script and change the boot parameter to `-boot c`. +This will indicate to QEMU to boot from the supplied hard disk, not the CDROM. + +The installed system can be used just like any other guest virtual machine. +However, the underlying architecture of the guest is Sparc64, not x86_64. + +[TIP] +==== +If you have halted the Sparc64 system and find yourself at the OpenBios console prompt `0 >`, enter `power-off` to exit the system. +==== +Figure xref:qemu-sparc64-login-to-installed-system[{counter:figure}] shows a root login to the installed system and running man:uname[1]. + +[[qemu-sparc64-login-to-installed-system]] +.QEMU Booting from CDROM During User Mode Emualation +image::qemu-freebsd09.png[] + +[[qemu-using-qemu-monitor]] +=== Using the QEMU Monitor + +The link:https://www.qemu.org/docs/master/system/monitor.html[QEMU monitor] controls a running QEMU emulator (guest VM). + +Using the monitor, it is possible to: + +* Dynamically remove or insert devices, including disks, network interfaces, CD-ROMs, or floppies +* Freeze/unfreeze the guest VM, and save or restore its state from a disk file +* Gather information about the state of the VM and devices +* Change device settings on the fly + +and many other operations. +The most common uses of the monitor are to examine the state of the VM, and to add, delete, or change devices. +Some operations such as migrations are only available under hypervisor accelerators such as KVM, Xen, etc. and are not supported on FreeBSD hosts. + +When using a graphical desktop environment, the simplest way to use the QEMU monitor is the `-monitor stdio` option when launching QEMU from a terminal session. + +[.programlisting] +.... +sudo /usr/local/bin/qemu-system-x86_64 -monitor stdio \ + -cpu qemu64 \ + -vga cirrus \ + -m 4096 -smp 4 \ + ... +.... + +This results in a new prompt `(qemu)` in the terminal window as shown in Figure xref:qemu-monitor-operation[{counter:figure}]. + +[[qemu-monitor-operation]] +.QEMU Monitor Prompt and "stop" Command +image::qemu-freebsd13.png[] + +The image also shows the `stop` command freezing the system during the FreeBSD boot sequence. +The system will remain frozen until the `cont` command is entered in the monitor. + +[[qemu-adding-new-disk]] +==== Adding a New Disk to the VM + +To add a new disk to a running VM, the disk needs to be prepared as above: + +[source, shell] +.... +% cd ~/QEMU/VM +% qemu-img create -f raw new10G.img 10G +.... + +Figure xref:qemu-add-new-disk-figure[{counter:figure}] shows the monitor command sequence needed to add a new disk in the VM. +Once the device has been added with the `device_add` command in the monitor it shows up on the FreeBSD system console shown in the lower part of the figure. +The disk can be configured as needed. + +Note that the new disk must be added to the startup script if it is to be used after a VM reboot. + +[[qemu-add-new-disk-figure]] +.QEMU Monitor Commands to Add a New Disk +image::qemu-freebsd14.png[] + +[[qemu-using-monitor-manage-snapshots]] +==== Using the QEMU Monitor to Manage Snapshots + +QEMU's documentation describes several similar concepts when using the term *snapshot*. +There is the `-snapshot` option on the command line which refers to using a drive or portion of a drive to contain a copy of a device. +Then there are the monitor commands `snapshot_blkdev` and `snapshot_blkdev_internal` which describe the actual act of copying the blockdev device. +Finally, there are the monitor commands `savevm`, `loadvm`, and `delvm` commands which refer to creating and saving, loading, or deleting a copy of an entire virtual machine. +Along with the latter, the monitor `info snapshots` command lists out details of recent snapshots. + +This section will focus on creating, saving, and loading a complete VM image and will use the term *snapshot* for this purpose. + +To start, recreate the "left" VM from scratch, this time using the `qcow2` format. + +[source, shell] +.... +% cd ~/QEMU/VM +% rm left.img +% qemu-img create -f qcow2 left.qcow2 16G # Clean file for a new FreeBSD installation. +% cd ../SCRIPTS +% sudo /bin/sh left.sh # See the below program listing. +.... + +Once the installation is complete, reboot, this time using the `-monitor stdio` option to allow use of the monitor. + +[.programlisting] +.... +# left VM script. +sudo /usr/local/bin/qemu-system-x86_64 -monitor stdio \ + -cpu qemu64 \ + -vga std \ + -m 4096 \ + -smp 4 \ + -cdrom ../ISO/fbsd.iso \ + -boot order=cd,menu=on \ + -blockdev driver=file,aio=threads,node-name=imgleft,filename=../VM/left.qcow2 \ + -blockdev driver=qcow2,node-name=drive0,file=imgleft \ + -device virtio-blk-pci,drive=drive0,bootindex=1 \ + -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \ + -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \ + -name \"left\" +.... + +To demonstrate snapshots, the following procedure can be used: + +. Install FreeBSD from scratch +. Prepare the environment and take a snapshot with the `savevm` monitor command +. Install several packages +. Shut down the system +. Restart a bare QEMU instance and utilize the monitor command `loadvm` to restore the VM +. Observe that the restored VM does not have any packages + +During the "Prepare the environment" step, in a separate virtual console (ttyv1), an editing session with man:vi[1] is initiated simulating user activity. +Additional programs may be be started if desired. +The snapshot should account for the state of all applications running at the time the snapshot is taken. + +Figure xref:qemu-using-monitor-snapshots[{counter:figure}] shows the newly installed FreeBSD system with no packages, and separately, the editing sesssion on ttyv1. +The man:vi[1] editor is currently in `insert` mode with the typist typing the word "broadcast". + +[[qemu-using-monitor-snapshots]] +.QEMU VM Before First Snapshot +image::qemu-freebsd15.png[] + +To generate the snapshot, enter `savevm` in the monitor. +Be sure to give it a tag (such as `original_install`). + +[source,shell] +.... +QEMU 9.0.1 monitor - type 'help' for more information +(qemu) +(qemu) savevm original_install +.... + +Next, in the main console window, install a package, such as man:zip[1] which has no dependencies. +Once that completes, renter the monitor and create another snapshot (`snap1_pkg+zip`). + +Figure xref:qemu-after-monitor-snapshots[{counter:figure}] shows the results of the above commands and the output of the `info shapshots` command. + +[[qemu-after-monitor-snapshots]] +.QEMU Using Monitor Commands for Snapshots +image::qemu-freebsd16.png[] + +Reboot the system, and before FreeBSD starts up, switch to the monitor and enter `stop`. +The VM will stop. + +Enter `loadvm` with the tag you used above (here `original_install`). + +[source, shell] +.... +QEMU 9.0.1 monitor - type 'help' for more information +(qemu) stop +(qemu) loadvm original_install +(qemu) cont +.... + +Immediately, the VM screen will switch to the exact moment the `savevm` command was entered above. +Note that the VM is still stopped. + +Enter `cont` to start the VM, switch to the editing session on `ttyv1`, and type one letter on the keyboard. +The editor, still in insert mode, should respond accordingly. +Any other programs running at the time the snapshot was taken should be unaffected. + +The above steps show how a snapshot can be taken, the system modified, and then "rolled back" by restoring the previous snapshot. + +By default QEMU stores snapshot data in the same file as the image. +You can view the list of snapshots with man:qemu-img[1] as shown below in Figure xref:qemu-examine-monitor-snapshots[{counter:figure}]. + +[[qemu-examine-monitor-snapshots]] +.QEMU Using man:qemu-img[1] to Examine Snapshots +image::qemu-freebsd17.png[] + +[[qemu-using-qemu-usb-devices]] +=== Using QEMU USB Devices + +QEMU supports the creation of virtual USB devices that are backed by an image file. +These are virtual USB devices that can be partitioned, formatted, mounted, and used just like a real USB device. + +[.programlisting] +.... +sudo /usr/local/bin/qemu-system-x86_64 -monitor stdio \ + -cpu qemu64 \ + -vga cirrus \ + -m 4096 -smp 4 \ + -cdrom ../ISO/fbsd.iso \ + -boot order=cd,menu=on \ + -drive if=none,id=usbstick,format=raw,file=../VM/foo.img \ + -usb \ + -device usb-ehci,id=ehci \ + -device usb-storage,bus=ehci.0,drive=usbstick \ + -device usb-mouse \ + -blockdev driver=file,node-name=img1,filename=../VM/right.qcow2 \ + -blockdev driver=qcow2,node-name=drive0,file=img1 \ + -device virtio-blk-pci,drive=drive0,bootindex=1 \ + -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,br=bridge0 \ + -device e1000,netdev=nd0,mac=02:72:69:67:68:74 \ + -name \"right\" +.... + +This configuration includes a `-drive` specification with the `id=usbstick`, raw format, and an image file (must be created by man:qemu-img[1]). +The next line contains the `-device usb-ehci` specification for a USB EHCI controller, with `id=ehci`. +Finally, a `-device usb-storage` specification ties the above drive to the EHCI USB bus. + +When the system is booted, FreeBSD will recognize a USB hub, add the attached USB device, and assign it to `da0` as shown in Figure xref:qemu-usb-internal-storage[{counter:figure}]. + +[[qemu-usb-internal-storage]] +.QEMU Created USB Hub and Mass Storage Device +image::qemu-freebsd12.png[] + +The device is ready to be partitioned with man:gpart[8], and formatted with man:newfs[8]. +Because the USB device is backed by a man:qemu-img[1] created file, data written to the device will persist across reboots. + +[[qemu-using-host-usb-devices]] +=== Using Host USB Devices via Passthrough + +QEMU USB passthrough support is listed as experimental in version 9.0.1 (Summer, 2024). +However, the following steps show how a USB stick mounted on the host can be used by the guest VM. + +For more information and examples, see: + +* link:https://www.qemu.org/docs/master/system/devices/usb.html[] + +The upper part of Figure xref:qemu-usb-passthrough[{counter:figure}] shows the the QEMU monitor commands: + +* `info usbhost` shows information about all USB devices on the host system. +Find the desired USB device on the host system and note the two hexadecimal values on that line. +(In the example below the host USB device is a Memorex Mini, with vendorid 0718, and productid 0619.) +Use the two values shown by the `info usbhost` command in the `device_add` step below. +* `device_add` adds a USB device to the guest VM. + +[[qemu-usb-passthrough]] +.QEMU Monitor Commands to Access a USB Device on the Host +image::qemu-freebsd18.png[] + +As before, once `device_add` completes, the FreeBSD kernel recognizes a new USB device, as shown in the lower half of the Figure. + +Using the new device is shown in Figure xref:qemu-usb-passthrough2[{counter:figure}]. + +[[qemu-usb-passthrough2]] +.Using the Host USB Device via Passthrough +image::qemu-freebsd19.png[] + +If the USB device is formatted as a FAT16 or FAT32 filesystem it can be mounted as an MS-DOS(TM) filesystem with man:mount_msdosfs[8] as in the example shown. +The `/etc/hosts` file is copied to the newly mounted drive and checksums are taken to verify the integrity of the file on the USB device. +The device is then unmounted with man:umount[8]. + +If the USB device is formatted with NTFS it is necessary to install the `fusefs-ntfs` package and use man:ntfs-3g[8] to access the device: + +[source, shell] +.... +# pkg install fusefs-ntfs +# kldload fusefs +# gpart show da1 +# ntfs-3g /dev/da1s1 /mnt + +Access the drive as needed. When finished: + +# umount /mnt +.... + +Change the above device identifiers to match the installed hardware. +Consult man:ntfs-3g[8] for additional information on working with NTFS filesystems. + +[[qemu-summary]] +=== QEMU on FreeBSD Summary + +As noted above, QEMU works with several different hypervisor accelerators. +All the examples use in this section used the `Tiny Code Generator (tcg)` accelerator as that is the only supported accelerator on FreeBSD at present. + +The list of supported +link:https://www.qemu.org/docs/master/system/introduction.html#virtualisation-accelerators[Virtualization Accelerators] +includes: + +* `KVM` on Linux supporting 64 bit Arm, MIPS, PPC, RISC-V, s390x, and x86 +* `Xen` on Linux as dom0 supporting Arm, x86 +* `Hypervisor Framework (hvf)` on MacOS supporting x86 and Arm (both 64 bit only) +* `Windows Hypervisor Platform (whpx)` on Windows supporting x86 +* `NetBSD Virutal Machine Monitor (nvmm)` on NetBSD supporting x86 +* `Tiny Code Generator (tcg)` on Linux and other POSIX, Windows, MacOS supporting Arm, x86, Loongarch64, MIPS, PPC, s390x, and Sparc64. + [[virtualization-host-bhyve]] == FreeBSD as a Host with bhyve