Paths

Table of Contentst

-
sys/
-
net/
32/32
route.h
32/32
route.c
-
route/
32/32
route_ctl.c
-
netinet/
32/32
ip_input.c
-
netinet6/
-
ip6_input.c

Flush remaining routes from the routing table during VNET shutdown.
ClosedPublic
Actions

Authored by melifaro on Mar 7 2021, 6:38 PM.

Details

Reviewers

kp
bz

Group Reviewers

network

Commits

rG6f4f8a441aaa: Flush remaining routes from the routing table during VNET shutdown.
rG8aafa7a02763: Flush remaining routes from the routing table during VNET shutdown.
rGb1d63265ac39: Flush remaining routes from the routing table during VNET shutdown.

Summary

This fixes rtentry leak for the cloned interfaces created inside the
VNET.

PR: 253998
Reported by: rashey at superbox.pl
MFC after: 3 days

Current code cleans up protocol state in ip[6]_destroy(). It includes flushing route entries on a per-interface basis using rt_flushifroutes_af(). The latter function removes all interface routes except ones with RTF_PINNED flag (e.g. interface routes).
As a result, some of the rtentries may leak.

Add rib_flush_routes_af() which removes all of the routes for the specified family in a light-weight fashion and switch IPv4/IPv6 stack teardown to use it instead of iterating over all interfaces with rt_flushroutes_af().

Test Plan

# kyua test -k /usr/tests/sys/netpfil/pf/Kyuafile set_skip:set_skip_group_lo
set_skip:set_skip_group_lo  ->  passed  [0.053s]
tail -n 200 /var/log/messages | grep rtentry
#

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 37685
Build 34574: arc lint + arc unit

Event Timeline

melifaro created this revision.Mar 7 2021, 6:38 PM

Herald added subscribers: ae, imp. · View Herald TranscriptMar 7 2021, 6:38 PM

melifaro requested review of this revision.Mar 7 2021, 6:38 PM

Harbormaster completed remote builds in B37658: Diff 85301.Mar 7 2021, 6:38 PM

melifaro edited the summary of this revision. (Show Details)Mar 7 2021, 6:51 PM

melifaro edited the test plan for this revision. (Show Details)

kp accepted this revision as: kp.Mar 8 2021, 10:37 AM

This revision is now accepted and ready to land.Mar 8 2021, 10:37 AM

Implement a cleaner fix.

This revision now requires review to proceed.Mar 8 2021, 11:21 PM

Harbormaster completed remote builds in B37685: Diff 85362.Mar 8 2021, 11:21 PM

melifaro edited the summary of this revision. (Show Details)Mar 9 2021, 8:07 AM

kp accepted this revision as: kp.Mar 9 2021, 10:51 AM

This revision is now accepted and ready to land.Mar 9 2021, 10:51 AM

Closed by commit rGb1d63265ac39: Flush remaining routes from the routing table during VNET shutdown. (authored by melifaro). · Explain WhyMar 10 2021, 9:29 PM

This revision was automatically updated to reflect the committed changes.

melifaro added a commit: rGb1d63265ac39: Flush remaining routes from the routing table during VNET shutdown..

melifaro added a commit: rG8aafa7a02763: Flush remaining routes from the routing table during VNET shutdown..Mar 13 2021, 8:26 PM

melifaro added a commit: rG6f4f8a441aaa: Flush remaining routes from the routing table during VNET shutdown..Mar 28 2021, 8:50 PM

Revision Contents
Changeset List

Path

Size

sys/

net/

route.h

2 lines

route.c

15 lines

route/

route_ctl.c

36 lines

netinet/

ip_input.c

6 lines

netinet6/

ip6_input.c

5 lines

Diff 85362

View Options

sys/net/route.h

	Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Lines
	....			....
	# service devfs restart			# service devfs restart
	....			....

	[[virtualization-host-bhyve]]			[[virtualization-host-bhyve]]
	== FreeBSD as a Host with bhyve			== FreeBSD as a Host with bhyve

	The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.			The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.
	This hypervisor supports a number of guests, including FreeBSD, OpenBSD, and many Linux(R) distributions.			This hypervisor supports a number of guests, including FreeBSD, OpenBSD, many Linux(R) distributions, and Microsoft Windows(R).
				jrmUnsubmitted Done Inline Actions Should we use the Oxford comma like in the original? jrm: Should we use the Oxford comma like in the original?
				emasteUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. emaste: I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
				rgrimesUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. I had the same feeling when I read this change before seeing Ed's comment rgrimes: > I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
	By default, bhyve provides access to serial console and does not emulate a graphical console.			By default, bhyve provides access to serial console and does not emulate a graphical console.
	Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.			Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.

	The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).			The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).
	Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).			Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).
	Most newer processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.			Most processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.
				emasteUnsubmitted Done Inline Actions Processors with the necessary support have been available for years so we no longer need "newer." In a subsequent commit we may want to change this to list the one or two families of CPUs that lack support. emaste: Processors with the necessary support have been available for years so we no longer need "newer.
				rgrimesUnsubmitted Done Inline Actions Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point rgrimes: Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point
				emasteUnsubmitted Done Inline Actions I think Nehalem is old enough that it's not going to be clear to users today where that lands. Maybe something like "Most processors made in 2008 or later?" (or whichever year) Or "Intel processors made in 2010 and later support these features" (picking up UG). emaste: I think Nehalem is old enough that it's not going to be clear to users today where that lands.
				rgrimesUnsubmitted Done Inline Actions Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page, and as a member of the list of Intel CPU microarchitectures. The reduction to "most processors" is wrong, that would imply arm, risc-v, or even an at-meta has support for bhyve. rgrimes: Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point. I too was feeling this is overly specific. I'll attempt to condense this further. freebsd_ny-central.org: Good point. I too was feeling this is overly specific. I'll attempt to condense this further.
				emasteUnsubmitted Done Inline Actions Mentioning together "newer processors" and a processor family that is about a decade and a half old doesn't make any sense. To a first order approximation bhyve can be used on any x86_64 processor in use today. Sure, there are some old machines around still and there are retrocomputing enthusiasts, but for the majority of readers this isn't useful information. We also don't give any information about which AMD families are supported. I would just take this whole sentence out. emaste: Mentioning together "newer processors" and a processor family that is about a decade and a half…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Fair point. I too was wondering about AMD, but since I couldn't find any readily available listing of processors with RVI, I left it as is. I guess it's a reasonable compromise to just take it out completely, because we have a paragraph on how to check whether the processor supports bhyve. I'll quickly remove the sentence and update the diff so we can get this landed. freebsd_ny-central.org: Fair point. I too was wondering about AMD, but since I couldn't find any readily available…
	UG support was introduced with Intel's Westmere micro-architecture.			UG support was introduced with Intel's Westmere micro-architecture.
	For a complete list of Intel(R) processors that support EPT, refer to https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[].			For a complete list of Intel(R) processors that support EPT, refer to link:https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[Intel's feature documentation].
	RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.			RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.

	The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.			The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.

	[[virtualization-bhyve-prep]]			[[virtualization-bhyve-prep]]
	=== Preparing the Host			=== Preparing the Host

	The first step to creating a virtual machine in bhyve is configuring the host system.			The first step to creating a virtual machine in bhyve is configuring the host system.
	First, load the bhyve kernel module:			First, load the bhyve kernel module:

	[source,shell]			[source,shell]
	....			....
	# kldload vmm			# kldload vmm
	....			....

	Then, create a [.filename]#tap# interface for the network device in the virtual machine to attach to.			There are several ways to connect a virtual machine guest to a host's network; one straightforward way to accomplish this is to create a [.filename]#tap# interface for the network device in the virtual machine to attach to.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we don't need the comma after `this`. jrm: Grammarly tells me we don't need the comma after `this`.
	In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.			In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.
	In this example, the physical interface is _igb0_:			In this example, the physical interface is _igb0_:

	[source,shell]			[source,shell]
	....			....
	# ifconfig tap0 create			# ifconfig tap0 create
	# sysctl net.link.tap.up_on_open=1			# sysctl net.link.tap.up_on_open=1
	net.link.tap.up_on_open: 0 -> 1			net.link.tap.up_on_open: 0 -> 1
	Show All 12 Lines
	....			....
	# truncate -s 16G guest.img			# truncate -s 16G guest.img
	....			....

	Download an installation image of FreeBSD to install:			Download an installation image of FreeBSD to install:

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
	FreeBSD-13.1-RELEASE-amd64-bootonly.iso 366 MB 16 MBps 22s			FreeBSD-14.0-RELEASE-amd64-bootonly.iso 426 MB 16 MBps 22s
	....			....

	FreeBSD comes with an example script for running a virtual machine in bhyve.			FreeBSD comes with an example script `vmrun.sh` for running a virtual machine in bhyve.
				jrmUnsubmitted Done Inline Actions I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with an example script, vmrun.sh, for running a virtual machine in bhyve. Maybe it was intentional to not refer to the script here because if the name ever changes, it would require more updates. I still think it sounds awkward to only refer to it as `The script`. jrm: I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with…
				rgrimesUnsubmitted Done Inline Actions vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major releases I would say that stop gap status has sailed and directly mentioning it here would be good. rgrimes: vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major…
	The script will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.			It will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.
	The script takes a number of options to control the configuration of the machine: `-c` controls the number of virtual CPUs, `-m` limits the amount of memory available to the guest, `-t` defines which [.filename]#tap# device to use, `-d` indicates which disk image to use, `-i` tells bhyve to boot from the CD image instead of the disk, and `-I` defines which CD image to use.			`vmrun.sh` takes a number of options to control the configuration of the machine, including:
				jrmUnsubmitted Done Inline Actions I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we could phrase it something like this. It takes a number of options to control the configuration of the machine, including: jrm: I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we…
				rgrimesUnsubmitted Done Inline Actions Dont document vmrun.sh here, simply link to the man page. rgrimes: Dont document vmrun.sh here, simply link to the man page.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2-RELEASE. Might be worthwhile to create a man page for this in a separate review, I suppose. freebsd_ny-central.org: There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2…
				rgrimesUnsubmitted Done Inline Actions Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even partially and NOT providing a method to get the full syntax avaliable. Also the documenting here is likely to bit rot rather quickly rgrimes: Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions You're certainly right with the potential for bitrot. For now, I just changed the formatting of the previously listed options. I'll add some info about --usage to allow users to get a full list of available options. freebsd_ny-central.org: You're certainly right with the potential for bitrot. For now, I just changed the formatting of…
	The last parameter is the name of the virtual machine, used to track the running machines.
				* `-c` controls the number of virtual CPUs,
				* `-m` limits the amount of memory available to the guest,
				* `-t` defines which [.filename]#tap# device to use,
				* `-d` indicates which disk image to use,
				* `-i` tells bhyve to boot from the CD image instead of the disk, and
				* `-I` defines which CD image to use.

				The last parameter is the name of the virtual machine and used to track the running machines.
	This example starts the virtual machine in installation mode:			This example starts the virtual machine in installation mode:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-13.1-RELEASE-amd64-bootonly.iso guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-14.0-RELEASE-amd64-bootonly.iso guestname
				rgrimesUnsubmitted Done Inline Actions Is the "sh" even needed, this file should already be installed mode 755 so can be directly invoked. If its not installed 755 that should be fixed. rgrimes: Is the "sh" even needed, this file should already be installed mode 755 so can be directly…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE at the moment, at least. I agree that it would be good to change that. I'll have a look at that for another review. freebsd_ny-central.org: Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE…
	....			....

	The virtual machine will boot and start the installer.			The virtual machine will boot and start the installer.
	After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].			After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].

	Reboot the virtual machine.			Reboot the virtual machine.
	While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.			While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.
	When this happens, choose the reboot option from the boot loader menu in order to escape the loop.			When this happens, choose the reboot option from the boot loader menu in order to escape the loop.
	Now the guest can be started from the virtual disk:			Now the guest can be started from the virtual disk:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname
	....			....

	[[virtualization-bhyve-linux]]			[[virtualization-bhyve-linux]]
	=== Creating a Linux(R) Guest			=== Creating a Linux(R) Guest

	In order to boot operating systems other than FreeBSD, the package:sysutils/grub2-bhyve[] port must be first installed.			Linux guests can be booted either like any other regular crossref:virtualization[virtualization-bhyve-uefi,"UEFI-based guest"] virtual machine, or alternatively, you can make use of the package:sysutils/grub2-bhyve[] port.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we also need a comma after `alternatively`. jrm: Grammarly tells me we also need a comma after `alternatively`.
				emasteUnsubmitted Done Inline Actions We don't need the (R) after the first use of a trademark. I know the existing text has it everywhere but no need to add more of them. emaste: We don't need the (R) after the first use of a trademark. I know the existing text has it…

	Next, create a file to use as the virtual disk for the guest machine:			To do this, first ensure that the port is installed, then create a file to use as the virtual disk for the guest machine:

	[source,shell]			[source,shell]
	....			....
	# truncate -s 16G linux.img			# truncate -s 16G linux.img
	....			....

	Starting a virtual machine with bhyve is a two step process.			Starting a Linux virtual machine with `grub2-bhyve` is a two step process.
	First a kernel must be loaded, then the guest can be started.
	The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].			. First a kernel must be loaded, then the guest can be started.
				jrmUnsubmitted Done Inline Actions I don't think this renders as you intended. Did you mean to make this an unordered list? jrm: I don't think this renders as you intended. Did you mean to make this an unordered list?
				. The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].

	Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:			Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:

	[.programlisting]			[.programlisting]
	....			....
	(hd0) ./linux.img			(hd0) ./linux.img
	(cd0) ./somelinux.iso			(cd0) ./somelinux.iso
	....			....

	▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=linuxguest			# bhyvectl --destroy --vm=linuxguest
	....			....

	[[virtualization-bhyve-uefi]]			[[virtualization-bhyve-uefi]]
	=== Booting bhyve Virtual Machines with UEFI Firmware			=== Booting bhyve Virtual Machines with UEFI Firmware

				rgrimesUnsubmitted Done Inline Actions strike userspace, its nonsensical in the context of firmware. rgrimes: strike userspace, its nonsensical in the context of firmware.
	In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.			In addition to `bhyveload` and `grub-bhyve`, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.
	This option may support guest operating systems that are not supported by the other loaders.			This option may support guest operating systems that are not supported by the other loaders.

	In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.			In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.
	This can be done by installing package:sysutils/bhyve-firmware[] port or package.			This can be done by installing package:sysutils/bhyve-firmware[] port or package.

	With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.			With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.
	The actual bhyve command may look like this:			The actual bhyve command may look like this:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
	guest			guest
	....			....

				rgrimesUnsubmitted Done Inline Actions We should be consistent on EFI vs UEFI, use one or the other in all places please rgrimes: We should be consistent on EFI vs UEFI, use one or the other in all places please
				emasteUnsubmitted Done Inline Actions Heh, from efivar(8) efivar – UEFI environment variable interaction emaste: Heh, from efivar(8) efivar – UEFI environment variable interaction
				To allow a guest to store EFI variables, you can use a variables file appended to the `-l` flag. Note that bhyve will write guest modifications to the given variables file.
				Therefore, be sure to first create a per-guest-copy of the variables template file:

				[source,shell]
				....
				# cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /path/to/vm-image/BHYVE_UEFI_VARS.fd
				....

				Then, add that variables file into your bhyve arguments:

				[source,shell]
				....
				# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
				-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
				-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
				-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/path/to/vm-image/BHYVE_UEFI_VARS.fd \
				guest
				....

	package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:			package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
	Show All 39 Lines
	When starting the VM, specify the ZFS volume as the disk drive:			When starting the VM, specify the ZFS volume as the disk drive:

	[source,shell]			[source,shell]
	....			....
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \
	-l com1,stdio -c 4 -m 1024M linuxguest			-l com1,stdio -c 4 -m 1024M linuxguest
	....			....

				If you are using ZFS for the host as well as inside a guest, keep in mind the competing memory pressure of both systems caching the virtual machine's contents.
				jrmUnsubmitted Done Inline Actions Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache. jrm: Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache.
				rgrimesUnsubmitted Done Inline Actions Also recommend the use of zvol's vs a file in a file system for the disk image. rgrimes: Also recommend the use of zvol's vs a file in a file system for the disk image.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point - there is a statement to this a few lines up; in my previous diff, this wasn't visible because I missed the -U99999 initially on my patch. Apologies for that. freebsd_ny-central.org: Good point - there is a statement to this a few lines up; in my previous diff, this wasn't…
				To alleviate this, consider setting the host's ZFS filesystems to use metadata-only cache.
				To do this, apply the following settings to ZFS filesystems on the host, replacing `<name>` with the name of the specific zvol dataset name of the virtual machine.

				[source,shell]
				....
				# zfs set primarycache=metadata <name>
				....

	[[virtualization-bhyve-nmdm]]			[[virtualization-bhyve-nmdm]]
	=== Virtual Machine Consoles			=== Virtual Machine Consoles

	It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.			It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.
	It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.			It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.
	To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.			To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.
	The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).			The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).
	See man:nmdm[4] for more information.			See man:nmdm[4] for more information.

	[source,shell]			[source,shell]
	....			....
	# kldload nmdm			# kldload nmdm
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \
	-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest			-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest
	# cu -l /dev/nmdm0B			# cu -l /dev/nmdm0B
	Connected			Connected

	Ubuntu 13.10 handbook ttyS0			Ubuntu 13.10 handbook ttyS0

	handbook login:			handbook login:
	....			....

				rgrimesUnsubmitted Done Inline Actions I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not follow a newline you wont get the disconnect, or at least that is how it works every where else. rgrimes: I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not…
				rgrimesUnsubmitted Done Inline Actions I do not see that this was incorporated??? rgrimes: I do not see that this was incorporated???
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a dot list syntax mistake I made previously. I'll fix this right now and update the diff one more time. freebsd_ny-central.org: Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a…
				To disconnect from a console, press `~` followed by Ctrl+D or `.`. Keep in mind that only the connection is dropped while the login session remains active.
				Another user connecting to the same console could therefore make use of any active sessions without having to first authenticate.
				For security reasons, it's therefore recommended to logout before disconnecting.

				The number in the [.filename]#nmdm# device path must be unique for each virtual machine and must not be used by any other processes before bhyve starts.
				The number can be chosen arbitrarily and does not need to be taken from a consecutive sequence of numbers.
				The device node pair (i.e. [.filename]#/dev/nmdm0a# and [.filename]#/dev/nmdm0b#) are created dynamically when bhyve connects its console and destroyed when it shuts down.
				Keep this in mind when creating scripts to start your virtual machines: you need to make sure that all virtual machines are assigned unique [.filename]#nmdm# devices.

	[[virtualization-bhyve-managing]]			[[virtualization-bhyve-managing]]
	=== Managing Virtual Machines			=== Managing Virtual Machines

	A device node is created in [.filename]#/dev/vmm# for each virtual machine.			A device node is created in [.filename]#/dev/vmm# for each virtual machine.
	This allows the administrator to easily see a list of the running virtual machines:			This allows the administrator to easily see a list of the running virtual machines:

	[source,shell]			[source,shell]
	....			....
	# ls -al /dev/vmm			# ls -al /dev/vmm
	total 1			total 1
	dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./			dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./
	dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../			dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../
	crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname			crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname
	crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest			crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest
	crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest			crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest
	....			....

	A specified virtual machine can be destroyed using `bhyvectl`:			A specified virtual machine can be destroyed using `bhyvectl`:

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=guestname			# bhyvectl --destroy --vm=guestname
	....			....

				Destroying a virtual machine this way means killing it immediately. Any unsaved data will be lost, open files and filesystems may get corrupted.
				To gracefully shut down a virtual machine, you can instead send a `TERM` signal to its bhyve process. This triggers an ACPI shutdown event for the guest:

				[source,shell]
				....
				# ps ax \| grep bhyve
				17424 - SC 56:48.27 bhyve: guestvm (bhyve)
				# kill 17424
				....

				[[virtualization-tools-utilities]]
				=== Tools and Utilities

				There are numerous utilities and applications available in ports to help simplify setting up and managing bhyve virtual machines:

				.bhyve Managers
				[options="header", cols="1,1,1,1"]
				\|===
				\| Name \| License \| Package \| Documentation

				\| vm-bhyve
				\| BSD-2
				\| package:sysutils/vm-bhyve[]
				\| link:https://github.com/churchers/vm-bhyve[Documentation]

				\| CBSD
				\| BSD-2
				\| package:sysutils/cbsd[]
				\| link:https://www.bsdstore.ru[Documentation]

				\| Virt-Manager
				\| LGPL-3
				\| package:deskutils/virt-manager[]
				\| link:https://virt-manager.org/[Documentation]

				\| Bhyve RC Script
				\| Unknown
				\| package:sysutils/bhyve-rc[]
				\| link:https://www.freshports.org/sysutils/bhyve-rc/[Documentation]

				\| vmstated
				\| BSD-2
				\| package:sysutils/vmstated[]
				\| link:https://github.com/christian-moerz/vmstated[Documentation]

				\|===

	[[virtualization-bhyve-onboot]]			[[virtualization-bhyve-onboot]]
	=== Persistent Configuration			=== Persistent Configuration

	In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:			In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:

	[.procedure]			[.procedure]
	. [.filename]#/etc/sysctl.conf#			. [.filename]#/etc/sysctl.conf#
	+			+
	▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
	Unprivileged domains consist of a configuration file and virtual or physical hard disks.			Unprivileged domains consist of a configuration file and virtual or physical hard disks.
	Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].			Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].
	In this example, a 20 GB volume is used.			In this example, a 20 GB volume is used.
	A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.			A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.
	The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.			The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso -o freebsd.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso -o freebsd.iso
	....			....

	A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.			A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.

	[source,shell]			[source,shell]
	....			....
	# zfs create -V20G -o volmode=dev zroot/xendisk0			# zfs create -V20G -o volmode=dev zroot/xendisk0
	....			....
	▲ Show 20 Lines • Show All 133 Lines • Show Last 20 Lines

View Options

sys/net/route.c

	Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Lines
	....			....
	# service devfs restart			# service devfs restart
	....			....

	[[virtualization-host-bhyve]]			[[virtualization-host-bhyve]]
	== FreeBSD as a Host with bhyve			== FreeBSD as a Host with bhyve

	The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.			The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.
	This hypervisor supports a number of guests, including FreeBSD, OpenBSD, and many Linux(R) distributions.			This hypervisor supports a number of guests, including FreeBSD, OpenBSD, many Linux(R) distributions, and Microsoft Windows(R).
				jrmUnsubmitted Done Inline Actions Should we use the Oxford comma like in the original? jrm: Should we use the Oxford comma like in the original?
				emasteUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. emaste: I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
				rgrimesUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. I had the same feeling when I read this change before seeing Ed's comment rgrimes: > I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
	By default, bhyve provides access to serial console and does not emulate a graphical console.			By default, bhyve provides access to serial console and does not emulate a graphical console.
	Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.			Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.

	The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).			The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).
	Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).			Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).
	Most newer processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.			Most processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.
				emasteUnsubmitted Done Inline Actions Processors with the necessary support have been available for years so we no longer need "newer." In a subsequent commit we may want to change this to list the one or two families of CPUs that lack support. emaste: Processors with the necessary support have been available for years so we no longer need "newer.
				rgrimesUnsubmitted Done Inline Actions Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point rgrimes: Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point
				emasteUnsubmitted Done Inline Actions I think Nehalem is old enough that it's not going to be clear to users today where that lands. Maybe something like "Most processors made in 2008 or later?" (or whichever year) Or "Intel processors made in 2010 and later support these features" (picking up UG). emaste: I think Nehalem is old enough that it's not going to be clear to users today where that lands.
				rgrimesUnsubmitted Done Inline Actions Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page, and as a member of the list of Intel CPU microarchitectures. The reduction to "most processors" is wrong, that would imply arm, risc-v, or even an at-meta has support for bhyve. rgrimes: Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point. I too was feeling this is overly specific. I'll attempt to condense this further. freebsd_ny-central.org: Good point. I too was feeling this is overly specific. I'll attempt to condense this further.
				emasteUnsubmitted Done Inline Actions Mentioning together "newer processors" and a processor family that is about a decade and a half old doesn't make any sense. To a first order approximation bhyve can be used on any x86_64 processor in use today. Sure, there are some old machines around still and there are retrocomputing enthusiasts, but for the majority of readers this isn't useful information. We also don't give any information about which AMD families are supported. I would just take this whole sentence out. emaste: Mentioning together "newer processors" and a processor family that is about a decade and a half…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Fair point. I too was wondering about AMD, but since I couldn't find any readily available listing of processors with RVI, I left it as is. I guess it's a reasonable compromise to just take it out completely, because we have a paragraph on how to check whether the processor supports bhyve. I'll quickly remove the sentence and update the diff so we can get this landed. freebsd_ny-central.org: Fair point. I too was wondering about AMD, but since I couldn't find any readily available…
	UG support was introduced with Intel's Westmere micro-architecture.			UG support was introduced with Intel's Westmere micro-architecture.
	For a complete list of Intel(R) processors that support EPT, refer to https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[].			For a complete list of Intel(R) processors that support EPT, refer to link:https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[Intel's feature documentation].
	RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.			RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.

	The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.			The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.

	[[virtualization-bhyve-prep]]			[[virtualization-bhyve-prep]]
	=== Preparing the Host			=== Preparing the Host

	The first step to creating a virtual machine in bhyve is configuring the host system.			The first step to creating a virtual machine in bhyve is configuring the host system.
	First, load the bhyve kernel module:			First, load the bhyve kernel module:

	[source,shell]			[source,shell]
	....			....
	# kldload vmm			# kldload vmm
	....			....

	Then, create a [.filename]#tap# interface for the network device in the virtual machine to attach to.			There are several ways to connect a virtual machine guest to a host's network; one straightforward way to accomplish this is to create a [.filename]#tap# interface for the network device in the virtual machine to attach to.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we don't need the comma after `this`. jrm: Grammarly tells me we don't need the comma after `this`.
	In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.			In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.
	In this example, the physical interface is _igb0_:			In this example, the physical interface is _igb0_:

	[source,shell]			[source,shell]
	....			....
	# ifconfig tap0 create			# ifconfig tap0 create
	# sysctl net.link.tap.up_on_open=1			# sysctl net.link.tap.up_on_open=1
	net.link.tap.up_on_open: 0 -> 1			net.link.tap.up_on_open: 0 -> 1
	Show All 12 Lines
	....			....
	# truncate -s 16G guest.img			# truncate -s 16G guest.img
	....			....

	Download an installation image of FreeBSD to install:			Download an installation image of FreeBSD to install:

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
	FreeBSD-13.1-RELEASE-amd64-bootonly.iso 366 MB 16 MBps 22s			FreeBSD-14.0-RELEASE-amd64-bootonly.iso 426 MB 16 MBps 22s
	....			....

	FreeBSD comes with an example script for running a virtual machine in bhyve.			FreeBSD comes with an example script `vmrun.sh` for running a virtual machine in bhyve.
				jrmUnsubmitted Done Inline Actions I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with an example script, vmrun.sh, for running a virtual machine in bhyve. Maybe it was intentional to not refer to the script here because if the name ever changes, it would require more updates. I still think it sounds awkward to only refer to it as `The script`. jrm: I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with…
				rgrimesUnsubmitted Done Inline Actions vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major releases I would say that stop gap status has sailed and directly mentioning it here would be good. rgrimes: vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major…
	The script will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.			It will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.
	The script takes a number of options to control the configuration of the machine: `-c` controls the number of virtual CPUs, `-m` limits the amount of memory available to the guest, `-t` defines which [.filename]#tap# device to use, `-d` indicates which disk image to use, `-i` tells bhyve to boot from the CD image instead of the disk, and `-I` defines which CD image to use.			`vmrun.sh` takes a number of options to control the configuration of the machine, including:
				jrmUnsubmitted Done Inline Actions I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we could phrase it something like this. It takes a number of options to control the configuration of the machine, including: jrm: I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we…
				rgrimesUnsubmitted Done Inline Actions Dont document vmrun.sh here, simply link to the man page. rgrimes: Dont document vmrun.sh here, simply link to the man page.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2-RELEASE. Might be worthwhile to create a man page for this in a separate review, I suppose. freebsd_ny-central.org: There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2…
				rgrimesUnsubmitted Done Inline Actions Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even partially and NOT providing a method to get the full syntax avaliable. Also the documenting here is likely to bit rot rather quickly rgrimes: Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions You're certainly right with the potential for bitrot. For now, I just changed the formatting of the previously listed options. I'll add some info about --usage to allow users to get a full list of available options. freebsd_ny-central.org: You're certainly right with the potential for bitrot. For now, I just changed the formatting of…
	The last parameter is the name of the virtual machine, used to track the running machines.
				* `-c` controls the number of virtual CPUs,
				* `-m` limits the amount of memory available to the guest,
				* `-t` defines which [.filename]#tap# device to use,
				* `-d` indicates which disk image to use,
				* `-i` tells bhyve to boot from the CD image instead of the disk, and
				* `-I` defines which CD image to use.

				The last parameter is the name of the virtual machine and used to track the running machines.
	This example starts the virtual machine in installation mode:			This example starts the virtual machine in installation mode:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-13.1-RELEASE-amd64-bootonly.iso guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-14.0-RELEASE-amd64-bootonly.iso guestname
				rgrimesUnsubmitted Done Inline Actions Is the "sh" even needed, this file should already be installed mode 755 so can be directly invoked. If its not installed 755 that should be fixed. rgrimes: Is the "sh" even needed, this file should already be installed mode 755 so can be directly…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE at the moment, at least. I agree that it would be good to change that. I'll have a look at that for another review. freebsd_ny-central.org: Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE…
	....			....

	The virtual machine will boot and start the installer.			The virtual machine will boot and start the installer.
	After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].			After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].

	Reboot the virtual machine.			Reboot the virtual machine.
	While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.			While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.
	When this happens, choose the reboot option from the boot loader menu in order to escape the loop.			When this happens, choose the reboot option from the boot loader menu in order to escape the loop.
	Now the guest can be started from the virtual disk:			Now the guest can be started from the virtual disk:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname
	....			....

	[[virtualization-bhyve-linux]]			[[virtualization-bhyve-linux]]
	=== Creating a Linux(R) Guest			=== Creating a Linux(R) Guest

	In order to boot operating systems other than FreeBSD, the package:sysutils/grub2-bhyve[] port must be first installed.			Linux guests can be booted either like any other regular crossref:virtualization[virtualization-bhyve-uefi,"UEFI-based guest"] virtual machine, or alternatively, you can make use of the package:sysutils/grub2-bhyve[] port.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we also need a comma after `alternatively`. jrm: Grammarly tells me we also need a comma after `alternatively`.
				emasteUnsubmitted Done Inline Actions We don't need the (R) after the first use of a trademark. I know the existing text has it everywhere but no need to add more of them. emaste: We don't need the (R) after the first use of a trademark. I know the existing text has it…

	Next, create a file to use as the virtual disk for the guest machine:			To do this, first ensure that the port is installed, then create a file to use as the virtual disk for the guest machine:

	[source,shell]			[source,shell]
	....			....
	# truncate -s 16G linux.img			# truncate -s 16G linux.img
	....			....

	Starting a virtual machine with bhyve is a two step process.			Starting a Linux virtual machine with `grub2-bhyve` is a two step process.
	First a kernel must be loaded, then the guest can be started.
	The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].			. First a kernel must be loaded, then the guest can be started.
				jrmUnsubmitted Done Inline Actions I don't think this renders as you intended. Did you mean to make this an unordered list? jrm: I don't think this renders as you intended. Did you mean to make this an unordered list?
				. The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].

	Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:			Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:

	[.programlisting]			[.programlisting]
	....			....
	(hd0) ./linux.img			(hd0) ./linux.img
	(cd0) ./somelinux.iso			(cd0) ./somelinux.iso
	....			....

	▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=linuxguest			# bhyvectl --destroy --vm=linuxguest
	....			....

	[[virtualization-bhyve-uefi]]			[[virtualization-bhyve-uefi]]
	=== Booting bhyve Virtual Machines with UEFI Firmware			=== Booting bhyve Virtual Machines with UEFI Firmware

				rgrimesUnsubmitted Done Inline Actions strike userspace, its nonsensical in the context of firmware. rgrimes: strike userspace, its nonsensical in the context of firmware.
	In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.			In addition to `bhyveload` and `grub-bhyve`, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.
	This option may support guest operating systems that are not supported by the other loaders.			This option may support guest operating systems that are not supported by the other loaders.

	In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.			In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.
	This can be done by installing package:sysutils/bhyve-firmware[] port or package.			This can be done by installing package:sysutils/bhyve-firmware[] port or package.

	With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.			With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.
	The actual bhyve command may look like this:			The actual bhyve command may look like this:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
	guest			guest
	....			....

				rgrimesUnsubmitted Done Inline Actions We should be consistent on EFI vs UEFI, use one or the other in all places please rgrimes: We should be consistent on EFI vs UEFI, use one or the other in all places please
				emasteUnsubmitted Done Inline Actions Heh, from efivar(8) efivar – UEFI environment variable interaction emaste: Heh, from efivar(8) efivar – UEFI environment variable interaction
				To allow a guest to store EFI variables, you can use a variables file appended to the `-l` flag. Note that bhyve will write guest modifications to the given variables file.
				Therefore, be sure to first create a per-guest-copy of the variables template file:

				[source,shell]
				....
				# cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /path/to/vm-image/BHYVE_UEFI_VARS.fd
				....

				Then, add that variables file into your bhyve arguments:

				[source,shell]
				....
				# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
				-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
				-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
				-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/path/to/vm-image/BHYVE_UEFI_VARS.fd \
				guest
				....

	package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:			package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
	Show All 39 Lines
	When starting the VM, specify the ZFS volume as the disk drive:			When starting the VM, specify the ZFS volume as the disk drive:

	[source,shell]			[source,shell]
	....			....
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \
	-l com1,stdio -c 4 -m 1024M linuxguest			-l com1,stdio -c 4 -m 1024M linuxguest
	....			....

				If you are using ZFS for the host as well as inside a guest, keep in mind the competing memory pressure of both systems caching the virtual machine's contents.
				jrmUnsubmitted Done Inline Actions Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache. jrm: Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache.
				rgrimesUnsubmitted Done Inline Actions Also recommend the use of zvol's vs a file in a file system for the disk image. rgrimes: Also recommend the use of zvol's vs a file in a file system for the disk image.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point - there is a statement to this a few lines up; in my previous diff, this wasn't visible because I missed the -U99999 initially on my patch. Apologies for that. freebsd_ny-central.org: Good point - there is a statement to this a few lines up; in my previous diff, this wasn't…
				To alleviate this, consider setting the host's ZFS filesystems to use metadata-only cache.
				To do this, apply the following settings to ZFS filesystems on the host, replacing `<name>` with the name of the specific zvol dataset name of the virtual machine.

				[source,shell]
				....
				# zfs set primarycache=metadata <name>
				....

	[[virtualization-bhyve-nmdm]]			[[virtualization-bhyve-nmdm]]
	=== Virtual Machine Consoles			=== Virtual Machine Consoles

	It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.			It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.
	It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.			It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.
	To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.			To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.
	The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).			The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).
	See man:nmdm[4] for more information.			See man:nmdm[4] for more information.

	[source,shell]			[source,shell]
	....			....
	# kldload nmdm			# kldload nmdm
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \
	-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest			-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest
	# cu -l /dev/nmdm0B			# cu -l /dev/nmdm0B
	Connected			Connected

	Ubuntu 13.10 handbook ttyS0			Ubuntu 13.10 handbook ttyS0

	handbook login:			handbook login:
	....			....

				rgrimesUnsubmitted Done Inline Actions I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not follow a newline you wont get the disconnect, or at least that is how it works every where else. rgrimes: I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not…
				rgrimesUnsubmitted Done Inline Actions I do not see that this was incorporated??? rgrimes: I do not see that this was incorporated???
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a dot list syntax mistake I made previously. I'll fix this right now and update the diff one more time. freebsd_ny-central.org: Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a…
				To disconnect from a console, press `~` followed by Ctrl+D or `.`. Keep in mind that only the connection is dropped while the login session remains active.
				Another user connecting to the same console could therefore make use of any active sessions without having to first authenticate.
				For security reasons, it's therefore recommended to logout before disconnecting.

				The number in the [.filename]#nmdm# device path must be unique for each virtual machine and must not be used by any other processes before bhyve starts.
				The number can be chosen arbitrarily and does not need to be taken from a consecutive sequence of numbers.
				The device node pair (i.e. [.filename]#/dev/nmdm0a# and [.filename]#/dev/nmdm0b#) are created dynamically when bhyve connects its console and destroyed when it shuts down.
				Keep this in mind when creating scripts to start your virtual machines: you need to make sure that all virtual machines are assigned unique [.filename]#nmdm# devices.

	[[virtualization-bhyve-managing]]			[[virtualization-bhyve-managing]]
	=== Managing Virtual Machines			=== Managing Virtual Machines

	A device node is created in [.filename]#/dev/vmm# for each virtual machine.			A device node is created in [.filename]#/dev/vmm# for each virtual machine.
	This allows the administrator to easily see a list of the running virtual machines:			This allows the administrator to easily see a list of the running virtual machines:

	[source,shell]			[source,shell]
	....			....
	# ls -al /dev/vmm			# ls -al /dev/vmm
	total 1			total 1
	dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./			dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./
	dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../			dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../
	crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname			crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname
	crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest			crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest
	crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest			crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest
	....			....

	A specified virtual machine can be destroyed using `bhyvectl`:			A specified virtual machine can be destroyed using `bhyvectl`:

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=guestname			# bhyvectl --destroy --vm=guestname
	....			....

				Destroying a virtual machine this way means killing it immediately. Any unsaved data will be lost, open files and filesystems may get corrupted.
				To gracefully shut down a virtual machine, you can instead send a `TERM` signal to its bhyve process. This triggers an ACPI shutdown event for the guest:

				[source,shell]
				....
				# ps ax \| grep bhyve
				17424 - SC 56:48.27 bhyve: guestvm (bhyve)
				# kill 17424
				....

				[[virtualization-tools-utilities]]
				=== Tools and Utilities

				There are numerous utilities and applications available in ports to help simplify setting up and managing bhyve virtual machines:

				.bhyve Managers
				[options="header", cols="1,1,1,1"]
				\|===
				\| Name \| License \| Package \| Documentation

				\| vm-bhyve
				\| BSD-2
				\| package:sysutils/vm-bhyve[]
				\| link:https://github.com/churchers/vm-bhyve[Documentation]

				\| CBSD
				\| BSD-2
				\| package:sysutils/cbsd[]
				\| link:https://www.bsdstore.ru[Documentation]

				\| Virt-Manager
				\| LGPL-3
				\| package:deskutils/virt-manager[]
				\| link:https://virt-manager.org/[Documentation]

				\| Bhyve RC Script
				\| Unknown
				\| package:sysutils/bhyve-rc[]
				\| link:https://www.freshports.org/sysutils/bhyve-rc/[Documentation]

				\| vmstated
				\| BSD-2
				\| package:sysutils/vmstated[]
				\| link:https://github.com/christian-moerz/vmstated[Documentation]

				\|===

	[[virtualization-bhyve-onboot]]			[[virtualization-bhyve-onboot]]
	=== Persistent Configuration			=== Persistent Configuration

	In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:			In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:

	[.procedure]			[.procedure]
	. [.filename]#/etc/sysctl.conf#			. [.filename]#/etc/sysctl.conf#
	+			+
	▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
	Unprivileged domains consist of a configuration file and virtual or physical hard disks.			Unprivileged domains consist of a configuration file and virtual or physical hard disks.
	Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].			Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].
	In this example, a 20 GB volume is used.			In this example, a 20 GB volume is used.
	A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.			A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.
	The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.			The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso -o freebsd.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso -o freebsd.iso
	....			....

	A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.			A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.

	[source,shell]			[source,shell]
	....			....
	# zfs create -V20G -o volmode=dev zroot/xendisk0			# zfs create -V20G -o volmode=dev zroot/xendisk0
	....			....
	▲ Show 20 Lines • Show All 133 Lines • Show Last 20 Lines

View Options

sys/net/route/route_ctl.c

	Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Lines
	....			....
	# service devfs restart			# service devfs restart
	....			....

	[[virtualization-host-bhyve]]			[[virtualization-host-bhyve]]
	== FreeBSD as a Host with bhyve			== FreeBSD as a Host with bhyve

	The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.			The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.
	This hypervisor supports a number of guests, including FreeBSD, OpenBSD, and many Linux(R) distributions.			This hypervisor supports a number of guests, including FreeBSD, OpenBSD, many Linux(R) distributions, and Microsoft Windows(R).
				jrmUnsubmitted Done Inline Actions Should we use the Oxford comma like in the original? jrm: Should we use the Oxford comma like in the original?
				emasteUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. emaste: I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
				rgrimesUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. I had the same feeling when I read this change before seeing Ed's comment rgrimes: > I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
	By default, bhyve provides access to serial console and does not emulate a graphical console.			By default, bhyve provides access to serial console and does not emulate a graphical console.
	Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.			Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.

	The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).			The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).
	Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).			Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).
	Most newer processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.			Most processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.
				emasteUnsubmitted Done Inline Actions Processors with the necessary support have been available for years so we no longer need "newer." In a subsequent commit we may want to change this to list the one or two families of CPUs that lack support. emaste: Processors with the necessary support have been available for years so we no longer need "newer.
				rgrimesUnsubmitted Done Inline Actions Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point rgrimes: Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point
				emasteUnsubmitted Done Inline Actions I think Nehalem is old enough that it's not going to be clear to users today where that lands. Maybe something like "Most processors made in 2008 or later?" (or whichever year) Or "Intel processors made in 2010 and later support these features" (picking up UG). emaste: I think Nehalem is old enough that it's not going to be clear to users today where that lands.
				rgrimesUnsubmitted Done Inline Actions Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page, and as a member of the list of Intel CPU microarchitectures. The reduction to "most processors" is wrong, that would imply arm, risc-v, or even an at-meta has support for bhyve. rgrimes: Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point. I too was feeling this is overly specific. I'll attempt to condense this further. freebsd_ny-central.org: Good point. I too was feeling this is overly specific. I'll attempt to condense this further.
				emasteUnsubmitted Done Inline Actions Mentioning together "newer processors" and a processor family that is about a decade and a half old doesn't make any sense. To a first order approximation bhyve can be used on any x86_64 processor in use today. Sure, there are some old machines around still and there are retrocomputing enthusiasts, but for the majority of readers this isn't useful information. We also don't give any information about which AMD families are supported. I would just take this whole sentence out. emaste: Mentioning together "newer processors" and a processor family that is about a decade and a half…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Fair point. I too was wondering about AMD, but since I couldn't find any readily available listing of processors with RVI, I left it as is. I guess it's a reasonable compromise to just take it out completely, because we have a paragraph on how to check whether the processor supports bhyve. I'll quickly remove the sentence and update the diff so we can get this landed. freebsd_ny-central.org: Fair point. I too was wondering about AMD, but since I couldn't find any readily available…
	UG support was introduced with Intel's Westmere micro-architecture.			UG support was introduced with Intel's Westmere micro-architecture.
	For a complete list of Intel(R) processors that support EPT, refer to https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[].			For a complete list of Intel(R) processors that support EPT, refer to link:https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[Intel's feature documentation].
	RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.			RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.

	The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.			The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.

	[[virtualization-bhyve-prep]]			[[virtualization-bhyve-prep]]
	=== Preparing the Host			=== Preparing the Host

	The first step to creating a virtual machine in bhyve is configuring the host system.			The first step to creating a virtual machine in bhyve is configuring the host system.
	First, load the bhyve kernel module:			First, load the bhyve kernel module:

	[source,shell]			[source,shell]
	....			....
	# kldload vmm			# kldload vmm
	....			....

	Then, create a [.filename]#tap# interface for the network device in the virtual machine to attach to.			There are several ways to connect a virtual machine guest to a host's network; one straightforward way to accomplish this is to create a [.filename]#tap# interface for the network device in the virtual machine to attach to.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we don't need the comma after `this`. jrm: Grammarly tells me we don't need the comma after `this`.
	In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.			In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.
	In this example, the physical interface is _igb0_:			In this example, the physical interface is _igb0_:

	[source,shell]			[source,shell]
	....			....
	# ifconfig tap0 create			# ifconfig tap0 create
	# sysctl net.link.tap.up_on_open=1			# sysctl net.link.tap.up_on_open=1
	net.link.tap.up_on_open: 0 -> 1			net.link.tap.up_on_open: 0 -> 1
	Show All 12 Lines
	....			....
	# truncate -s 16G guest.img			# truncate -s 16G guest.img
	....			....

	Download an installation image of FreeBSD to install:			Download an installation image of FreeBSD to install:

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
	FreeBSD-13.1-RELEASE-amd64-bootonly.iso 366 MB 16 MBps 22s			FreeBSD-14.0-RELEASE-amd64-bootonly.iso 426 MB 16 MBps 22s
	....			....

	FreeBSD comes with an example script for running a virtual machine in bhyve.			FreeBSD comes with an example script `vmrun.sh` for running a virtual machine in bhyve.
				jrmUnsubmitted Done Inline Actions I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with an example script, vmrun.sh, for running a virtual machine in bhyve. Maybe it was intentional to not refer to the script here because if the name ever changes, it would require more updates. I still think it sounds awkward to only refer to it as `The script`. jrm: I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with…
				rgrimesUnsubmitted Done Inline Actions vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major releases I would say that stop gap status has sailed and directly mentioning it here would be good. rgrimes: vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major…
	The script will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.			It will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.
	The script takes a number of options to control the configuration of the machine: `-c` controls the number of virtual CPUs, `-m` limits the amount of memory available to the guest, `-t` defines which [.filename]#tap# device to use, `-d` indicates which disk image to use, `-i` tells bhyve to boot from the CD image instead of the disk, and `-I` defines which CD image to use.			`vmrun.sh` takes a number of options to control the configuration of the machine, including:
				jrmUnsubmitted Done Inline Actions I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we could phrase it something like this. It takes a number of options to control the configuration of the machine, including: jrm: I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we…
				rgrimesUnsubmitted Done Inline Actions Dont document vmrun.sh here, simply link to the man page. rgrimes: Dont document vmrun.sh here, simply link to the man page.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2-RELEASE. Might be worthwhile to create a man page for this in a separate review, I suppose. freebsd_ny-central.org: There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2…
				rgrimesUnsubmitted Done Inline Actions Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even partially and NOT providing a method to get the full syntax avaliable. Also the documenting here is likely to bit rot rather quickly rgrimes: Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions You're certainly right with the potential for bitrot. For now, I just changed the formatting of the previously listed options. I'll add some info about --usage to allow users to get a full list of available options. freebsd_ny-central.org: You're certainly right with the potential for bitrot. For now, I just changed the formatting of…
	The last parameter is the name of the virtual machine, used to track the running machines.
				* `-c` controls the number of virtual CPUs,
				* `-m` limits the amount of memory available to the guest,
				* `-t` defines which [.filename]#tap# device to use,
				* `-d` indicates which disk image to use,
				* `-i` tells bhyve to boot from the CD image instead of the disk, and
				* `-I` defines which CD image to use.

				The last parameter is the name of the virtual machine and used to track the running machines.
	This example starts the virtual machine in installation mode:			This example starts the virtual machine in installation mode:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-13.1-RELEASE-amd64-bootonly.iso guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-14.0-RELEASE-amd64-bootonly.iso guestname
				rgrimesUnsubmitted Done Inline Actions Is the "sh" even needed, this file should already be installed mode 755 so can be directly invoked. If its not installed 755 that should be fixed. rgrimes: Is the "sh" even needed, this file should already be installed mode 755 so can be directly…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE at the moment, at least. I agree that it would be good to change that. I'll have a look at that for another review. freebsd_ny-central.org: Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE…
	....			....

	The virtual machine will boot and start the installer.			The virtual machine will boot and start the installer.
	After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].			After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].

	Reboot the virtual machine.			Reboot the virtual machine.
	While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.			While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.
	When this happens, choose the reboot option from the boot loader menu in order to escape the loop.			When this happens, choose the reboot option from the boot loader menu in order to escape the loop.
	Now the guest can be started from the virtual disk:			Now the guest can be started from the virtual disk:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname
	....			....

	[[virtualization-bhyve-linux]]			[[virtualization-bhyve-linux]]
	=== Creating a Linux(R) Guest			=== Creating a Linux(R) Guest

	In order to boot operating systems other than FreeBSD, the package:sysutils/grub2-bhyve[] port must be first installed.			Linux guests can be booted either like any other regular crossref:virtualization[virtualization-bhyve-uefi,"UEFI-based guest"] virtual machine, or alternatively, you can make use of the package:sysutils/grub2-bhyve[] port.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we also need a comma after `alternatively`. jrm: Grammarly tells me we also need a comma after `alternatively`.
				emasteUnsubmitted Done Inline Actions We don't need the (R) after the first use of a trademark. I know the existing text has it everywhere but no need to add more of them. emaste: We don't need the (R) after the first use of a trademark. I know the existing text has it…

	Next, create a file to use as the virtual disk for the guest machine:			To do this, first ensure that the port is installed, then create a file to use as the virtual disk for the guest machine:

	[source,shell]			[source,shell]
	....			....
	# truncate -s 16G linux.img			# truncate -s 16G linux.img
	....			....

	Starting a virtual machine with bhyve is a two step process.			Starting a Linux virtual machine with `grub2-bhyve` is a two step process.
	First a kernel must be loaded, then the guest can be started.
	The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].			. First a kernel must be loaded, then the guest can be started.
				jrmUnsubmitted Done Inline Actions I don't think this renders as you intended. Did you mean to make this an unordered list? jrm: I don't think this renders as you intended. Did you mean to make this an unordered list?
				. The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].

	Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:			Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:

	[.programlisting]			[.programlisting]
	....			....
	(hd0) ./linux.img			(hd0) ./linux.img
	(cd0) ./somelinux.iso			(cd0) ./somelinux.iso
	....			....

	▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=linuxguest			# bhyvectl --destroy --vm=linuxguest
	....			....

	[[virtualization-bhyve-uefi]]			[[virtualization-bhyve-uefi]]
	=== Booting bhyve Virtual Machines with UEFI Firmware			=== Booting bhyve Virtual Machines with UEFI Firmware

				rgrimesUnsubmitted Done Inline Actions strike userspace, its nonsensical in the context of firmware. rgrimes: strike userspace, its nonsensical in the context of firmware.
	In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.			In addition to `bhyveload` and `grub-bhyve`, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.
	This option may support guest operating systems that are not supported by the other loaders.			This option may support guest operating systems that are not supported by the other loaders.

	In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.			In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.
	This can be done by installing package:sysutils/bhyve-firmware[] port or package.			This can be done by installing package:sysutils/bhyve-firmware[] port or package.

	With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.			With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.
	The actual bhyve command may look like this:			The actual bhyve command may look like this:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
	guest			guest
	....			....

				rgrimesUnsubmitted Done Inline Actions We should be consistent on EFI vs UEFI, use one or the other in all places please rgrimes: We should be consistent on EFI vs UEFI, use one or the other in all places please
				emasteUnsubmitted Done Inline Actions Heh, from efivar(8) efivar – UEFI environment variable interaction emaste: Heh, from efivar(8) efivar – UEFI environment variable interaction
				To allow a guest to store EFI variables, you can use a variables file appended to the `-l` flag. Note that bhyve will write guest modifications to the given variables file.
				Therefore, be sure to first create a per-guest-copy of the variables template file:

				[source,shell]
				....
				# cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /path/to/vm-image/BHYVE_UEFI_VARS.fd
				....

				Then, add that variables file into your bhyve arguments:

				[source,shell]
				....
				# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
				-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
				-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
				-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/path/to/vm-image/BHYVE_UEFI_VARS.fd \
				guest
				....

	package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:			package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
	Show All 39 Lines
	When starting the VM, specify the ZFS volume as the disk drive:			When starting the VM, specify the ZFS volume as the disk drive:

	[source,shell]			[source,shell]
	....			....
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \
	-l com1,stdio -c 4 -m 1024M linuxguest			-l com1,stdio -c 4 -m 1024M linuxguest
	....			....

				If you are using ZFS for the host as well as inside a guest, keep in mind the competing memory pressure of both systems caching the virtual machine's contents.
				jrmUnsubmitted Done Inline Actions Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache. jrm: Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache.
				rgrimesUnsubmitted Done Inline Actions Also recommend the use of zvol's vs a file in a file system for the disk image. rgrimes: Also recommend the use of zvol's vs a file in a file system for the disk image.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point - there is a statement to this a few lines up; in my previous diff, this wasn't visible because I missed the -U99999 initially on my patch. Apologies for that. freebsd_ny-central.org: Good point - there is a statement to this a few lines up; in my previous diff, this wasn't…
				To alleviate this, consider setting the host's ZFS filesystems to use metadata-only cache.
				To do this, apply the following settings to ZFS filesystems on the host, replacing `<name>` with the name of the specific zvol dataset name of the virtual machine.

				[source,shell]
				....
				# zfs set primarycache=metadata <name>
				....

	[[virtualization-bhyve-nmdm]]			[[virtualization-bhyve-nmdm]]
	=== Virtual Machine Consoles			=== Virtual Machine Consoles

	It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.			It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.
	It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.			It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.
	To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.			To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.
	The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).			The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).
	See man:nmdm[4] for more information.			See man:nmdm[4] for more information.

	[source,shell]			[source,shell]
	....			....
	# kldload nmdm			# kldload nmdm
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \
	-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest			-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest
	# cu -l /dev/nmdm0B			# cu -l /dev/nmdm0B
	Connected			Connected

	Ubuntu 13.10 handbook ttyS0			Ubuntu 13.10 handbook ttyS0

	handbook login:			handbook login:
	....			....

				rgrimesUnsubmitted Done Inline Actions I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not follow a newline you wont get the disconnect, or at least that is how it works every where else. rgrimes: I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not…
				rgrimesUnsubmitted Done Inline Actions I do not see that this was incorporated??? rgrimes: I do not see that this was incorporated???
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a dot list syntax mistake I made previously. I'll fix this right now and update the diff one more time. freebsd_ny-central.org: Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a…
				To disconnect from a console, press `~` followed by Ctrl+D or `.`. Keep in mind that only the connection is dropped while the login session remains active.
				Another user connecting to the same console could therefore make use of any active sessions without having to first authenticate.
				For security reasons, it's therefore recommended to logout before disconnecting.

				The number in the [.filename]#nmdm# device path must be unique for each virtual machine and must not be used by any other processes before bhyve starts.
				The number can be chosen arbitrarily and does not need to be taken from a consecutive sequence of numbers.
				The device node pair (i.e. [.filename]#/dev/nmdm0a# and [.filename]#/dev/nmdm0b#) are created dynamically when bhyve connects its console and destroyed when it shuts down.
				Keep this in mind when creating scripts to start your virtual machines: you need to make sure that all virtual machines are assigned unique [.filename]#nmdm# devices.

	[[virtualization-bhyve-managing]]			[[virtualization-bhyve-managing]]
	=== Managing Virtual Machines			=== Managing Virtual Machines

	A device node is created in [.filename]#/dev/vmm# for each virtual machine.			A device node is created in [.filename]#/dev/vmm# for each virtual machine.
	This allows the administrator to easily see a list of the running virtual machines:			This allows the administrator to easily see a list of the running virtual machines:

	[source,shell]			[source,shell]
	....			....
	# ls -al /dev/vmm			# ls -al /dev/vmm
	total 1			total 1
	dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./			dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./
	dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../			dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../
	crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname			crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname
	crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest			crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest
	crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest			crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest
	....			....

	A specified virtual machine can be destroyed using `bhyvectl`:			A specified virtual machine can be destroyed using `bhyvectl`:

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=guestname			# bhyvectl --destroy --vm=guestname
	....			....

				Destroying a virtual machine this way means killing it immediately. Any unsaved data will be lost, open files and filesystems may get corrupted.
				To gracefully shut down a virtual machine, you can instead send a `TERM` signal to its bhyve process. This triggers an ACPI shutdown event for the guest:

				[source,shell]
				....
				# ps ax \| grep bhyve
				17424 - SC 56:48.27 bhyve: guestvm (bhyve)
				# kill 17424
				....

				[[virtualization-tools-utilities]]
				=== Tools and Utilities

				There are numerous utilities and applications available in ports to help simplify setting up and managing bhyve virtual machines:

				.bhyve Managers
				[options="header", cols="1,1,1,1"]
				\|===
				\| Name \| License \| Package \| Documentation

				\| vm-bhyve
				\| BSD-2
				\| package:sysutils/vm-bhyve[]
				\| link:https://github.com/churchers/vm-bhyve[Documentation]

				\| CBSD
				\| BSD-2
				\| package:sysutils/cbsd[]
				\| link:https://www.bsdstore.ru[Documentation]

				\| Virt-Manager
				\| LGPL-3
				\| package:deskutils/virt-manager[]
				\| link:https://virt-manager.org/[Documentation]

				\| Bhyve RC Script
				\| Unknown
				\| package:sysutils/bhyve-rc[]
				\| link:https://www.freshports.org/sysutils/bhyve-rc/[Documentation]

				\| vmstated
				\| BSD-2
				\| package:sysutils/vmstated[]
				\| link:https://github.com/christian-moerz/vmstated[Documentation]

				\|===

	[[virtualization-bhyve-onboot]]			[[virtualization-bhyve-onboot]]
	=== Persistent Configuration			=== Persistent Configuration

	In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:			In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:

	[.procedure]			[.procedure]
	. [.filename]#/etc/sysctl.conf#			. [.filename]#/etc/sysctl.conf#
	+			+
	▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
	Unprivileged domains consist of a configuration file and virtual or physical hard disks.			Unprivileged domains consist of a configuration file and virtual or physical hard disks.
	Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].			Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].
	In this example, a 20 GB volume is used.			In this example, a 20 GB volume is used.
	A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.			A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.
	The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.			The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso -o freebsd.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso -o freebsd.iso
	....			....

	A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.			A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.

	[source,shell]			[source,shell]
	....			....
	# zfs create -V20G -o volmode=dev zroot/xendisk0			# zfs create -V20G -o volmode=dev zroot/xendisk0
	....			....
	▲ Show 20 Lines • Show All 133 Lines • Show Last 20 Lines

View Options

sys/netinet/ip_input.c

	Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Lines
	....			....
	# service devfs restart			# service devfs restart
	....			....

	[[virtualization-host-bhyve]]			[[virtualization-host-bhyve]]
	== FreeBSD as a Host with bhyve			== FreeBSD as a Host with bhyve

	The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.			The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE.
	This hypervisor supports a number of guests, including FreeBSD, OpenBSD, and many Linux(R) distributions.			This hypervisor supports a number of guests, including FreeBSD, OpenBSD, many Linux(R) distributions, and Microsoft Windows(R).
				jrmUnsubmitted Done Inline Actions Should we use the Oxford comma like in the original? jrm: Should we use the Oxford comma like in the original?
				emasteUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. emaste: I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
				rgrimesUnsubmitted Done Inline Actions I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we support Windows. I had the same feeling when I read this change before seeing Ed's comment rgrimes: > I'd probably drop "even" -- hopefully at this point it isn't surprising or unusual that we…
	By default, bhyve provides access to serial console and does not emulate a graphical console.			By default, bhyve provides access to serial console and does not emulate a graphical console.
	Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.			Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.

	The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).			The bhyve design requires a processor that supports Intel(R) Extended Page Tables (EPT) or AMD(R) Rapid Virtualization Indexing (RVI) or Nested Page Tables (NPT).
	Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).			Hosting Linux(R) guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG).
	Most newer processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.			Most processors, specifically the Intel(R) Core(TM) i3/i5/i7 and Intel(R) Xeon(TM) E3/E5/E7, support these features.
				emasteUnsubmitted Done Inline Actions Processors with the necessary support have been available for years so we no longer need "newer." In a subsequent commit we may want to change this to list the one or two families of CPUs that lack support. emaste: Processors with the necessary support have been available for years so we no longer need "newer.
				rgrimesUnsubmitted Done Inline Actions Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point rgrimes: Perhaps simply "any processor nehalem or newer"? IIRC thats the cut off point
				emasteUnsubmitted Done Inline Actions I think Nehalem is old enough that it's not going to be clear to users today where that lands. Maybe something like "Most processors made in 2008 or later?" (or whichever year) Or "Intel processors made in 2010 and later support these features" (picking up UG). emaste: I think Nehalem is old enough that it's not going to be clear to users today where that lands.
				rgrimesUnsubmitted Done Inline Actions Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page, and as a member of the list of Intel CPU microarchitectures. The reduction to "most processors" is wrong, that would imply arm, risc-v, or even an at-meta has support for bhyve. rgrimes: Clear or not Nehalem is a well documented intel micro architecture, both with its own Wiki page…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point. I too was feeling this is overly specific. I'll attempt to condense this further. freebsd_ny-central.org: Good point. I too was feeling this is overly specific. I'll attempt to condense this further.
				emasteUnsubmitted Done Inline Actions Mentioning together "newer processors" and a processor family that is about a decade and a half old doesn't make any sense. To a first order approximation bhyve can be used on any x86_64 processor in use today. Sure, there are some old machines around still and there are retrocomputing enthusiasts, but for the majority of readers this isn't useful information. We also don't give any information about which AMD families are supported. I would just take this whole sentence out. emaste: Mentioning together "newer processors" and a processor family that is about a decade and a half…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Fair point. I too was wondering about AMD, but since I couldn't find any readily available listing of processors with RVI, I left it as is. I guess it's a reasonable compromise to just take it out completely, because we have a paragraph on how to check whether the processor supports bhyve. I'll quickly remove the sentence and update the diff so we can get this landed. freebsd_ny-central.org: Fair point. I too was wondering about AMD, but since I couldn't find any readily available…
	UG support was introduced with Intel's Westmere micro-architecture.			UG support was introduced with Intel's Westmere micro-architecture.
	For a complete list of Intel(R) processors that support EPT, refer to https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[].			For a complete list of Intel(R) processors that support EPT, refer to link:https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&0_ExtendedPageTables=True[Intel's feature documentation].
	RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.			RVI is found on the third generation and later of the AMD Opteron(TM) (Barcelona) processors.

	The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.			The easiest way to tell if a processor supports bhyve is to run `dmesg` or look in [.filename]#/var/run/dmesg.boot# for the `POPCNT` processor feature flag on the `Features2` line for AMD(R) processors or `EPT` and `UG` on the `VT-x` line for Intel(R) processors.

	[[virtualization-bhyve-prep]]			[[virtualization-bhyve-prep]]
	=== Preparing the Host			=== Preparing the Host

	The first step to creating a virtual machine in bhyve is configuring the host system.			The first step to creating a virtual machine in bhyve is configuring the host system.
	First, load the bhyve kernel module:			First, load the bhyve kernel module:

	[source,shell]			[source,shell]
	....			....
	# kldload vmm			# kldload vmm
	....			....

	Then, create a [.filename]#tap# interface for the network device in the virtual machine to attach to.			There are several ways to connect a virtual machine guest to a host's network; one straightforward way to accomplish this is to create a [.filename]#tap# interface for the network device in the virtual machine to attach to.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we don't need the comma after `this`. jrm: Grammarly tells me we don't need the comma after `this`.
	In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.			In order for the network device to participate in the network, also create a bridge interface containing the [.filename]#tap# interface and the physical interface as members.
	In this example, the physical interface is _igb0_:			In this example, the physical interface is _igb0_:

	[source,shell]			[source,shell]
	....			....
	# ifconfig tap0 create			# ifconfig tap0 create
	# sysctl net.link.tap.up_on_open=1			# sysctl net.link.tap.up_on_open=1
	net.link.tap.up_on_open: 0 -> 1			net.link.tap.up_on_open: 0 -> 1
	Show All 12 Lines
	....			....
	# truncate -s 16G guest.img			# truncate -s 16G guest.img
	....			....

	Download an installation image of FreeBSD to install:			Download an installation image of FreeBSD to install:

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
	FreeBSD-13.1-RELEASE-amd64-bootonly.iso 366 MB 16 MBps 22s			FreeBSD-14.0-RELEASE-amd64-bootonly.iso 426 MB 16 MBps 22s
	....			....

	FreeBSD comes with an example script for running a virtual machine in bhyve.			FreeBSD comes with an example script `vmrun.sh` for running a virtual machine in bhyve.
				jrmUnsubmitted Done Inline Actions I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with an example script, vmrun.sh, for running a virtual machine in bhyve. Maybe it was intentional to not refer to the script here because if the name ever changes, it would require more updates. I still think it sounds awkward to only refer to it as `The script`. jrm: I wonder if we should refer to the script at this point. Something like, "FreeBSD comes with…
				rgrimesUnsubmitted Done Inline Actions vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major releases I would say that stop gap status has sailed and directly mentioning it here would be good. rgrimes: vmrun.sh was suppose to be just a stop gap thing, but since it has lasted throught 4 major…
	The script will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.			It will start the virtual machine and run it in a loop, so it will automatically restart if it crashes.
	The script takes a number of options to control the configuration of the machine: `-c` controls the number of virtual CPUs, `-m` limits the amount of memory available to the guest, `-t` defines which [.filename]#tap# device to use, `-d` indicates which disk image to use, `-i` tells bhyve to boot from the CD image instead of the disk, and `-I` defines which CD image to use.			`vmrun.sh` takes a number of options to control the configuration of the machine, including:
				jrmUnsubmitted Done Inline Actions I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we could phrase it something like this. It takes a number of options to control the configuration of the machine, including: jrm: I like turning this into a list. I see this isn't an exhaustive list of options, so maybe we…
				rgrimesUnsubmitted Done Inline Actions Dont document vmrun.sh here, simply link to the man page. rgrimes: Dont document vmrun.sh here, simply link to the man page.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2-RELEASE. Might be worthwhile to create a man page for this in a separate review, I suppose. freebsd_ny-central.org: There appears to be no man page? At least I'm not getting anything for vmrun.sh on my 13.2…
				rgrimesUnsubmitted Done Inline Actions Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even partially and NOT providing a method to get the full syntax avaliable. Also the documenting here is likely to bit rot rather quickly rgrimes: Ok, there is "vmrun.sh -usage" though. I dislike the idea of documenting it here, even…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions You're certainly right with the potential for bitrot. For now, I just changed the formatting of the previously listed options. I'll add some info about --usage to allow users to get a full list of available options. freebsd_ny-central.org: You're certainly right with the potential for bitrot. For now, I just changed the formatting of…
	The last parameter is the name of the virtual machine, used to track the running machines.
				* `-c` controls the number of virtual CPUs,
				* `-m` limits the amount of memory available to the guest,
				* `-t` defines which [.filename]#tap# device to use,
				* `-d` indicates which disk image to use,
				* `-i` tells bhyve to boot from the CD image instead of the disk, and
				* `-I` defines which CD image to use.

				The last parameter is the name of the virtual machine and used to track the running machines.
	This example starts the virtual machine in installation mode:			This example starts the virtual machine in installation mode:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-13.1-RELEASE-amd64-bootonly.iso guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-14.0-RELEASE-amd64-bootonly.iso guestname
				rgrimesUnsubmitted Done Inline Actions Is the "sh" even needed, this file should already be installed mode 755 so can be directly invoked. If its not installed 755 that should be fixed. rgrimes: Is the "sh" even needed, this file should already be installed mode 755 so can be directly…
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE at the moment, at least. I agree that it would be good to change that. I'll have a look at that for another review. freebsd_ny-central.org: Good catch. Unfortunately, it seems to be 444 right now. That's what it is on my 13.2-RELEASE…
	....			....

	The virtual machine will boot and start the installer.			The virtual machine will boot and start the installer.
	After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].			After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose btn:[Yes].

	Reboot the virtual machine.			Reboot the virtual machine.
	While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.			While rebooting the virtual machine causes bhyve to exit, the [.filename]#vmrun.sh# script runs `bhyve` in a loop and will automatically restart it.
	When this happens, choose the reboot option from the boot loader menu in order to escape the loop.			When this happens, choose the reboot option from the boot loader menu in order to escape the loop.
	Now the guest can be started from the virtual disk:			Now the guest can be started from the virtual disk:

	[source,shell]			[source,shell]
	....			....
	# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname			# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname
	....			....

	[[virtualization-bhyve-linux]]			[[virtualization-bhyve-linux]]
	=== Creating a Linux(R) Guest			=== Creating a Linux(R) Guest

	In order to boot operating systems other than FreeBSD, the package:sysutils/grub2-bhyve[] port must be first installed.			Linux guests can be booted either like any other regular crossref:virtualization[virtualization-bhyve-uefi,"UEFI-based guest"] virtual machine, or alternatively, you can make use of the package:sysutils/grub2-bhyve[] port.
				jrmUnsubmitted Done Inline Actions Grammarly tells me we also need a comma after `alternatively`. jrm: Grammarly tells me we also need a comma after `alternatively`.
				emasteUnsubmitted Done Inline Actions We don't need the (R) after the first use of a trademark. I know the existing text has it everywhere but no need to add more of them. emaste: We don't need the (R) after the first use of a trademark. I know the existing text has it…

	Next, create a file to use as the virtual disk for the guest machine:			To do this, first ensure that the port is installed, then create a file to use as the virtual disk for the guest machine:

	[source,shell]			[source,shell]
	....			....
	# truncate -s 16G linux.img			# truncate -s 16G linux.img
	....			....

	Starting a virtual machine with bhyve is a two step process.			Starting a Linux virtual machine with `grub2-bhyve` is a two step process.
	First a kernel must be loaded, then the guest can be started.
	The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].			. First a kernel must be loaded, then the guest can be started.
				jrmUnsubmitted Done Inline Actions I don't think this renders as you intended. Did you mean to make this an unordered list? jrm: I don't think this renders as you intended. Did you mean to make this an unordered list?
				. The Linux(R) kernel is loaded with package:sysutils/grub2-bhyve[].

	Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:			Create a [.filename]#device.map# that grub will use to map the virtual devices to the files on the host system:

	[.programlisting]			[.programlisting]
	....			....
	(hd0) ./linux.img			(hd0) ./linux.img
	(cd0) ./somelinux.iso			(cd0) ./somelinux.iso
	....			....

	▲ Show 20 Lines • Show All 71 Lines • ▼ Show 20 Lines

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=linuxguest			# bhyvectl --destroy --vm=linuxguest
	....			....

	[[virtualization-bhyve-uefi]]			[[virtualization-bhyve-uefi]]
	=== Booting bhyve Virtual Machines with UEFI Firmware			=== Booting bhyve Virtual Machines with UEFI Firmware

				rgrimesUnsubmitted Done Inline Actions strike userspace, its nonsensical in the context of firmware. rgrimes: strike userspace, its nonsensical in the context of firmware.
	In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.			In addition to `bhyveload` and `grub-bhyve`, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware.
	This option may support guest operating systems that are not supported by the other loaders.			This option may support guest operating systems that are not supported by the other loaders.

	In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.			In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images.
	This can be done by installing package:sysutils/bhyve-firmware[] port or package.			This can be done by installing package:sysutils/bhyve-firmware[] port or package.

	With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.			With the firmware in place, add the flags `-l bootrom,_/path/to/firmware_` to your bhyve command line.
	The actual bhyve command may look like this:			The actual bhyve command may look like this:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
	guest			guest
	....			....

				rgrimesUnsubmitted Done Inline Actions We should be consistent on EFI vs UEFI, use one or the other in all places please rgrimes: We should be consistent on EFI vs UEFI, use one or the other in all places please
				emasteUnsubmitted Done Inline Actions Heh, from efivar(8) efivar – UEFI environment variable interaction emaste: Heh, from efivar(8) efivar – UEFI environment variable interaction
				To allow a guest to store EFI variables, you can use a variables file appended to the `-l` flag. Note that bhyve will write guest modifications to the given variables file.
				Therefore, be sure to first create a per-guest-copy of the variables template file:

				[source,shell]
				....
				# cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /path/to/vm-image/BHYVE_UEFI_VARS.fd
				....

				Then, add that variables file into your bhyve arguments:

				[source,shell]
				....
				# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
				-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
				-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
				-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/path/to/vm-image/BHYVE_UEFI_VARS.fd \
				guest
				....

	package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:			package:sysutils/bhyve-firmware[] also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:

	[source,shell]			[source,shell]
	....			....
	# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \			# bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
	-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \			-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
	-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \			-s 4:0,ahci-cd,./install.iso -c 4 -m 1024M \
	-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \			-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
	Show All 39 Lines
	When starting the VM, specify the ZFS volume as the disk drive:			When starting the VM, specify the ZFS volume as the disk drive:

	[source,shell]			[source,shell]
	....			....
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0 \
	-l com1,stdio -c 4 -m 1024M linuxguest			-l com1,stdio -c 4 -m 1024M linuxguest
	....			....

				If you are using ZFS for the host as well as inside a guest, keep in mind the competing memory pressure of both systems caching the virtual machine's contents.
				jrmUnsubmitted Done Inline Actions Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache. jrm: Grammarly recommends that we hyphenate the adjective, so it's `metadata-only` cache.
				rgrimesUnsubmitted Done Inline Actions Also recommend the use of zvol's vs a file in a file system for the disk image. rgrimes: Also recommend the use of zvol's vs a file in a file system for the disk image.
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Good point - there is a statement to this a few lines up; in my previous diff, this wasn't visible because I missed the -U99999 initially on my patch. Apologies for that. freebsd_ny-central.org: Good point - there is a statement to this a few lines up; in my previous diff, this wasn't…
				To alleviate this, consider setting the host's ZFS filesystems to use metadata-only cache.
				To do this, apply the following settings to ZFS filesystems on the host, replacing `<name>` with the name of the specific zvol dataset name of the virtual machine.

				[source,shell]
				....
				# zfs set primarycache=metadata <name>
				....

	[[virtualization-bhyve-nmdm]]			[[virtualization-bhyve-nmdm]]
	=== Virtual Machine Consoles			=== Virtual Machine Consoles

	It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.			It is advantageous to wrap the bhyve console in a session management tool such as package:sysutils/tmux[] or package:sysutils/screen[] in order to detach and reattach to the console.
	It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.			It is also possible to have the console of bhyve be a null modem device that can be accessed with `cu`.
	To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.			To do this, load the [.filename]#nmdm# kernel module and replace `-l com1,stdio` with `-l com1,/dev/nmdm0A`.
	The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).			The [.filename]#/dev/nmdm# devices are created automatically as needed, where each is a pair, corresponding to the two ends of the null modem cable ([.filename]#/dev/nmdm0A# and [.filename]#/dev/nmdm0B#).
	See man:nmdm[4] for more information.			See man:nmdm[4] for more information.

	[source,shell]			[source,shell]
	....			....
	# kldload nmdm			# kldload nmdm
	# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \			# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./linux.img \
	-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest			-l com1,/dev/nmdm0A -c 4 -m 1024M linuxguest
	# cu -l /dev/nmdm0B			# cu -l /dev/nmdm0B
	Connected			Connected

	Ubuntu 13.10 handbook ttyS0			Ubuntu 13.10 handbook ttyS0

	handbook login:			handbook login:
	....			....

				rgrimesUnsubmitted Done Inline Actions I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not follow a newline you wont get the disconnect, or at least that is how it works every where else. rgrimes: I believe the correct and full disconnect sequence is Newline, tilde, dot. If your ~ does not…
				rgrimesUnsubmitted Done Inline Actions I do not see that this was incorporated??? rgrimes: I do not see that this was incorporated???
				freebsd_ny-central.orgAuthorUnsubmitted Done Inline Actions Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a dot list syntax mistake I made previously. I'll fix this right now and update the diff one more time. freebsd_ny-central.org: Uh, right - apologies. Thanks for the feedback. I didn't read that right and mixed it up with a…
				To disconnect from a console, press `~` followed by Ctrl+D or `.`. Keep in mind that only the connection is dropped while the login session remains active.
				Another user connecting to the same console could therefore make use of any active sessions without having to first authenticate.
				For security reasons, it's therefore recommended to logout before disconnecting.

				The number in the [.filename]#nmdm# device path must be unique for each virtual machine and must not be used by any other processes before bhyve starts.
				The number can be chosen arbitrarily and does not need to be taken from a consecutive sequence of numbers.
				The device node pair (i.e. [.filename]#/dev/nmdm0a# and [.filename]#/dev/nmdm0b#) are created dynamically when bhyve connects its console and destroyed when it shuts down.
				Keep this in mind when creating scripts to start your virtual machines: you need to make sure that all virtual machines are assigned unique [.filename]#nmdm# devices.

	[[virtualization-bhyve-managing]]			[[virtualization-bhyve-managing]]
	=== Managing Virtual Machines			=== Managing Virtual Machines

	A device node is created in [.filename]#/dev/vmm# for each virtual machine.			A device node is created in [.filename]#/dev/vmm# for each virtual machine.
	This allows the administrator to easily see a list of the running virtual machines:			This allows the administrator to easily see a list of the running virtual machines:

	[source,shell]			[source,shell]
	....			....
	# ls -al /dev/vmm			# ls -al /dev/vmm
	total 1			total 1
	dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./			dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./
	dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../			dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../
	crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname			crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname
	crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest			crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest
	crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest			crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest
	....			....

	A specified virtual machine can be destroyed using `bhyvectl`:			A specified virtual machine can be destroyed using `bhyvectl`:

	[source,shell]			[source,shell]
	....			....
	# bhyvectl --destroy --vm=guestname			# bhyvectl --destroy --vm=guestname
	....			....

				Destroying a virtual machine this way means killing it immediately. Any unsaved data will be lost, open files and filesystems may get corrupted.
				To gracefully shut down a virtual machine, you can instead send a `TERM` signal to its bhyve process. This triggers an ACPI shutdown event for the guest:

				[source,shell]
				....
				# ps ax \| grep bhyve
				17424 - SC 56:48.27 bhyve: guestvm (bhyve)
				# kill 17424
				....

				[[virtualization-tools-utilities]]
				=== Tools and Utilities

				There are numerous utilities and applications available in ports to help simplify setting up and managing bhyve virtual machines:

				.bhyve Managers
				[options="header", cols="1,1,1,1"]
				\|===
				\| Name \| License \| Package \| Documentation

				\| vm-bhyve
				\| BSD-2
				\| package:sysutils/vm-bhyve[]
				\| link:https://github.com/churchers/vm-bhyve[Documentation]

				\| CBSD
				\| BSD-2
				\| package:sysutils/cbsd[]
				\| link:https://www.bsdstore.ru[Documentation]

				\| Virt-Manager
				\| LGPL-3
				\| package:deskutils/virt-manager[]
				\| link:https://virt-manager.org/[Documentation]

				\| Bhyve RC Script
				\| Unknown
				\| package:sysutils/bhyve-rc[]
				\| link:https://www.freshports.org/sysutils/bhyve-rc/[Documentation]

				\| vmstated
				\| BSD-2
				\| package:sysutils/vmstated[]
				\| link:https://github.com/christian-moerz/vmstated[Documentation]

				\|===

	[[virtualization-bhyve-onboot]]			[[virtualization-bhyve-onboot]]
	=== Persistent Configuration			=== Persistent Configuration

	In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:			In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:

	[.procedure]			[.procedure]
	. [.filename]#/etc/sysctl.conf#			. [.filename]#/etc/sysctl.conf#
	+			+
	▲ Show 20 Lines • Show All 154 Lines • ▼ Show 20 Lines
	Unprivileged domains consist of a configuration file and virtual or physical hard disks.			Unprivileged domains consist of a configuration file and virtual or physical hard disks.
	Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].			Virtual disk storage for the DomU can be files created by man:truncate[1] or ZFS volumes as described in crossref:zfs[zfs-zfs-volume,“Creating and Destroying Volumes”].
	In this example, a 20 GB volume is used.			In this example, a 20 GB volume is used.
	A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.			A VM is created with the ZFS volume, a FreeBSD ISO image, 1 GB of RAM and two virtual CPUs.
	The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.			The ISO installation file is retrieved with man:fetch[1] and saved locally in a file called [.filename]#freebsd.iso#.

	[source,shell]			[source,shell]
	....			....
	# fetch https://download.freebsd.org/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-bootonly.iso -o freebsd.iso			# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso -o freebsd.iso
	....			....

	A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.			A ZFS volume of 20 GB called [.filename]#xendisk0# is created to serve as the disk space for the VM.

	[source,shell]			[source,shell]
	....			....
	# zfs create -V20G -o volmode=dev zroot/xendisk0			# zfs create -V20G -o volmode=dev zroot/xendisk0
	....			....
	▲ Show 20 Lines • Show All 133 Lines • Show Last 20 Lines

View Options

Flush remaining routes from the routing table during VNET shutdown.ClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 85362

sys/net/route.h

sys/net/route.c

sys/net/route/route_ctl.c

sys/netinet/ip_input.c

sys/netinet6/ip6_input.c

Flush remaining routes from the routing table during VNET shutdown.
ClosedPublic
Actions

Revision Contents
Changeset List