Page MenuHomeFreeBSD
Differential D41340

release: Build UFS and ZFS VM images
ClosedPublic
Actions

Authored by cperciva on Aug 7 2023, 2:21 AM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Apr 14, 1:00 AM
Unknown Object (File)
Mar 10 2025, 8:23 PM
Unknown Object (File)
Mar 1 2025, 1:06 PM
Unknown Object (File)
Feb 22 2025, 12:06 AM
Unknown Object (File)
Feb 14 2025, 4:48 PM
Unknown Object (File)
Feb 13 2025, 2:58 PM
Unknown Object (File)
Jan 24 2025, 8:10 AM
Unknown Object (File)
Jan 11 2025, 3:18 PM
View All Files
Subscribers
emaste
gjb
imp
jrtc27

Details

Reviewers
markj
gjb
emaste
Group Reviewers
releng
Commits
rG32ae9a6b3937: release: Build UFS and ZFS VM images
Summary

A new option 'VMFSLIST' controls the list of filesystems for which VM
images will be built; the default is to build both UFS and ZFS.

The vm-install target installs these as ${OSRELEASE}-${FS}.${FORMAT},
e.g. FreeBSD-14.0-CURRENT-amd64-zfs.vmdk. For backwards compatibility,
the ${VMFS} image is linked to the previously used ${OSRELEASE}.${FORMAT}
name.

Cloudware building will be updated in a later commit.

Sponsored by: https://www.patreon.com/cperciva

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 53001
Build 49892: arc lint + arc unit

Event Timeline

cperciva created this revision.Aug 7 2023, 2:21 AM
Herald added subscribers: jrtc27, imp. · View Herald TranscriptAug 7 2023, 2:21 AM
cperciva requested review of this revision.Aug 7 2023, 2:21 AM
Harbormaster completed remote builds in B53001: Diff 125647.Aug 7 2023, 2:21 AM
cperciva added a child revision: D41341: release: Build cloudware with multiple FS types.Aug 7 2023, 2:21 AM
cperciva added a reviewer: markj.Aug 7 2023, 2:22 AM
emaste added a subscriber: emaste.Aug 7 2023, 3:18 AM
emaste added inline comments.
release/Makefile.vm
11

Should we just get rid of VMFS (perhaps after a short period)? Or is the intent that we'll want to continue offering an image name that does not specify the fs type (and could perhaps switch from UFS to ZFS in the future)?

cperciva added inline comments.Aug 7 2023, 3:25 AM
release/Makefile.vm
11

Yes, in the long term we should probably get rid of VMFS. I was leaving it there for backwards-compatibility purposes in case people have scripts which grab e.g. FreeBSD-14.0-CURRENT-amd64.vhd from ftp-master. It's also used as a default filesystem for the ec2ami target (for people who want to build a single AMI) but that could change to an EC2_FS_DEFAULT option or something like that.

markj added a comment.Aug 7 2023, 1:51 PM

This diff should update release.7.

release/Makefile.vm
11

VMFS is only present in main, so I don't know that there's that much of a backward compatibility concern?

Alternately, just extend the meaning of VMFS, so that setting VMFS=zfs will do what it did before, and VMFS=ufs zfs now does what you want VMFSLIST=ufs zfs to do. Though, that becomes a bit confusing if you only embed the FS name when multiple filesystems are specified. So on second thought, the separate VMFSLIST variable seems better.

179
.if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES)
COMPSUFF=.xz
.endif

.for FORMAT in ${VMFORMATS}
    ln -f ${DESTDIR}/vmimages/${OSRELEASE}-${VMFS}.${FORMAT}{COMPSUFF} ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}${COMPSUFF}
.endfor

just to avoid some of this duplication.

gjb accepted this revision.Aug 9 2023, 4:00 PM
gjb added a subscriber: gjb.

Note, parts of release/Makefile.mirrors will need to change as well, notably in 'vm-stage-images'. I will take a look at this during the week. Though, I think your compatibility symlinks resolves the immediate thing regarding the 'ftp-stage' target I am thinking of.

This revision is now accepted and ready to land.Aug 9 2023, 4:00 PM
cperciva updated this revision to Diff 125782.Aug 9 2023, 8:31 PM

Update release.7.

This revision now requires review to proceed.Aug 9 2023, 8:31 PM
Harbormaster completed remote builds in B53021: Diff 125782.Aug 9 2023, 8:31 PM
markj accepted this revision.Aug 9 2023, 8:35 PM
This revision is now accepted and ready to land.Aug 9 2023, 8:35 PM
cperciva added a comment.Aug 9 2023, 8:50 PM

Note, parts of release/Makefile.mirrors will need to change as well, notably in 'vm-stage-images'. I will take a look at this during the week. Though, I think your compatibility symlinks resolves the immediate thing regarding the 'ftp-stage' target I am thinking of.

Note: Hard links, not symlinks. Yes, I wasn't sure what to do with Makefile.mirrors but I hoped that the compatibility links would result in things continuing to work (with the single UFS image) for now.

release/Makefile.vm
179

@markj Are you ok with me leaving this as it is for now? In the long term we're going to be getting rid of the compatibility links anyway.

emaste accepted this revision.Aug 9 2023, 8:53 PM
emaste added inline comments.
release/Makefile.vm
173–179

can we just move this inside the existing . if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES) block?

markj added inline comments.Aug 9 2023, 9:07 PM
release/Makefile.vm
179

I'm ok with leaving it as it is.

I don't quite understand Ed's suggestion: we have a loop for both the compressed and uncompressed cases.

cperciva updated this revision to Diff 125787.Aug 9 2023, 9:10 PM

Fix missing filesystem loop around xz command.

This revision now requires review to proceed.Aug 9 2023, 9:10 PM
Harbormaster completed remote builds in B53023: Diff 125787.Aug 9 2023, 9:10 PM
cperciva added inline comments.Aug 9 2023, 9:14 PM
release/Makefile.vm
173–179

Not really; the ln -f for compressed images could be merged but we would still need an "else for FORMAT in $VMFORMATS" so I think it would be even more confusing.

The if/for logic could be deduplicated a bit here, but not completely.

gjb accepted this revision.Aug 9 2023, 10:00 PM
This revision is now accepted and ready to land.Aug 9 2023, 10:00 PM
gjb added a comment.Aug 9 2023, 10:01 PM
In D41340#942620, @cperciva wrote:

Note, parts of release/Makefile.mirrors will need to change as well, notably in 'vm-stage-images'. I will take a look at this during the week. Though, I think your compatibility symlinks resolves the immediate thing regarding the 'ftp-stage' target I am thinking of.

Note: Hard links, not symlinks. Yes, I wasn't sure what to do with Makefile.mirrors but I hoped that the compatibility links would result in things continuing to work (with the single UFS image) for now.

Right. Thank you for the correction. That is indeed what I meant.

Closed by commit rG32ae9a6b3937: release: Build UFS and ZFS VM images (authored by cperciva). · Explain WhyAug 9 2023, 10:45 PM
This revision was automatically updated to reflect the committed changes.
cperciva added a commit: rG32ae9a6b3937: release: Build UFS and ZFS VM images.

Revision Contents
Changeset List

PathSize
release/
31 lines

Diff 125647

View Options

release/Makefile.vm

# #
# $FreeBSD$ # $FreeBSD$
# #
# #
# Makefile for building virtual machine and cloud provider disk images. # Makefile for building virtual machine and cloud provider disk images.
# #
VMTARGETS= vm-image VMTARGETS= vm-image
VMFORMATS?= vhd vmdk qcow2 raw VMFORMATS?= vhd vmdk qcow2 raw
VMSIZE?= 5120m VMSIZE?= 5120m
VMFS?= ufs VMFS?= ufs
emasteUnsubmitted
Not Done
Inline Actions

Should we just get rid of VMFS (perhaps after a short period)? Or is the intent that we'll want to continue offering an image name that does not specify the fs type (and could perhaps switch from UFS to ZFS in the future)?

emaste: Should we just get rid of `VMFS` (perhaps after a short period)? Or is the intent that we'll…
cpercivaAuthorUnsubmitted
Done
Inline Actions

Yes, in the long term we should probably get rid of VMFS. I was leaving it there for backwards-compatibility purposes in case people have scripts which grab e.g. FreeBSD-14.0-CURRENT-amd64.vhd from ftp-master. It's also used as a default filesystem for the ec2ami target (for people who want to build a single AMI) but that could change to an EC2_FS_DEFAULT option or something like that.

cperciva: Yes, in the long term we should probably get rid of `VMFS`. I was leaving it there for…
markjUnsubmitted
Not Done
Inline Actions

VMFS is only present in main, so I don't know that there's that much of a backward compatibility concern?

Alternately, just extend the meaning of VMFS, so that setting VMFS=zfs will do what it did before, and VMFS=ufs zfs now does what you want VMFSLIST=ufs zfs to do. Though, that becomes a bit confusing if you only embed the FS name when multiple filesystems are specified. So on second thought, the separate VMFSLIST variable seems better.

markj: VMFS is only present in `main`, so I don't know that there's that much of a backward…
VMFSLIST?= ufs zfs
SWAPSIZE?= 1g SWAPSIZE?= 1g
VMBASE?= vm VMBASE?= vm
VHD_DESC= Azure, VirtualPC, Hyper-V, Xen disk image VHD_DESC= Azure, VirtualPC, Hyper-V, Xen disk image
VMDK_DESC= VMWare, VirtualBox disk image VMDK_DESC= VMWare, VirtualBox disk image
QCOW2_DESC= Qemu, KVM disk image QCOW2_DESC= Qemu, KVM disk image
RAW_DESC= Unformatted raw disk image RAW_DESC= Unformatted raw disk image
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Lines.endif
touch ${.TARGET} touch ${.TARGET}
. endfor . endfor
.endif .endif
.if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES) .if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES)
CLEANDIRS+= ${VMTARGETS} CLEANDIRS+= ${VMTARGETS}
. for FORMAT in ${VMFORMATS} . for FORMAT in ${VMFORMATS}
CLEANFILES+= ${FORMAT}.img . for FS in ${VMFSLIST}
CLEANFILES+= ${VMBASE}.${FORMAT} CLEANFILES+= ${FORMAT}.${FS}.img
CLEANFILES+= ${VMBASE}.${FS}.${FORMAT}
. endfor . endfor
. endfor
.endif .endif
vm-base: vm-image vm-base: vm-image
vm-image: vm-image:
.if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES) .if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES)
. for FORMAT in ${VMFORMATS} . for FORMAT in ${VMFORMATS}
. for FS in ${VMFSLIST}
mkdir -p ${.OBJDIR}/${.TARGET} mkdir -p ${.OBJDIR}/${.TARGET}
env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} SWAPSIZE=${SWAPSIZE} \ env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} SWAPSIZE=${SWAPSIZE} \
${.CURDIR}/scripts/mk-vmimage.sh \ ${.CURDIR}/scripts/mk-vmimage.sh \
-C ${.CURDIR}/tools/vmimage.subr -d ${.OBJDIR}/${.TARGET} -F ${VMFS} \ -C ${.CURDIR}/tools/vmimage.subr -d ${.OBJDIR}/${.TARGET} -F ${FS} \
-i ${.OBJDIR}/${FORMAT}.img -s ${VMSIZE} -f ${FORMAT} \ -i ${.OBJDIR}/${FORMAT}.${FS}.img -s ${VMSIZE} -f ${FORMAT} \
-S ${WORLDDIR} -o ${.OBJDIR}/${VMBASE}.${FORMAT} -S ${WORLDDIR} -o ${.OBJDIR}/${VMBASE}.${FS}.${FORMAT}
. endfor . endfor
. endfor
.endif .endif
touch ${.TARGET} touch ${.TARGET}
vm-cloudware: ${CLOUDTARGETS} vm-cloudware: ${CLOUDTARGETS}
list-vmtargets: list-cloudware list-vmtargets: list-cloudware
@${ECHO} @${ECHO}
@${ECHO} "Supported virtual machine disk image formats:" @${ECHO} "Supported virtual machine disk image formats:"
Show All 9 Lines. for _CW in ${CLOUDWARE}
@${ECHO} " ${_CW:tu}: ${${_CW:tu}_DESC}" @${ECHO} " ${_CW:tu}: ${${_CW:tu}_DESC}"
. endfor . endfor
.endif .endif
vm-install: vm-install:
.if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES) .if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES)
mkdir -p ${DESTDIR}/vmimages mkdir -p ${DESTDIR}/vmimages
. for FORMAT in ${VMFORMATS} . for FORMAT in ${VMFORMATS}
cp -p ${VMBASE}.${FORMAT} \ . for FS in ${VMFSLIST}
${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT} cp -p ${VMBASE}.${FS}.${FORMAT} \
${DESTDIR}/vmimages/${OSRELEASE}-${FS}.${FORMAT}
. endfor . endfor
. endfor
. if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES) . if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES)
. for FORMAT in ${VMFORMATS} . for FORMAT in ${VMFORMATS}
# Don't keep the originals. There is a copy in ${.OBJDIR} if needed. # Don't keep the originals. There is a copy in ${.OBJDIR} if needed.
${XZ_CMD} ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT} ${XZ_CMD} ${DESTDIR}/vmimages/${OSRELEASE}-${FS}.${FORMAT}
. endfor . endfor
. endif . endif
# Compatibility hardlinks from "foo.bar" to "foo-${VMFS}.bar".
. for FORMAT in ${VMFORMATS}
. if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES)
ln -f ${DESTDIR}/vmimages/${OSRELEASE}-${VMFS}.${FORMAT}.xz ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}.xz
. else
ln -f ${DESTDIR}/vmimages/${OSRELEASE}-${VMFS}.${FORMAT} ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}
. endif
markjUnsubmitted
Not Done
Inline Actions
.if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES)
COMPSUFF=.xz
.endif

.for FORMAT in ${VMFORMATS}
    ln -f ${DESTDIR}/vmimages/${OSRELEASE}-${VMFS}.${FORMAT}{COMPSUFF} ${DESTDIR}/vmimages/${OSRELEASE}.${FORMAT}${COMPSUFF}
.endfor

just to avoid some of this duplication.

markj: ``` .if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES) COMPSUFF=.xz .
cpercivaAuthorUnsubmitted
Done
Inline Actions

@markj Are you ok with me leaving this as it is for now? In the long term we're going to be getting rid of the compatibility links anyway.

cperciva: @markj Are you ok with me leaving this as it is for now? In the long term we're going to be…
markjUnsubmitted
Not Done
Inline Actions

I'm ok with leaving it as it is.

I don't quite understand Ed's suggestion: we have a loop for both the compressed and uncompressed cases.

markj: I'm ok with leaving it as it is. I don't quite understand Ed's suggestion: we have a loop for…
emasteUnsubmitted
Not Done
Inline Actions

can we just move this inside the existing . if defined(WITH_COMPRESSED_VMIMAGES) && !empty(WITH_COMPRESSED_VMIMAGES) block?

emaste: can we just move this inside the existing `. if defined(WITH_COMPRESSED_VMIMAGES) && !empty…
cpercivaAuthorUnsubmitted
Done
Inline Actions

Not really; the ln -f for compressed images could be merged but we would still need an "else for FORMAT in $VMFORMATS" so I think it would be even more confusing.

The if/for logic could be deduplicated a bit here, but not completely.

cperciva: Not really; the `ln -f` for compressed images could be merged but we would still need an "else…
. endfor
cd ${DESTDIR}/vmimages && sha512 ${OSRELEASE}* > \ cd ${DESTDIR}/vmimages && sha512 ${OSRELEASE}* > \
${DESTDIR}/vmimages/CHECKSUM.SHA512 ${DESTDIR}/vmimages/CHECKSUM.SHA512
cd ${DESTDIR}/vmimages && sha256 ${OSRELEASE}* > \ cd ${DESTDIR}/vmimages && sha256 ${OSRELEASE}* > \
${DESTDIR}/vmimages/CHECKSUM.SHA256 ${DESTDIR}/vmimages/CHECKSUM.SHA256
.endif .endif
vm-release: vm-release:
.if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES) .if defined(WITH_VMIMAGES) && !empty(WITH_VMIMAGES)
Show All 14 Lines