Page MenuHomeFreeBSD
Differential D48382

release: add cloudware oracle targets to package and upload .oci files
Needs ReviewPublic
Actions

Authored by dch on Wed, Jan 8, 8:51 PM.
Tags
None
Referenced Files
F107268658: D48382.id.diff
Sat, Jan 11, 8:08 PM
Unknown Object (File)
Sat, Jan 11, 7:12 AM
Subscribers
emaste
imp
jrtc27
lwhsu

Details

Reviewers
cperciva
andrew
emaste
Summary

oracle upload script for releng tooling ready for review now.

  • added portinstall for qemu-img, curl
  • uses base tar, sed now, 2 dependencies less
  • waiting on https://reviews.freebsd.org/D48415 to switch to simpler emulators/qemu@tools flavour
  • tested through to Oracle boot, amd64 is 🍰 but arm64 only loads kernel and then WARNING! Trying to fire up the kernel, but no device tree blob found! which is probably not a releng issue

I will build 14.2-RELEASE + this patch to check arm64 issue.

Test Plan

This is how I use it. Needs the $ORACLE_PAR_URL to upload.

# as root
cd /usr/src
export SRCCONF=/dev/null
export SRC_ENV_CONF=/dev/null
make -j2C buildworld  TARGET_ARCH=amd64 TARGET=amd64 -s \
&& make -j2.0 buildkernel TARGET_ARCH=amd64 TARGET=amd64 KERNCONF=GENERIC -s \
&& cd ./release \
&& -j2.0 clean \
&& make  KERNCONF=GENERIC TARGET_ARCH=amd64 TARGET=amd64 \
   WITH_CLOUDWARE=yes \
   CLOUDWARE=ORACLE \
   ORACLE_PAR_URL=$(vault read -field=url secret/releng/upload) \
   -s clean cloudware-release cw-oracle-upload

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 61657
Build 58541: arc lint + arc unit

Event Timeline

dch created this revision.Wed, Jan 8, 8:51 PM
Herald added a reviewer: andrew. · View Herald TranscriptWed, Jan 8, 8:51 PM
Herald added a reviewer: andrew. · View Herald Transcript
Herald added subscribers: jrtc27, emaste, imp. · View Herald Transcript
dch requested review of this revision.Wed, Jan 8, 8:51 PM
Harbormaster completed remote builds in B61580: Diff 148942.Wed, Jan 8, 8:51 PM
dch edited the summary of this revision. (Show Details)Wed, Jan 8, 9:01 PM
dch edited the test plan for this revision. (Show Details)
dch added a reviewer: emaste.
dch updated this revision to Diff 148943.Wed, Jan 8, 9:08 PM

dont print out PAR_URL as info, its supposed to be a secret

Harbormaster completed remote builds in B61581: Diff 148943.Wed, Jan 8, 9:08 PM
cperciva added inline comments.Wed, Jan 8, 10:48 PM
release/Makefile.oracle
19

I don't think there's anything needing to be fixed here; we put staging directories under /usr/obj/.../release all the time.

24

Can't this just be done with sed?

32

Am I understanding this correctly to mean that our qcow2 support in mkimg doesn't do zstd compression?

Unless this is really critical, I'd prefer to omit the compression until it can be added to mkimg, rather than pulling in yet another tool from ports.

37

I would be very surprised if it didn't work with tarballs generated by bsdtar.

Also, s/OCI/Oracle/ ? Just to avoid confusion with OCI containers, which also use tarballs...

42

Don't do this; leave it for make clean to deal with.

53

Do we need to make any sort of API call to tell Oracle "hey we uploaded an image", or does anything uploaded to the magic URL get added to their listings?

release/scripts/oracle/amd64.shapeCompatibilities.json
1 ↗(On Diff #148943)

Deliberately empty file?

dch updated this revision to Diff 148969.Thu, Jan 9, 9:27 AM

re-push with missing commits

Harbormaster completed remote builds in B61595: Diff 148969.Thu, Jan 9, 9:27 AM
lwhsu added a subscriber: lwhsu.Thu, Jan 9, 10:32 AM
lwhsu added inline comments.
release/scripts/oracle/base.json
22 ↗(On Diff #148969)

extra white line?

release/scripts/oracle/imageCapabilityData.json
1 ↗(On Diff #148969)

is it possible/good to format this?

dch updated this revision to Diff 148980.Thu, Jan 9, 11:28 AM

add portinstall target

Harbormaster completed remote builds in B61597: Diff 148980.Thu, Jan 9, 11:28 AM
dch edited the summary of this revision. (Show Details)Thu, Jan 9, 11:30 AM
dch updated this revision to Diff 148986.Thu, Jan 9, 1:05 PM

switch to qemu-nox11 as tools flavor doesnt support zstd

raw 9.0GiB vs zip at 6.0GiB and zstd at 1.6GiB

Harbormaster completed remote builds in B61602: Diff 148986.Thu, Jan 9, 1:06 PM
dch marked 4 inline comments as done.Thu, Jan 9, 1:36 PM
dch added inline comments.
release/Makefile.oracle
24

phab seems to have got lines confused. Are you referring to the jq invocation?

TLDR yes; I wanted to get it tested & working with the tools I already use today, before swapping out lighter ones.

This would be a good next step.

32

its a massive difference - 9GiB/image raw -> 6GiB with mkimg, vs 1.6GiB with qemu native qcow2 & zstd. This cost is not just borne by my bucket where we upload the images, it flows down to every user who spins up the resulting image.

Alternatively I can do this manually (outside build process). But then we lose the nice chain of provenance that we have presently.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263029 for the mkimg compression.

37

ok I'll call them ORACLE_QCOW_LIST then. ORACLE_ORACLE_LIST feels weird.

bsdtar testing will be next week.

42

you mean modifying CLEANFILES+= ? will make clean remove the tmpdir as well? Or did you mean something else?

53

Sadly the magical bit is Dave clicking in their web pages, its around 30-60m to do.

Its not yet automatable but they promise me 2025Q1.

For the moment, I get a notification each time something is added to the bucket. Once their automation is available I can make this happen.

release/scripts/oracle/amd64.shapeCompatibilities.json
1 ↗(On Diff #148943)

yes, strictly its not JSON without [] or {}.

release/scripts/oracle/imageCapabilityData.json
1 ↗(On Diff #148969)

I vaguely recall trying this last year and IIRC it wasn't happy with the extra whitespace and new lines. I will check and update if possible.

cperciva added inline comments.Thu, Jan 9, 5:26 PM
release/Makefile.oracle
24

Yes, I was referring to replacing jq with sed so we don't need to pull in the extra build tool.

32

Ok, we can pull in qemu until this can be added to mkimg.

37

Sorry that's phabricator getting confused about lines. This was about the comment "# TODO does OCI only support GNU tar format".

42

CLEANDIRS += ${ORACLE_TMP} rather than the manual rm -rf.

53

Got it. If this is the best we can do, it's the best we can do. :-)

emaste added inline comments.Thu, Jan 9, 7:17 PM
release/Makefile.oracle
10

what do we need gtar for?

53

Got it. If this is the best we can do, it's the best we can do. :-)

Probably worth a comment to this effect in the src

dch updated this revision to Diff 149087.Fri, Jan 10, 6:55 PM
dch marked 3 inline comments as done.
  • switch out jq for sed
  • use qemu-nox11 temporarily, I sent bofh@ a patch to allow use to use qemu-tools instead
  • switch to bsd tar with gnutar format
  • by switching back to sed, we have to keep the JSON blobs in their raw unreadable format
  • remove the tmpdir cleanups, rely on CLEANDIRS for that

amd64 images seem to boot fine
arm64 loads kernel, but don't complete booting, likely not a build issue but a kernel one

Harbormaster completed remote builds in B61657: Diff 149087.Fri, Jan 10, 6:55 PM
dch edited the summary of this revision. (Show Details)Fri, Jan 10, 7:03 PM
dch updated this revision to Diff 149101.Fri, Jan 10, 10:58 PM

if GITREV is blank (e.g. during releases) then skip it and a preceding -

Harbormaster completed remote builds in B61664: Diff 149101.Fri, Jan 10, 10:58 PM
dch marked 11 inline comments as done.Fri, Jan 10, 11:12 PM

Is this enough for including as a block comment in the Makefile outlining the process?
I'm likely to write a blog post with more detail, I can include that future markdown
somehere in releng repo if thats useful.

# Overview:
#
# The base image is created by cloudware-release already.
# Construct the custom OCI metdata, derived by exporting official OCI images.
# It is architecture-specific but appears mostly stable over time.
# Compress the raw image and place it in the same directory as the metadata.
# Make a GNU format tarball of these files.
# Upload the tarball to Oracle Cloud via a pre-approved curl URI, into
# the FreeBSD Foundation's Oracle Cloud account.
#
# These images go into dch@ "re" bucket in us-ashburn-1 region, which
# is mounted into the FreeBSD Foundation Oracle Marketplace account.
# Once uploaded, a manual step is needed to import the images as local
# custom images. These can then be tested within the us-ashburn-1 region.
# Once tested, follow the manual Oracle Marketplace import process to
# create a new FreeBSD version, attach the images, and initiate validation
# by Oracle. This can take up to 5 working days. Once complete, a final
# manual step is needed to make the currently private images, public.
# Syncing to all sites should take 2-3 hours.
release/Makefile.oracle
10

bsd tar with --format=gnutar is adequate

32

waiting on ports commit + build of qemu-tools with zstd compression to

dch marked an inline comment as done.Fri, Jan 10, 11:43 PM

Revision Contents
Changeset List

Diff 149087

View Options

release/Makefile.oracle

Loading...
View Options

release/Makefile.vm

Loading...
View Options

release/scripts/oracle/arm64_shape_compatibilities.json

Loading...
View Options

release/scripts/oracle/default_shape_compatibilities.json

Loading...
View Options

release/scripts/oracle/image_capability_data.raw

Loading...
View Options

release/scripts/oracle/image_metadata.json

Loading...