add minimal OCI-compliant image builder

[dch]

Summary

First cut of an OCI release target for A1-Flex, the Ampere KVM-based VM
implementation in Oracle Cloud Infrastructure, based on 13.1-RELEASE.

• ✅ built with 13.1-RELEASE & this diff
• ✅ "works on my machine" (A1.Flex VM only)
• ✅ automated deployment steps to usable cloud-init image
• 💔 documented & automated Makefile.oci for re@ deployment
• ✅ cloudinit runs at startup
• ✅ cloudinit runs custom scripts (if provided)
• ✅ ssh-keys fetched via cloud-init
• ✅ ssh works with keys only (to a freebsd@ user)
• ✅ freebsd@ user can sudo to root
• ✅ root user disabled over ssh & console
• ✅ uses OCI ntp server
• 💔 linuxisms - cloudinit puts files into /var/lib & /run

I could do with some advice on what re@ needs to put this
into production. Any prior art for other clown providers?

Not planned in this release:

• zfs flavour
• support A1.BM (Altra bare metal instances)

Test Plan

Build & test via:

cd /usr/src \
    && sudo chflags -R noschg /usr/obj/*/*/arm64.aarch64/release/ \
    ;  sudo rm -rf /usr/obj/*/*/arm64.aarch64/release /tmp/FreeBSD-* /tmp/freebsd-arm64-schema.json \
    ;  make -j32 buildworld  TARGET_ARCH=aarch64 TARGET=arm64 -s \
    && make -j32 buildkernel TARGET_ARCH=aarch64 TARGET=arm64 KERNCONF=GENERIC -s \
    && cd ./release \
    && sudo make -j32 clean \
    && sudo make -DNOPORTS -DNOSRC \
      KERNCONF=GENERIC TARGET_ARCH=aarch64 TARGET=arm64 \
      WITH_CLOUDWARE=yes \
      CLOUDWARE=OCI -s cloudware-release \
    && upload-to-oci

• either follow manual steps per: https://people.freebsd.org/~dch/posts/2022-04-04-freebsd-oci.html

• or the automated fish script: https://gist.github.com/dch/b39090703558e5aa582423163141faaf

[dch]

add base64, minor tweaks

[dch]

• enable working cloudinit
• drop personal ssh keys
• leave pkg list untrimmed until further cloudinit testing is done

[gjb]

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

[dch]

In D34746#788403, @gjb wrote:

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

Absolutely! I'm still working on docs & testing myself.

[gjb]

In D34746#788407, @dch wrote:

In D34746#788403, @gjb wrote:

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

Absolutely! I'm still working on docs & testing myself.

Awesome. Thank you!

[dch]

• trim packages
• still needs a few TODOs to be pruned
• can we exclude ifconfig_DEFAULT=SYNCDHCP if cloudinit is enabled?

[dch]

tested more cloud-init functionality, LGTM

[dch]

• update for 13.1-RELEASE & 14.0-CURRENT
• update cloudinit test results

[emaste]

See also Oracle image guidelines at https://docs.oracle.com/en/cloud/marketplace/partner-portal/ocmpd/how-to-publish-an-oci-image-listing.pdf

[emaste]

release/Makefile.vm

39

Should we follow naming of other image types, ${OSRELEASE}.${OCI_FORMAT} or ${OSRELEASE}.oci.${OCI_FORMAT}

[gjb]

release/Makefile.vm

39

I think this does follow at least some of the naming schemes.

Notably, EC2_DISK=${OSRELEASE}.${EC2_FORMAT}, however GCE_DISK=disk.${GCE_FORMAT}.

I forget exactly why there was divergence there, to be honest.

185

I have not had a chance to take a very close look, but until these can be uncommented, I consider these two lines to be blockers.

There is a similar change in the pipeline to add back support for Azure, however one of the Azure provisioning scripts changes the hostname to 'localhost.localdomain', which I will not allow to be committed until that is resolved. Orthogonal, sure, but the point being, commented, not-yet-ready functionality is a no-go as far as I am concerned.

While these lines are commented, so should the 'OCI' addition to CLOUDWARE on line 22.

[emaste]

release/Makefile.vm

39

Huh, it looks like despite this we actually create oci.qcow2

185

I think it is reasonable to add oci.conf now, even if we don't have Makefile.oci yet. (I would not commit it commented out, but just omit it entirely.)

It is possible to use this infrastructure to build images for OCI even if the support for uploading them isn't available yet (it could be done via the OCI web interface in any case).

[emaste]

sudo make -DNOPORTS -DNOSRC \
        KERNCONF=GENERIC \
        WITH_CLOUDWARE=yes CLOUDWARE=OCI \
        -s cloudware-release \

this needs some TARGET and TARGET_ARCH?

[emaste]

And my attempt failed with:

===>  Installing for perl5-5.32.1_1
===>  Checking if perl5 is already installed
===>   Registering installation for perl5-5.32.1_1 as automatic
pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory
pkg-static: unknown keyword postexec: @postexec
pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory
pkg-static: unknown keyword postexec: @postexec
*** Error code 1

[emaste]

release/tools/oci.conf

76

OCI wants PermitRootLogin no (S14 in their requirements)

80

OCI also suggests UsePAM no but we might want to leave PAM enabled; it provides account and session processing for all auth types.

[emaste]

pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory

Solved by having a ports tree available.

[emaste]

release/tools/oci.conf

83

Perhaps pw -R ${DESTDIR} usermod root -w no

But that said, the cloud-init config currently installs the provided ssh key in /root/.ssh/authorized_keys.

[emaste]

+       # S14 Root user login must be disabled.
+       pw -R ${DESTDIR} usermod root -w no
+       cat <<-EOF >> ${DESTDIR}/usr/local/etc/cloud/cloud.cfg.d/98_oci.cfg
+    disable_root: true
+EOF
+

release/tools/oci.conf

83

Perhaps:

       # S14 Root user login must be disabled.
       pw -R ${DESTDIR} usermod root -w no
       cat <<-EOF >> ${DESTDIR}/usr/local/etc/cloud/cloud.cfg.d/98_oci.cfg
    disable_root: true
EOF

[emaste]

I built a freebsd/arm64 13.1 image for OCI using this oci.conf with the pw command and 98_oci.cfg file in the inline comment.

[dch]

release/Makefile.vm

39

hah I remember. GCE needs GNU tar/zip and also the image file had to be named disk.raw. Can't find the official docs but https://khushbu-parakh.medium.com/create-instances-in-google-cloud-with-private-image-328409916744 is what we need.

185

removed; I do have an upload script here that works, but it is missing the additional OCI Marketplace plumbing that Ed is still working through.

gjb: I'm happy to finish this, but I don't know what re@ needs wrt docs, usage - how you store the secrets for uploads etc, whether this runs in some CI or other build env?. Makefile.ec2 seems like a reasonable example to start from, once we have the Marketplace stuff working.

release/tools/oci.conf

83

I think we're ok here with the cloud-init line only.

[dch]

incorporate gjb@ & emaste@ feedback

[dch]

• oci: stop heredoc from doing expansion

[dch]

NB - still waiting on testing, and for build of devel/oci-cli to appear on backported quarterly branch before we can proceed.

[dch]

fix ntp.conf path

[dch]

fix egregious late-night-dev ntp.conf path

[dch]

looks like root without password from serial console works. we should fix this to comply.

[emaste]

In D34746#802372, @dch wrote:

looks like root without password from serial console works. we should fix this to comply.

I built with pw -R ${DESTDIR} usermod root -w no as in the above comment, which disables root passwd login.

[dch]

In D34746#802381, @emaste wrote:

In D34746#802372, @dch wrote:

looks like root without password from serial console works. we should fix this to comply.

I built with pw -R ${DESTDIR} usermod root -w no as in the above comment, which disables root passwd login.

added.

[dch]

final check testing in OCI VM

• rc.conf & loader.conf ok
• sshd_config settings ok
• firstboot scripts run
• cloudinit works
• root via console is blocked
• root via ssh is blocked
• freebsd user is created, gets ssh keys, and sudo
• ntp uses OCI servers

[emaste]

release/tools/oci.conf

8

should be in alpha order?

[dch]

release/tools/oci.conf

8

This is the placeholder for devel/oci-cli (the command line tool, that depends on py-oci as well) which I backported to quarterly last week. It's not yet gotten through the build pipeline:

pkg: No packages available to install matching 'devel/oci-cli' have been found in the repositories

[dch]

update since devel/oci-cli is in quarterly packages now

[dch]

prune egregious TODO

[dch]

do we need anything further to commit this yet?

[gjb]

In D34746#817400, @dch wrote:

do we need anything further to commit this yet?

Sorry for taking so long getting back to reviewing this. Please give me another day or two, as I had made some notes regarding this addition, but my notebooks recently rotated, so I need to find where I put them. Apologies.

[emaste]

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

[gjb]

In D34746#851841, @emaste wrote:

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

I thought we were waiting on API keys/credentials. Am I wrong in this regard?

In any case, provided OCI is not enabled by default in the CLOUDWARE list, approved.

release/Makefile.vm

22

See comment below, but please do not enable this by default yet.

[gjb]

release/Makefile.vm

22

To be more clear, do not enable this by default yet as I will need to add logic to thermite.sh to do the upload to OCI (with which, I will also need to have API keys/credentials for RE use).

[gjb]

In D34746#851847, @gjb wrote:

In D34746#851841, @emaste wrote:

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

I thought we were waiting on API keys/credentials. Am I wrong in this regard?

In any case, provided OCI is not enabled by default in the CLOUDWARE list, approved.

Apologies. Ed reminded me that upload operations are separete than building images, so you can keep the OCI image listed in the CLOUDWARE list.