Page MenuHomeFreeBSD

add minimal OCI-compliant image builder
ClosedPublic

Authored by dch on Apr 2 2022, 9:38 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Dec 21, 3:54 PM
Unknown Object (File)
Thu, Dec 19, 1:01 PM
Unknown Object (File)
Thu, Dec 19, 7:16 AM
Unknown Object (File)
Mon, Dec 16, 11:54 PM
Unknown Object (File)
Mon, Dec 16, 6:29 AM
Unknown Object (File)
Fri, Dec 13, 5:11 AM
Unknown Object (File)
Thu, Dec 12, 8:09 AM
Unknown Object (File)
Tue, Dec 3, 10:35 PM
Subscribers

Details

Summary

First cut of an OCI release target for A1-Flex, the Ampere KVM-based VM
implementation in Oracle Cloud Infrastructure, based on 13.1-RELEASE.

  • โœ… built with 13.1-RELEASE & this diff
  • โœ… "works on my machine" (A1.Flex VM only)
  • โœ… automated deployment steps to usable cloud-init image
  • ๐Ÿ’” documented & automated Makefile.oci for re@ deployment
  • โœ… cloudinit runs at startup
  • โœ… cloudinit runs custom scripts (if provided)
  • โœ… ssh-keys fetched via cloud-init
  • โœ… ssh works with keys only (to a freebsd@ user)
  • โœ… freebsd@ user can sudo to root
  • โœ… root user disabled over ssh & console
  • โœ… uses OCI ntp server
  • ๐Ÿ’” linuxisms - cloudinit puts files into /var/lib & /run

I could do with some advice on what re@ needs to put this
into production. Any prior art for other clown providers?

Not planned in this release:

  • zfs flavour
  • support A1.BM (Altra bare metal instances)
Test Plan

Build & test via:

cd /usr/src \
    && sudo chflags -R noschg /usr/obj/*/*/arm64.aarch64/release/ \
    ;  sudo rm -rf /usr/obj/*/*/arm64.aarch64/release /tmp/FreeBSD-* /tmp/freebsd-arm64-schema.json \
    ;  make -j32 buildworld  TARGET_ARCH=aarch64 TARGET=arm64 -s \
    && make -j32 buildkernel TARGET_ARCH=aarch64 TARGET=arm64 KERNCONF=GENERIC -s \
    && cd ./release \
    && sudo make -j32 clean \
    && sudo make -DNOPORTS -DNOSRC \
      KERNCONF=GENERIC TARGET_ARCH=aarch64 TARGET=arm64 \
      WITH_CLOUDWARE=yes \
      CLOUDWARE=OCI -s cloudware-release \
    && upload-to-oci

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 45824
Build 42712: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
  • enable working cloudinit
  • drop personal ssh keys
  • leave pkg list untrimmed until further cloudinit testing is done

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

dch marked 2 inline comments as done.
In D34746#788403, @gjb wrote:

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

Absolutely! I'm still working on docs & testing myself.

In D34746#788407, @dch wrote:
In D34746#788403, @gjb wrote:

It is entirely too late for this to be added for 13.1.

Let's wait until 13.1 is done, so I can properly allocate the time to test.

Absolutely! I'm still working on docs & testing myself.

Awesome. Thank you!

  • trim packages
  • still needs a few TODOs to be pruned
  • can we exclude ifconfig_DEFAULT=SYNCDHCP if cloudinit is enabled?

tested more cloud-init functionality, LGTM

dch updated this revision to Diff 105689.EditedMay 4 2022, 8:52 PM
  • update for 13.1-RELEASE & 14.0-CURRENT
  • update cloudinit test results
dch edited the test plan for this revision. (Show Details)
release/Makefile.vm
39

Should we follow naming of other image types, ${OSRELEASE}.${OCI_FORMAT} or ${OSRELEASE}.oci.${OCI_FORMAT}

release/Makefile.vm
39

I think this does follow at least some of the naming schemes.

Notably, EC2_DISK=${OSRELEASE}.${EC2_FORMAT}, however GCE_DISK=disk.${GCE_FORMAT}.

I forget exactly why there was divergence there, to be honest.

185

I have not had a chance to take a very close look, but until these can be uncommented, I consider these two lines to be blockers.

There is a similar change in the pipeline to add back support for Azure, however one of the Azure provisioning scripts changes the hostname to 'localhost.localdomain', which I will not allow to be committed until that is resolved. Orthogonal, sure, but the point being, commented, not-yet-ready functionality is a no-go as far as I am concerned.

While these lines are commented, so should the 'OCI' addition to CLOUDWARE on line 22.

release/Makefile.vm
39

Huh, it looks like despite this we actually create oci.qcow2

185

I think it is reasonable to add oci.conf now, even if we don't have Makefile.oci yet. (I would not commit it commented out, but just omit it entirely.)

It is possible to use this infrastructure to build images for OCI even if the support for uploading them isn't available yet (it could be done via the OCI web interface in any case).

sudo make -DNOPORTS -DNOSRC \
        KERNCONF=GENERIC \
        WITH_CLOUDWARE=yes CLOUDWARE=OCI \
        -s cloudware-release \

this needs some TARGET and TARGET_ARCH?

And my attempt failed with:

===>  Installing for perl5-5.32.1_1
===>  Checking if perl5 is already installed
===>   Registering installation for perl5-5.32.1_1 as automatic
pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory
pkg-static: unknown keyword postexec: @postexec
pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory
pkg-static: unknown keyword postexec: @postexec
*** Error code 1
release/tools/oci.conf
76

OCI wants PermitRootLogin no (S14 in their requirements)

80

OCI also suggests UsePAM no but we might want to leave PAM enabled; it provides account and session processing for all auth types.

pkg-static: cannot load keyword from /usr/ports/Keywords/postexec.ucl: No such file or directory

Solved by having a ports tree available.

release/tools/oci.conf
83

Perhaps pw -R ${DESTDIR} usermod root -w no

But that said, the cloud-init config currently installs the provided ssh key in /root/.ssh/authorized_keys.

+       # S14 Root user login must be disabled.
+       pw -R ${DESTDIR} usermod root -w no
+       cat <<-EOF >> ${DESTDIR}/usr/local/etc/cloud/cloud.cfg.d/98_oci.cfg
+    disable_root: true
+EOF
+
release/tools/oci.conf
83

Perhaps:

       # S14 Root user login must be disabled.
       pw -R ${DESTDIR} usermod root -w no
       cat <<-EOF >> ${DESTDIR}/usr/local/etc/cloud/cloud.cfg.d/98_oci.cfg
    disable_root: true
EOF

I built a freebsd/arm64 13.1 image for OCI using this oci.conf with the pw command and 98_oci.cfg file in the inline comment.

dch marked 9 inline comments as done.Jun 2 2022, 8:40 PM
dch added inline comments.
release/Makefile.vm
39

hah I remember. GCE needs GNU tar/zip and also the image file had to be named disk.raw. Can't find the official docs but https://khushbu-parakh.medium.com/create-instances-in-google-cloud-with-private-image-328409916744 is what we need.

185

removed; I do have an upload script here that works, but it is missing the additional OCI Marketplace plumbing that Ed is still working through.

gjb: I'm happy to finish this, but I don't know what re@ needs wrt docs, usage - how you store the secrets for uploads etc, whether this runs in some CI or other build env?. Makefile.ec2 seems like a reasonable example to start from, once we have the Marketplace stuff working.

release/tools/oci.conf
83

I think we're ok here with the cloud-init line only.

dch edited the summary of this revision. (Show Details)
dch edited the test plan for this revision. (Show Details)

incorporate gjb@ & emaste@ feedback

  • oci: stop heredoc from doing expansion

NB - still waiting on testing, and for build of devel/oci-cli to appear on backported quarterly branch before we can proceed.

fix egregious late-night-dev ntp.conf path

looks like root without password from serial console works. we should fix this to comply.

In D34746#802372, @dch wrote:

looks like root without password from serial console works. we should fix this to comply.

I built with pw -R ${DESTDIR} usermod root -w no as in the above comment, which disables root passwd login.

In D34746#802372, @dch wrote:

looks like root without password from serial console works. we should fix this to comply.

I built with pw -R ${DESTDIR} usermod root -w no as in the above comment, which disables root passwd login.

added.

dch edited the test plan for this revision. (Show Details)
dch edited the test plan for this revision. (Show Details)

final check testing in OCI VM

  • rc.conf & loader.conf ok
  • sshd_config settings ok
  • firstboot scripts run
  • cloudinit works
  • root via console is blocked
  • root via ssh is blocked
  • freebsd user is created, gets ssh keys, and sudo
  • ntp uses OCI servers
This revision is now accepted and ready to land.Jun 3 2022, 8:27 PM
release/tools/oci.conf
8

should be in alpha order?

release/tools/oci.conf
8

This is the placeholder for devel/oci-cli (the command line tool, that depends on py-oci as well) which I backported to quarterly last week. It's not yet gotten through the build pipeline:

pkg: No packages available to install matching 'devel/oci-cli' have been found in the repositories

update since devel/oci-cli is in quarterly packages now

This revision now requires review to proceed.Jun 9 2022, 5:54 PM
This revision is now accepted and ready to land.Jun 9 2022, 6:00 PM
This revision now requires review to proceed.Jun 9 2022, 6:33 PM
dch marked an inline comment as done.Jul 31 2022, 11:10 AM

do we need anything further to commit this yet?

In D34746#817400, @dch wrote:

do we need anything further to commit this yet?

Sorry for taking so long getting back to reviewing this. Please give me another day or two, as I had made some notes regarding this addition, but my notebooks recently rotated, so I need to find where I put them. Apologies.

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

I thought we were waiting on API keys/credentials. Am I wrong in this regard?

In any case, provided OCI is not enabled by default in the CLOUDWARE list, approved.

release/Makefile.vm
22

See comment below, but please do not enable this by default yet.

This revision is now accepted and ready to land.Nov 22 2022, 7:28 PM
release/Makefile.vm
22

To be more clear, do not enable this by default yet as I will need to add logic to thermite.sh to do the upload to OCI (with which, I will also need to have API keys/credentials for RE use).

In D34746#851847, @gjb wrote:

@gjb can we get this approved? (This is what was used to generate the 13.1 image available in Oracle Cloud.)

I thought we were waiting on API keys/credentials. Am I wrong in this regard?

In any case, provided OCI is not enabled by default in the CLOUDWARE list, approved.

Apologies. Ed reminded me that upload operations are separete than building images, so you can keep the OCI image listed in the CLOUDWARE list.

This revision was automatically updated to reflect the committed changes.