M1: CI Scripts for Developers

[bofh]

Summary

This is the first(1/6) installment of scripts of bringing CI for developers in the base tree. This patch brings the following changes:

• Changes in new ci directory:
  • Create a new directory ci where all ci related scripts/configs/Makefile will be kept
    • Create a firstboot_ci rc script which will change configurations or more likely enable/disable relevant tests after the testvm is booted. In reference to our jenkins scripts this replaces most of the replace/append of meta scripts.
    • Merge the original rc.local script from jenkins scripts
    • Adds multiple scripts mostly the test selections into `ci/scripts/test/subr' from our jenkins scripts
    • Adds multiple scripts mostly the run-* into `ci/scripts/test/subr' from our jenkins scripts. These scripts kickoffs the actual testing once the test vm boots into the testing environment.

• Changes in existing release directory:
  • Add 3 new vars in Makefile
    • WITH_COMPRESSED_CLOUDWARE: We will often need to share the same images in between jobs both locally and other CI environments where it is required to move the artifacts from one job to the cache host and download again in another job. This will help reduce the amounts to wait for transferring and also the required space in the cache server as there might be a limit on how much we can cache.
    • WITH_PACKAGESYSTEM: Currently we have tried to reproduce a distdir on different hosts so that we can use the same distdir on different jobs but the file flags and modes are hard to reproduce in between different jobs so it's easier to use packagesystem from the release system. However those artifacts are only used to create the bootonly, disc1 and dvd image and are not used at all in the VM images. This variable adds the option to use the packagesystem artifacts to create vm-images.
    • OBJDIR: As we will use packagesystem rather than installworld so we need to make sure that the mk-vmimage.sh and vmimage.subr knows about .OBJDIR.This variable is used to pass that to the other helper scripts.
  • Changes in Makefile.vm
    • Add new CLOUDWARE image CI. Initially I had a discussion with @manu about using BASIC-CI however their console requirements are different to us and hence I have created this new version of CLOUDWARE image.
    • Refactor the code to accommodate the new variables WITH_COMPRESSED_CLOUDWAREand WITH_PACKAGESYSTEM. Also utilize PACKAGESYSTEM_ARGS to pass the argument of OBJDIR to the other helper scripts.
  • Changes in scripts/mk-vmimage.sh
    • Add getopts arg P to accommodate passing the environment variable WITH_PACKAGESYSTEM
    • Add getopts arg D to accommodate passing the environment variable OBJDIR
  • Changes in new tools/ci.conf
    • Configuration file for the CI CLOUDWARE image.
  • Changes in tools/vmimage.subr
    • Based on the value of PACKAGESYSTEM modify vm_install_base to install from packagesystem artifacts; otherwise use the traditional installworld
    • During vm_extra_pkg_rmcache we need to mount and unmount devfs

Although mentioned we cannot yet issue the command make testvm and generate the images. These changes are trivial before we can go into that state.

Test Plan

In the host system we need linux support as we need to install some linux-* packages.

sysrc linux_enable=YES
service linux start

Then we can start build, packagesystem and create the image as following:

JFLAG=$(sysctl -n hw.ncpu)
make -j $JFLAG -DNO_CLEAN buildworld TARGET="amd64" TARGET_ARCH="amd64"
make -j $JFLAG -DNO_CLEAN buildkernel TARGET="amd64" TARGET_ARCH="amd64"

cd /usr/src/release
make clean
make -DNOPORTS -DNOSRC -DNODOC TARGET="amd64" TARGET_ARCH="amd64" packagesystem
make -DNO_PKG -DNOPORTS -DNOSRC -DNODOC -DWITH_COMPRESSED_CLOUDWARE -DWITH_CLOUDWARE -DWITH_PACKAGESYSTEM TARGET="amd64" TARGET_ARCH="amd64" CLOUDWARE="CI" cloudware-release

We should have a file ci.raw.xz in ${.OBJDIR}

[lwhsu]

Does using firstboot_ci guarantee it is the *last* thing in booting?

[emaste]

Context not available.

For any future updates to this review please generate diffs with full context, e.g. git diff -U999999 ... or git show -U999999 ...

[imp]

Can I share this with some folks that have expressed an interest in all things CI?

[lwhsu]

In D38815#884335, @imp wrote:

Can I share this with some folks that have expressed an interest in all things CI?

This is of course welcomed. We're seeking for comments and feedback from different CI setups and hope what in the base can be a shared foundation for them.

[manu]

release/Makefile.vm

33

I don't think that it's a correct descritpion, yes it's used by the CI at freedesktop and was made for it but it can be used for anything. It's just an image with ssh and serial setup

[bofh]

In D38815#883644, @lwhsu wrote:

Does using firstboot_ci guarantee it is the *last* thing in booting?

I need to recheck this. I am having a bad time running bhyve on 13.1 with a head guest due to zfs incompatibility.

[bofh]

In D38815#884334, @emaste wrote:

Context not available.

For any future updates to this review please generate diffs with full context, e.g. git diff -U999999 ... or git show -U999999 ...

Noted.

[bofh]

release/Makefile.vm

33

Yes. I just had a discussion with @lwhsu on today's CI meeting and will look into merging into one. I am just not sure whether if the additional packcages installed will create a regression for other CI tests. Because the rc scripts by default boots and starts the tests. Can you have a look at the release/tools/ci.conf file? I will try to optionize the usage of the rc script so that developers can run different tests of their choice or scripts.

[bofh]

release/scripts/mk-vmimage.sh

37

By the by this won't make into the commit. This is just for my help for debugging.

release/tools/ci.conf

29

@manu Here are the list of packages.

[emaste]

ci/scripts/test/rc/firstboot_ci

28

Where is $meta getting set?

ci/scripts/test/rc/rc.local

11

Would be nice to have a comment explaining these bits/settings

28

odd indentation in this block vs above

release/tools/vmimage.subr

54

strange indendation

[bofh]

@emaste I just figured out man style. Is it alright to use the editor configs?

ci/scripts/test/rc/firstboot_ci

28

Oops. It should be $metadir

ci/scripts/test/rc/rc.local

11

@lwhsu Can you advise? There were no comments on why we needed this.

28

Noted. Will fix.

[lwhsu]

ci/scripts/test/rc/rc.local

11

This is added for showing (hopefully) useful information on console on panic, along with the backtrace. reset is for terminating the bhyve vm, I'm not sure this is applicable for all the environments...

release/scripts/mk-vmimage.sh

3

Just noticed this, I think 2023 can be added.

4

And remove this one :)

[imp]

The size of this review has made it a little hard to review. Might make sense to make more, small reviews to help the turn-around time.

ci/scripts/test/rc/firstboot_ci

12

Maybe having a quick sentence here that this is for XXXX testing...

28

could this iterate over $metadir and do all the scripts it found?

ci/scripts/test/rc/rc.local

34

can drop mips64
And it might be good to have a Bugzilla bug and reference it here...

ci/scripts/test/subr/disable-notyet-tests.sh

3

This looks like it could be a series of function calls that do this disabling... would be less verbose and error prone to changes as we go through time.

ci/scripts/test/subr/run-kyua-ltp.sh

36

why not append to this file instead of doing the sed re-writing above?

release/tools/vmimage.subr

65

drop mergemaster reference.. and support... it's deprecated so we don't need it anymore, ,right?
I know it's there now...

[gjb]

Apologies for my silence on this review. I was a bit ... out when it was created, and did not see it until just today. I will try to get some comments in by the end of the week.

[zlei]

release/tools/ci.conf

80

For current and stable/13, by default net.add_addr_allfibs is 0.

Will this change be MFC'd into stable/12 ?

[emaste]

ci/scripts/test/rc/firstboot_ci

12

Yes a brief description of the purpose of this script would be useful

ci/scripts/test/rc/rc.local

34

Yes reference to a bug would be good. The issue is that our kernel doesn't know how to signal the shutdown?

38

odd shell style(9)?

ci/scripts/test/subr/disable-disks-tests.sh

3

It seems a bit strange to me that we have all of these tiny per-test files

ci/scripts/test/subr/run-kyua-ltp.sh

10–12

it seems a little strange to me that these dirs aren't created elsewhere.

ci/scripts/test/subr/run-kyua-tcttestsuite.sh

12–13

this seems a little magic, is the list of vtbd# devices described somewhere?

release/Makefile

36–37

It seems like these changes could be submitted separately in advance

[delphij]

ci/scripts/test/rc/firstboot_ci

29

(Minor issue) I think these should be installed to the image, instead of referencing /usr/src paths.

ci/scripts/test/subr/run-kyua-ltp.sh

41–46

Maybe use the output from mdconfig instead of hardcoding /dev/md0

ci/scripts/test/subr/run-kyua-tcttestsuite.sh

12

Can we use tmpfs for this? (it appears like we only need it as a scratch space and it's discarded afterward).

ci/scripts/test/subr/run-kyua-zfs.sh

12

Can we use tmpfs for this? (it appears like we only need it as a scratch space and it's discarded afterward).