build: skip the database check for the distributeworld target

[royger]

Summary

distributeworld is used to generate install media, so it makes no
sense to check the host database since the install media can be
generated from any box, regardless of the version of FreeBSD it's
running.

[• ian]

I'm not sure what led to putting these changes where they're at in the makefile, it seems a bit unrelated to the things immediately surrounding it. It should be noted that for this change to work properly, the new lines must appear before line 833 (pre-patched, or 838 with this patch included). It might make more sense to move this change to just before line 833; that would make it a lot more clear what the real effect of this change is.

Makefile.inc1

377 ↗

(On Diff #46018)

This comment focuses on the current pain and glosses over most of what DB_FROM_SRC actually implies. A comment that more directly indicates what's going on might be something like

# When generating install media, do not allow user and group information
# from the build host to affect the contents of the distribution.

[• ian]

Hrm, I didn't intend for my previous comments to include the "request changes" flag, don't consider my comments to be blocking, I was just throwing them out there to see what other people think.

[royger]

Change the wording of the comment and put it just before the first
usage of DB_FROM_SRC.

[• ian]

I'm 'accepting' this revision in the sense that I think this is the right way to achieve unconditional use of users and groups from the source being installed, for the distributeworld target. I can't say whether doing so is a good idea or not in general, because I don't know much about the distributeworld target.

If distributeworld is truly used only by the freebsd project to generate images for distribution media, then it probably makes sense. If the same target is used by end users to generate local images for site-wide distribution, then I can envision theoretical scenarios where this would be the wrong thing to do. For example, there could be a site-wide user/group database such as ldap, and that could theoretically differ in terms of name<->idnumber associations even for the 'standard' ids < 1000. A user doing something like that would see this as a regression, one that cannot be undone with local command line args or env vars. This might seem like a far-fetched scenario, but I've heard that the freebsd cluster machines do something like this.

In other words, maybe it would be good to get re@ and/or clusteradm@ to eyeball this too.

[royger]

I plan to commit this tomorrow unless there are any objections.

[royger]

So it seems there are still issues even after this fix, now I'm seeing:

sh /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/scripts/mm-mtree.sh -m /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/.. -F  "TARGET_ARCH=amd64 TARGET=amd64 "  -D "/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/dist/base"
*** Creating the temporary root environment in /var/tmp/temproot.GeKDEEWO
 *** /var/tmp/temproot.GeKDEEWO ready for use
 *** Creating and populating directory structure in /var/tmp/temproot.GeKDEEWO

cd /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/etc; MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE= CC="cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp -B/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd12.0 --sysroot=/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp -B/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd12.0 --sysroot=/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp -B/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/usr/bin"  AS="as" AR="ar" LD="ld" LLVM_LINK=""  NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" PATH=/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/legacy/usr/sbin:/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/legacy/usr/bin:/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/legacy/bin:/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/usr/sbin:/usr/home/osstest/build.125801.build-amd64-freebsd/obj/usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/amd64.amd64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin make    METALOG= distrib-dirs
for file in /usr/share/doc/usd/10.exref /usr/share/doc/usd/11.edit /usr/share/doc/usd/12.vi /usr/share/doc/usd/13.viref; do  if [ -f /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/dist/base/${file} ]; then  rm -f /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/dist/base/${file};  fi;  done
mtree -deU -i -f /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/etc/mtree/BSD.root.dist -p /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/dist/base/
mtree -deU -i -f /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/etc/mtree/BSD.var.dist -p /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/release/dist/base/var
mtree: unknown user `ntpd'
mtree: failed at line 49 of the specification
*** Error code 1

Stop.
make[3]: stopped in /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd/etc
*** Error code 1

Stop.
make[2]: stopped in /usr/home/osstest/build.125801.build-amd64-freebsd/freebsd
*** Error code 1

This is part of the 'ftp' target in the release directory that runs the release/scripts/mm-mtree.sh. I see that there are snapshots still being created without issues, so I wonder what runes are used inside of the cluster to generate the install media. Is something different from 'make -C release ftp' used to generate the install sets?

[gjb]

In D16507#353295, @royger wrote:

This is part of the 'ftp' target in the release directory that runs the release/scripts/mm-mtree.sh. I see that there are snapshots still being created without issues, so I wonder what runes are used inside of the cluster to generate the install media. Is something different from 'make -C release ftp' used to generate the install sets?

As mentioned out-of-band, I added the ntpd UID/GID to work around this issue.

[• ian]

In D16507#353334, @gjb wrote:

In D16507#353295, @royger wrote:

This is part of the 'ftp' target in the release directory that runs the release/scripts/mm-mtree.sh. I see that there are snapshots still being created without issues, so I wonder what runes are used inside of the cluster to generate the install media. Is something different from 'make -C release ftp' used to generate the install sets?

As mentioned out-of-band, I added the ntpd UID/GID to work around this issue.

Added it to what? IMO, it's a really bad idea to paper over this mess, it just means we have to suffer all this pain again next time a user is added to the base system.

[gjb]

In D16507#353346, @ian wrote:

In D16507#353334, @gjb wrote:

In D16507#353295, @royger wrote:

This is part of the 'ftp' target in the release directory that runs the release/scripts/mm-mtree.sh. I see that there are snapshots still being created without issues, so I wonder what runes are used inside of the cluster to generate the install media. Is something different from 'make -C release ftp' used to generate the install sets?

As mentioned out-of-band, I added the ntpd UID/GID to work around this issue.

Added it to what? IMO, it's a really bad idea to paper over this mess, it just means we have to suffer all this pain again next time a user is added to the base system.

Sorry for the incomplete statement. Added to the build host so 'installworld' to the target chroot directories did not error.

I could have just upgraded the build host, but chose to apply a band-aid for this one case.

[• ian]

In D16507#353355, @gjb wrote:

In D16507#353346, @ian wrote:

As mentioned out-of-band, I added the ntpd UID/GID to work around this issue.

Added it to what? IMO, it's a really bad idea to paper over this mess, it just means we have to suffer all this pain again next time a user is added to the base system.

Sorry for the incomplete statement. Added to the build host so 'installworld' to the target chroot directories did not error.

I could have just upgraded the build host, but chose to apply a band-aid for this one case.

Either one of those things is exactly what I mean by papering over the problem. Upgrading the build host is a way of saying that building images for a given version is only supported if the build host is running the version of the image you're trying to build. Is that a documented requirement of the release build process?

[gjb]

In D16507#353365, @ian wrote:

In D16507#353355, @gjb wrote:

In D16507#353346, @ian wrote:

As mentioned out-of-band, I added the ntpd UID/GID to work around this issue.

Added it to what? IMO, it's a really bad idea to paper over this mess, it just means we have to suffer all this pain again next time a user is added to the base system.

Sorry for the incomplete statement. Added to the build host so 'installworld' to the target chroot directories did not error.

I could have just upgraded the build host, but chose to apply a band-aid for this one case.

Either one of those things is exactly what I mean by papering over the problem. Upgrading the build host is a way of saying that building images for a given version is only supported if the build host is running the version of the image you're trying to build. Is that a documented requirement of the release build process?

No. In fact, it extremely rare for me to upgrade the builder during a release cycle, for example.

That said, I have run into this in the past, and had to use a similar band-aid.

[emaste]

Upgrading the build host is a way of saying that building images for a given version is only supported if the build host is running the version of the image you're trying to build.

Indeed, it may be the case that building on an older host often causes no trouble (i.e., because we don't frequently add new user IDs), but there's currently an implicit requirement. There are a few of these issues and they've existed for ages. We can address them as they're encountered, but we also ought to take a holistic look at the implicit dependencies.