Enable AUTO_OBJ by default if the OBJDIR is writable, only for in-tree builds.

[bdrewery]

Summary

This can be disabled by putting WITHOUT_AUTO_OBJ=yes in /etc/src-env.conf, not
/etc/src.conf, or passing it in the environment.

The purpose of this rather than simply flipping the default of AUTO_OBJ to yes
is to avoid hassling users with auto.obj.mk failures if the wanted OBJDIR is
not writable. It will fallback to writing to the source directory like it does
today if MAKEOBJDIRPREFIX is not writable.

The act of enabling MK_AUTO_OBJ disables all 'make obj' treewalks since
previous work has made those not run if MK_AUTO_OBJ==yes in Makefile.inc1.

Relnotes: yes
Sponsored by: Dell EMC Isilon

Test Plan

/root/git/freebsd2/bin/sh % id
uid=1003(bryan) gid=1003(bryan) groups=1003(bryan),5(operator)
/root/git/freebsd2/bin/sh % mkdir -p /usr/obj/root/git/freebsd2/bin/sh
mkdir: /usr/obj/root/git/freebsd2: Permission denied

# This user has no write access to the default wanted OBJDIR
# so it defaults to the current directory as normal.

/root/git/freebsd2/bin/sh % make -V .OBJDIR
/root/git/freebsd2/bin/sh
% make
Warning: Object directory not changed from original /root/git/freebsd2/bin/sh
...

# Providing a MAKEOBJDIRPREFIX that the user can write to enables
# AUTO_OBJ automatically.

/root/git/freebsd2/bin/sh % rm -rf /tmp/root
/root/git/freebsd2/bin/sh % MAKEOBJDIRPREFIX=/tmp make -V .OBJDIR
[Creating objdir /tmp/root/git/freebsd2/amd64.amd64/bin/sh...]
/tmp/root/git/freebsd2/amd64.amd64/bin/sh
/root/git/freebsd2/bin/sh % rm -rf /tmp/root
/root/git/freebsd2/bin/sh % MAKEOBJDIRPREFIX=/tmp make
[Creating objdir /tmp/root/git/freebsd2/amd64.amd64/bin/sh...]
Building /tmp/root/git/freebsd2/amd64.amd64/bin/sh/builtins.c
...



# Without this change this is the result of setting AUTO_OBJ to
# default yes for a user:

/root/git/freebsd2/bin/sh % make
[Creating objdir /usr/obj/root/git/freebsd2/amd64.amd64/bin/sh...]
mkdir: /usr/obj/root/git/freebsd2: Permission denied
make: "/root/git/freebsd2/share/mk/auto.obj.mk" line 67: could not use /usr/obj/root/git/freebsd2/amd64.amd64/bin/sh: .OBJDIR=/root/git/freebsd2/bin/sh

universe/buildworld,native-xtools/xdev/installworld/installkernel all fine. Ran into unrelated issue fixed in rS325249 which made me think these were failing.

[bdrewery]

Export MK_AUTO_OBJ==yes finding to sub-makes. This was not being checked
very often in buildworld but exporting the positive finding will save more time.

[bdrewery]

Uploaded the wrong commit, fix that.

[bdrewery]

Don't bother trying to enable MK_AUTO_OBJ if WITHOUT_AUTO_OBJ is set or
MK_AUTO_OBJ is in .MAKEOVERRIDES.

[bdrewery]

Disable opportunistic AUTO_OBJ for showconfig/print-dir

[bdrewery]

Fix 'make print-dir' mistake, add WITHOUT_AUTO_OBJ man file and force "on" for 'make showconfig'.
Also tested that makeman did the right thing and shows AUTO_OBJ as default enabled.

[bdrewery]

trap more signals for rmdir cleanup.
This should be final.

[sjg]

share/mk/src.sys.obj.mk

93 ↗

(On Diff #34485)

If this is not bound to ${.MAKE.LEVEL} == 0 you have race conditions to deal with below.

107 ↗

(On Diff #34485)

There's a race condition here you need to account for.
Which is why Mkdirs in auto.obj.mk tries mkdir -p more than once.
Between your [ -d $${dir} ] and mkdir $${dir} another instance of make may have made the same intermediate directory.
mkdir -p deals with this case sanely now, but used to throw an error.
The above has the same issue as early mkdir -p

123 ↗

(On Diff #34485)

This can fail - and should do so gracefully, in the case that another make process has created other subdirs.

[bdrewery]

share/mk/src.sys.obj.mk

93 ↗

(On Diff #34485)

Good point. It's intended to be. I'll add a ${.MAKE.LEVEL} == 0 check here. If it succeeds it does export MK_AUTO_OBJ=yes so it should cover all sub-makes.

[bdrewery]

share/mk/src.sys.obj.mk

107 ↗

(On Diff #34485)

I am speaking only of auto.obj.mk here...

By the way, the FreeBSD mkdir -p does not have a problem with concurrent mkdir -p <samepath> running. If a component for mkdir(2) returns EEXIST it just ignores the error unless it is the last component, and then build returns 2 which main does nothing with. It looks like NetBSD's mkdir handles it similarly.

Just thinking about this, doing mkdir -p /usr/obj/foo would hit the race you're worried about in auto.obj.mk since both of '/' and '/usr' will exist already. The nature of the -p flag makes this not a problem for auto.obj.mk.

Perhaps there is an issue on NFS?

[bdrewery]

share/mk/src.sys.obj.mk

107 ↗

(On Diff #34485)

Ah I missed you said "mkdir -p deals with this case sanely now". Ok we're on the same page.

[bdrewery]

share/mk/src.sys.obj.mk

123 ↗

(On Diff #34485)

Hm, I just realized this whole method is unnecessary. If the process can write (or even make any directory) to the last component before the first missing component, then it's a success.
I.e., if /usr/obj is writable and /usr/obj/usr doesn't exist, that's good enough for a success.
If /usr/obj is not writable and /usr/obj/usr doesn't exist, that's good enough for a failure.

[bdrewery]

• Simplify detection, no need to create directories, just look for the last missing component and if we can write to that directory.
• Simplify more of the change overall
• Check the wanted OBJDIR rather than only OBJROOT
• Support broken symlinks in the path by recursing on the new link's path, tested with 2 nested links in MAKEOBJDIRPREFIX both with and without missing dirs and writable dirs
• Fix more of the 'make showconfig' support, it actually runs a child with make -dg1 so we need to export to that one
• Add an UPDATING entry

[bdrewery]

Remove leftover comment to myself

[sjg]

As I said, recent FreeBSD has no problem with mkdir -p , but auto.obj.mk needs to work on lots of systems, and at least one - I can't recall now, did have a bug where mkdir -p had a race condition.

[bdrewery]

Any further comments? I'd like to commit this today.

[sjg]

A couple of nits but generally looks ok

share/mk/src.sys.obj.mk

103 ↗

(On Diff #34633)

fwiw you can remove the check for .TARGETS == "" by using ${.TARGETS:Uall:Nclean*.....} != ""

116 ↗

(On Diff #34633)

I would normally use [ ! -d

[bdrewery]

Thanks!

share/mk/src.sys.obj.mk

103 ↗

(On Diff #34633)

Sure I'll use that, I took the pattern from auto.obj.mk.

[bdrewery]

share/mk/src.sys.obj.mk

103 ↗

(On Diff #34633)

Actally .TARGETS:Uall doesn't work here as .TARGETS is defined blank for some reason at this point.

~/git/freebsd2 # make -C bin/sh -V .OBJDIR
make: "/root/git/freebsd2/share/mk/src.sys.obj.mk" line 100: set ''

From:

.info ${.TARGETS:Dset} '${.TARGETS}'