Hack to install symlinks atomically.

[markj]

Summary

We see frequent CI build failures due to libgcc_s.so missing.
libgcc_s.so is installed multiple times while building libraries, but we
do not pass -S to install(1) when creating the symlink, so there are
windows where an attempt to link against libgcc_s.so fails. Pass -S
like we do when installing the binary lib itself. See r322565.

This is a straw proposal, presumably we should specify -S in a more
generic way, and I'm not sure whether this will cause problems in
!FreeBSD build environments. For instance, on Linux -S means something
else, but maybe it's ok because we should already have bootstrapped
install(1) by this point?

[imp]

Why do we install libgcc_s multiple times?

This is a good hack to avoid that (and we can fold it into INSTALL_* macros only on FreeBSD if install hasn't been bootstrapped by this point), but not understanding why we do that, it seems like we're papering over things with this... A useful paper over, but still, why the multiple install?

[markj]

In D26453#588490, @imp wrote:

Why do we install libgcc_s multiple times?

This is a good hack to avoid that (and we can fold it into INSTALL_* macros only on FreeBSD if install hasn't been bootstrapped by this point), but not understanding why we do that, it seems like we're papering over things with this... A useful paper over, but still, why the multiple install?

The proximate reason is that libgcc_s is in _prereq_libs, _startup_libs and _generic_libs. PR 233769 has some more background (see comment 11), but I don't claim to really understand why we do things this way. I hope someone else can chime in here.

[emaste]

The proximate reason is that libgcc_s is in _prereq_libs, _startup_libs and _generic_libs.

I do not know why we do that, and believe doing so is incorrect - it should be axiomatic that any given file in the build has exactly one rule/path that builds it. We can still look at fixing that with this change papering over the problem though.

[markj]

In D26453#588503, @emaste wrote:

The proximate reason is that libgcc_s is in _prereq_libs, _startup_libs and _generic_libs.

I do not know why we do that, and believe doing so is incorrect - it should be axiomatic that any given file in the build has exactly one rule/path that builds it. We can still look at fixing that with this change papering over the problem though.

Indeed. I suspect that we want this change regardless, for the same reason that we install the binary lib with -S.

[emaste]

Yes, although maybe we want to either include -S into ${INSTALL_*SYMLINK} or a new ${INSTALL_*SYMLINK_FLAGS} as @imp suggested? But we don't do that for the lib itself, this is at least consistent with that.

[markj]

Try to formalize a bit. Add the perhaps too verbosely named
SHLINSTALLSYMLINKFLAGS to pair with SHLINSTALLFLAGS.

[emaste]

OK with me

[markj]

share/mk/bsd.lib.mk

425 ↗

(On Diff #77152)

Should we process INSTALLFLAGS_EDIT here? I don't see any uses of it, not sure what it comes in handy for.

[bdragon]

Regarding installing it multiple times, I suspect that something is going wrong in the target up to date logic for the library.

Poking through a (meta-mode against already-built world to reduce number of rebuilt targets, but I assume the same problem exists in a "normal" build) build with "-d m" to dump out logic, I see the following:

Examining libgcc_s.so.1.full...
modified 19:54:21 Sep 16, 2020...out-of-date
Building /usr/obj/root/bdragon28-freebsd/powerpc.powerpc64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Make_Update: libgcc_s.so.1.full
 recheck(libgcc_s.so.1.full): current update time: 19:55:47 Sep 16, 2020
inspect parent libgcc_s.so.1: flags 9, type b020001, made 1, unmade 1 - unmade children
inspect parent libgcc_s.so.1.debug: flags 9, type b020001, made 1, unmade 0 - libgcc_s.so.1.full made, schedule libgcc_s.so.1.debug (made 1)

....

Examining libgcc_s.so.1.full...
modified 19:55:47 Sep 16, 2020...out-of-date
Building /usr/obj/root/bdragon28-freebsd/powerpc.powerpc64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Make_Update: libgcc_s.so.1.full
 recheck(libgcc_s.so.1.full): current update time: 19:55:53 Sep 16, 2020
inspect parent libgcc_s.so.1: flags 9, type b020001, made 1, unmade 1 - unmade children
inspect parent libgcc_s.so.1.debug: flags 9, type b020001, made 1, unmade 0 - libgcc_s.so.1.full made, schedule libgcc_s.so.1.debug (made 1)

NOTE: This is without this phab, I have not retested things to see if there are any differences with it.

so I wonder if make itself is getting confused about the state of the target.

Not that I understand much about how make works internally. It does make me wonder though.

[markj]

In D26453#588916, @bdragon wrote:

Regarding installing it multiple times, I suspect that something is going wrong in the target up to date logic for the library.

Poking through a (meta-mode against already-built world to reduce number of rebuilt targets, but I assume the same problem exists in a "normal" build) build with "-d m" to dump out logic, I see the following:

Examining libgcc_s.so.1.full...
modified 19:54:21 Sep 16, 2020...out-of-date
Building /usr/obj/root/bdragon28-freebsd/powerpc.powerpc64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Make_Update: libgcc_s.so.1.full
 recheck(libgcc_s.so.1.full): current update time: 19:55:47 Sep 16, 2020
inspect parent libgcc_s.so.1: flags 9, type b020001, made 1, unmade 1 - unmade children
inspect parent libgcc_s.so.1.debug: flags 9, type b020001, made 1, unmade 0 - libgcc_s.so.1.full made, schedule libgcc_s.so.1.debug (made 1)

....

Examining libgcc_s.so.1.full...
modified 19:55:47 Sep 16, 2020...out-of-date
Building /usr/obj/root/bdragon28-freebsd/powerpc.powerpc64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Make_Update: libgcc_s.so.1.full
 recheck(libgcc_s.so.1.full): current update time: 19:55:53 Sep 16, 2020
inspect parent libgcc_s.so.1: flags 9, type b020001, made 1, unmade 1 - unmade children
inspect parent libgcc_s.so.1.debug: flags 9, type b020001, made 1, unmade 0 - libgcc_s.so.1.full made, schedule libgcc_s.so.1.debug (made 1)

NOTE: This is without this phab, I have not retested things to see if there are any differences with it.

so I wonder if make itself is getting confused about the state of the target.

Not that I understand much about how make works internally. It does make me wonder though.

It looks like library targets depend on .PHONY, which might explain this. I'm no make wizard though.

3032 .for _lib in ${_startup_libs} ${_prebuild_libs} ${_generic_libs}                                                                                                         
3033 ${_lib}__L: .PHONY .MAKE                                                                                                                                                 
3034 .if !defined(_MKSHOWCONFIG) && exists(${.CURDIR}/${_lib})                                                                                                                
3035         ${_+_}@${ECHODIR} "===> ${_lib} (obj,all,install)"; \                                                                                                            
3036                 cd ${.CURDIR}/${_lib}; \                                                                                                                                 
3037                 if [ -z "${NO_OBJWALK}" ]; then ${MAKE} MK_TESTS=no DIRPRFX=${_lib}/ obj; fi; \                                                                          
3038                 ${MAKE} MK_TESTS=no DIRPRFX=${_lib}/ all; \                                                                                                              
3039                 ${MAKE} MK_TESTS=no DIRPRFX=${_lib}/ install                                                                                                             
3040 .endif                                                                                                                                                                   
3041 .endfor

.PHONY    The target does not correspond to an actual file; it is always
          considered to be out of date, and will not be created with the
          -t option.  Suffix-transformation rules are not applied to
          .PHONY targets.

[markj]

In D26453#589070, @markj wrote:

It looks like library targets depend on .PHONY, which might explain this. I'm no make wizard though.

Hmm I guess this is unrelated. I'm not sure why libgcc_so.so.1.full ends up being stale. Reading through bsd.lib.mk I *think* it should only depend on object files; maybe one of them depends on an auto-generated header that gets updated during the build? I'm not sure.

[markj]

I plan to commit this later today if there are no objections.

[bdrewery]

Nice find!
I missed that the symlink installs didn't use $SHLINSTALLFLAGS back in rS322565. I banged my head on this problem for a long time with dhw@ hitting it daily.