Page MenuHomeFreeBSD

Add a BUILD_WITH_OPIPEFAIL option to use `bash -o pipefail` as the bmake shell
Needs ReviewPublic

Authored by arichardson on Aug 23 2018, 2:42 PM.

Details

Summary

This is useful to find failed commands if the output is piped. By default
shells will only return the exit code of the last command in the pipeline.
The -o pipefail option causes the pipelline exit code to be nonzero if any
of the intermediate commands return non-zero.
This has been extremely useful for me when debugging build failures on
non-FreeBSD hosts due to missing/incompatible commands.

This currently creates a sh binary that will execute bash -o pipefail but
once we have -o pipefail support in /bin/sh we could instead bootstrap a
version of /bin/sh that defaults to pipefail.

Diff Detail

Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 19099
Build 18726: arc lint + arc unit

Event Timeline

arichardson created this revision.Aug 23 2018, 2:42 PM

FreeBSD's sh grew pipefail support in rS344502 so this might be OBE.

Has this turned up any bad pipelines in Makefiles?

bdrewery added a comment.EditedFeb 27 2019, 11:27 PM

Has this turned up any bad pipelines in Makefiles?

Ah right there is that dd one from a while back.

Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?

#! /bin/sh
set -o pipefail 2>/dev/null || :
exec "$@"

Has this turned up any bad pipelines in Makefiles?

Ah right there is that dd one from a while back.
Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?

#! /bin/sh
set -o pipefail 2>/dev/null || :
exec "$@"

Hmm set -o pipefail || : doesn't actually ignore the error.

# sh -ec 'set -o pipefail || :; echo test'
set: Illegal option -o pipefail

:(

Has this turned up any bad pipelines in Makefiles?

Ah right there is that dd one from a while back.
Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?

#! /bin/sh
set -o pipefail 2>/dev/null || :
exec "$@"

Hmm set -o pipefail || : doesn't actually ignore the error.

# sh -ec 'set -o pipefail || :; echo test'
set: Illegal option -o pipefail

:(

This works and avoids a fork and error.

x=$(set -o); case $x in *pipefail*) set -o pipefail ;; esac

I'm trying to avoid adding fork/exec/grep overhead to a sh_pipefail.sh script since it will be executed a lot.

Has this turned up any bad pipelines in Makefiles?

Ah right there is that dd one from a while back.
Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?

#! /bin/sh
set -o pipefail 2>/dev/null || :
exec "$@"

Hmm set -o pipefail || : doesn't actually ignore the error.

# sh -ec 'set -o pipefail || :; echo test'
set: Illegal option -o pipefail

:(

This works and avoids a fork and error.

x=$(set -o); case $x in *pipefail*) set -o pipefail ;; esac

I'm trying to avoid adding fork/exec/grep overhead to a sh_pipefail.sh script since it will be executed a lot.

I wonder if we should just add an environment variable such as "SH_ENABLE_PIPEFAIL" and use that in a bootstrapped bin/sh to enable pipefail? That would avoid the fork() overhead. And as a fallback we could invoke the wrapper script/binary.

brooks added a subscriber: jilles.Feb 28 2019, 4:13 PM

Bring in Jilles to comment on the idea of adding an environmental variable to enable pipefail.

Jilles mentioned on IRC a simpler way in the wrapper to enable it without the set -o call:

command set -o pipefail 2>/dev/null || :