Page MenuHomeFreeBSD

.hooks: assorted bugfixes and efficiency overhaul, replacing external commands (awk) by shell built-ins
Needs ReviewPublic

Authored by mandree on Sun, Jul 19, 12:51 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Jul 24, 7:26 AM
Unknown Object (File)
Fri, Jul 24, 2:11 AM
Unknown Object (File)
Wed, Jul 22, 4:13 AM
Unknown Object (File)
Tue, Jul 21, 11:03 PM
Unknown Object (File)
Tue, Jul 21, 7:22 AM
Unknown Object (File)
Tue, Jul 21, 6:11 AM

Details

Reviewers
fernape
Group Reviewers
portmgr
ports secteam
Summary

Fix some robustness bugs in the scripts,
and reduce the number of external commands run so as to
speed up sweeping updates to multiple ports.

  • BUGFIX: add IFS=$'\n' to hook scripts so they really parse the common git diff --name-only output line-wise irrespective of what's on the lines. (Adds robustness)
  • BUGFIX: check_category_makefile breaks if a $category/Makefile contains a $portname subdir with "++", we put the portname into a grep -E pattern, which grep rejects with the error 'repetition-operator operand invalid'. Example ports that trigger this: audio/timidity++* and several devel/*, math/* and science port (47 in total).

    FIX by adding a sed command to common.sh that adds a backslash before each special character of a grep -E modern regular expression, and using that.
  • BUGFIX: replace echo by printf %s to print data we don't control. The echo man page tells us to use printf instead of echo if the string could contain dashes or backslash esacpes
  • BUGFIX: check_mk_indentations only printed the last component of an impromperly indented file, which is unhelpful. Print the full repository path ($mkfile) instead.
  • ROBUSTNESS: add a new check_aaa_filesnames script that rejects space, tab, quotes, backslashes, and anything that git's diff --name-only will escape (which includes the newline character).
  • SPEEDUP check_category_makefile, check_created_by, check_files: replace the expensive "run awk process" to split items by shell built-ins such as case/in/esac, printf, read -r.
  • FEATURE: pre-commit: support a PRE_COMMIT_KEEP_GOING env variable, similar to make -k, which will continue running further hook parts even after an error. This allows seeing all hook errors at once and fix them all at once, avoiding a fix-one-&-fail-again cycle.

PR: 296903
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296903

Test Plan

I have tested this on FreeBSD 15.1-RELEASE-p1 amd64.

  1. ensure your hooks are activated, git config get core.hookspath needs to print .hooks, else set it
  2. create a local branch git checkout -b my-disposable-review-branch
  3. then try to "git add" all sorts of breakage we intend to trap, adding files with invalid characters (blanks, quotes, tabs, newlines), broken indentation (for Mk files down deep), change a category/port/Makefile and remove the relevant SUBDIRS += line from the category/Makefile

The pre-commit-hook should reject the commit.

Diff Detail

Repository
R11 FreeBSD ports repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

mandree created this revision.
mandree edited the test plan for this revision. (Show Details)
mandree changed the edit policy from "All Users" to "No One".

Added "PR: line" to the commit message and re-diffed with git format-patch HEAD^ -U9999 so as to provide full context if reviewers want to look at context.

mandree changed the edit policy from "No One" to "All Users".Tue, Jul 21, 12:36 AM

Hi Matthias,

thanks for the cleanup — the input hardening in particular looks sensible.

I'm a bit curious why ports-secteam was added as a reviewer here. These are commit-side hook scripts, which I'd have read as portmgr territory. If secteam was pulled in specifically for the adversarial-input parts (check_aaa_filenames, the ERE escaping, printf %s, IFS=$'\n'), that's understandable — but I don't think that needs a formal secteam sign-off to proceed. Happy to resign ports-secteam from the review if that was the only reason; just let me know if you actually wanted a security look at something specific.

joneum — ports-secteam

fernape added inline comments.
.hooks/pre-commit.d/check_aaa_filenames
17

I think all variables should be dreferenced with "${}", specially for filenames which might contain spaces.

25

space between ] and ;

.hooks/pre-commit.d/check_category_makefile
14

In general I prefer the ; to follow the previous command.

.hooks/pre-commit.d/check_files
20

Does this intentionally check the return status of the assignment of "status=0"?

.hooks/pre-commit.d/check_mk_indentations
25

This can probably be:

if [ ! cmp -s "${tmpdir}"/* ]; then...

des added inline comments.
.hooks/pre-commit.d/check_aaa_filenames
17

Brackets make absolutely no difference wrt word splitting, their only purpose is to allow a variable to be embedded in other text, e.g. foo${bar}baz. Using them (or not) when they aren't required is a stylistic choice.

25

Matthias is correctly following the style of the existing code, even though it does not match our established (but unfortunately undocumented) style for shell code.

.hooks/pre-commit.d/check_mk_indentations
25

without brackets, yes, but again, Matthias is following the style of the existing code.

.hooks/pre-commit.d/check_aaa_filenames
17

Yes, not difference wrt word splitting. Nevertheless, using brackets is consistent syntax for using something higher that $9 and for using string substitution. "${}" works in 99% of the cases regardless of the usage of the variable.
If anything for consistency, since this same patch adds the following line somewhere below:

case "${firstline}"

Why in this case is it dereferenced with brackets?

But not a stopper for me.

25

The existing code is inconsistent. See:

if [ -r "${common_functions}" ]; then

which is already in the script.

Again not a stopper for me.

So, let me upload a new version as followup, which adds two more tools
addresses the ${var} brace syntax,
and also fix shellcheck complaints about the scripts,
except those telling us what we can't have in POSIX sh, as in shellcheck -s sh -e1090,3003,3037,3040,3043.

.hooks/pre-commit.d/check_files
20

I don't know what Tobias's intentions were when he added the code like that in

commit 537c7c00047fd4ba18247c3e160529f697f7d1a2
Author: Tobias C. Berner <tcberner@FreeBSD.org>
Date:   Mon Nov 14 19:34:18 2022 +0100

    framework: new hook to only allow 'default' files in category/port

but I agree it doesn't make sense that way, and it escaped me. We should fix that.

  • after tcberner's accident with a status=0 invalidating the if [ $? -ne 0 ] let's just inline everyting into the if.
  • add check_no_binaries to reject files (using od and grep) that have certain control characters or NUL bytes in the first 16 kB
  • add check_unmerged scripts that rejects conflict markers (we only check <<<<<<< ||||||| >>>>>>>, not ======= which might be in MD or RST or plain text files individually)
  • make check_category_makefile also check on $CATEGORY/Makefile edits that we don't break the SUBDIR += ... enumeration versus what's actually in Git.
  • set -o pipefile in a few places,
  • ensure we run git's core.quotePath=true (via environment)
  • quote all shell variables unless we safely set them ourselves
  • fix shellcheck warnings (pkg install hs-ShellCheck)
  • fix excess ":" prefixes on some multi-line error messages by switching from $* to $@ in common.sh
  • tweak logging

+ assorted minor fixes found during testing

mandree added inline comments.
.hooks/pre-commit.d/check_aaa_filenames
25

I have ignored that ]; blank discussion. I will only look at such items if we have canonical documentation in a central place and are willing to enforce them, and do a one-off sweeping "reformat the world" commit.

.hooks/pre-commit.d/check_category_makefile
14

see above. Marking "Done" because it's duplicate.

.hooks/pre-commit.d/check_mk_indentations
25

True, but the suggestion was valid (and also shellcheck emits it).

I've ignored formatting matters since those would be going forth and back, and I'd look to portmgr@ for setting up a style guide before we should start looking over style. I don't mind much if I add the ; at the end of the command or with a blank before the ; - either is arguable.

Hi Matthias,

thanks for the cleanup — the input hardening in particular looks sensible.

I'm a bit curious why ports-secteam was added as a reviewer here. These are commit-side hook scripts, which I'd have read as portmgr territory. If secteam was pulled in specifically for the adversarial-input parts (check_aaa_filenames, the ERE escaping, printf %s, IFS=$'\n'), that's understandable — but I don't think that needs a formal secteam sign-off to proceed. Happy to resign ports-secteam from the review if that was the only reason; just let me know if you actually wanted a security look at something specific.

joneum — ports-secteam

ports-secteam as I see it would oversee what bits get merged to quarterly, no? There is the occasional "direct commit" that would benefit, so you'd want to see, and Fernando picked up the ball :-)

This revision is now accepted and ready to land.Wed, Jul 22, 8:22 PM
.hooks/pre-commit.d/check_files
20

One more thing...

  • You need to have the status=0 at an outer/top level so it's reliably set to a number
  • And - guessing - Tobias probably wanted to only run in case grep -E found something, i. e. there was something as result of git diff --cached + grep -E to avoid trouble from processing an e
This revision now requires review to proceed.Wed, Jul 22, 8:28 PM