Page MenuHomeFreeBSD

Speed up *-old-* make targets by using sed instead of xargs
ClosedPublic

Authored by emaste on Apr 1 2022, 11:04 PM.

Details

Summary
Targets like 'list-old-files' used "xargs -n1" to produce a list of one
file per line.  Using xargs resulted in one fork+exec for each argument,
resulting in rather long runtime.  Use sed to split the list instead.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

emaste created this revision.
emaste added a reviewer: jhb.

On one machine make list-old-files took 30s wall clock time with xargs and less than 1s with sed.

Is this an April fools' joke?, if not nice!

How does trperform? (something like tr -s "[:space:]" "\n" ?

This revision is now accepted and ready to land.Apr 8 2022, 10:05 PM

How does trperform? (something like tr -s "[:space:]" "\n" ?

I didn't try it, but the tens of thousands of forks were responsible for the xargs slowness, and sed does (and tr would) avoid that. sed is already included in ITOOLS, so is going to be available in any context where this could be used

I had made a similar change in my local tree, but used make's :ts flag instead. I guess the sed expression more portable, though:

@@ -3202,9 +3211,9 @@
 list-old-files: .PHONY
        @cd ${.CURDIR}; \
        ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} ${.TARGET} \
-           -V OLD_FILES -V "OLD_FILES:Musr/share/*.gz:R" \
-           -V "OLD_FILES:Mlib/*.so.*:S,^lib,usr/lib32," \
-           -V "OLD_FILES:Musr/lib/*:S,^usr/lib,usr/lib32," | xargs -n1 | sort
+           -V "OLD_FILES:ts\n" -V "OLD_FILES:Musr/share/*.gz:R:ts\n" \
+           -V "OLD_FILES:Mlib/*.so.*:S,^lib,usr/lib32,:ts\n" \
+           -V "OLD_FILES:Musr/lib/*:S,^usr/lib,usr/lib32,:ts\n" | sort
 
 delete-old-files: .PHONY
        @echo ">>> Removing old files (only deletes safe to delete libs)"

There's a similar speedup that can be done by avoiding the ... | while read var pattern. The shell has to read from pipes one character at a time, and the old-files list is almost 1MB. It's much faster to send to a temp file, which the shell will then read in blocksize chunks:

@@ -3215,7 +3224,8 @@ delete-old-files: .PHONY
 # the Makefile parser segfault.
        @exec 3<&0; \
        cd ${.CURDIR}; \
-       ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} list-old-files | \
+       OLDTMP=$$(mktemp -t old); \
+       ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} list-old-files > $$OLDTMP; \
        while read file; do \
                if [ -f "${DESTDIR}/$${file}" -o -L "${DESTDIR}/$${file}" ]; then \
                        chflags noschg "${DESTDIR}/$${file}" 2>/dev/null || true; \
@@ -3228,7 +3238,8 @@ delete-old-files: .PHONY
                              <&3; \
                  fi; \
                done; \
-       done
+       done < $$OLDTMP; \
+       rm -f $$OLDTMP
 # Remove catpages without corresponding manpages.
        @exec 3<&0; \
        find ${DESTDIR}/usr/share/man/cat* ! -type d 2>/dev/null | sort | \

sed is a win over xargs but using :ts\n would be better still.