Page MenuHomeFreeBSD

Significantly speed up Mk/Scripts/qa.sh
ClosedPublic

Authored by feld on Mar 23 2015, 8:00 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, May 2, 12:20 PM
Unknown Object (File)
Thu, May 2, 12:20 PM
Unknown Object (File)
Thu, May 2, 7:10 AM
Unknown Object (File)
Fri, Apr 26, 1:27 AM
Unknown Object (File)
Mar 25 2024, 10:29 PM
Unknown Object (File)
Mar 2 2024, 4:38 PM
Unknown Object (File)
Dec 31 2023, 5:50 AM
Unknown Object (File)
Dec 19 2023, 10:59 PM
Subscribers
None

Details

Reviewers
bapt
bdrewery
mat
Group Reviewers
portmgr
Summary

qa.sh is painfully slow when you have a large amount of files being
installed.

Ex: net-mgmt/observium

$ wc -l pkg-plist

8082 pkg-plist

There are a *lot* of text files in here. /usr/bin/file is painfully slow
to examine them in detail which makes the stage-qa process hang for an
unreasonable amount of time. In fact, I've delayed updating this port
because at one point I thought poudriere was hanging.

It appears we can solve this by telling file(1) to exclude doing
detailed analysis of ascii files when we're checking to see if there are
unstripped binaries.

My test results show the following results from /usr/bin/time:

Now:
558.37 real 557.25 user 1.24 sys

With this patch:
2.77 real 2.21 user 0.56 sys

Also piping find to xargs and using -n 20 -P 4 to run in parallel:
1.15 real 2.88 user 1.77 sys

It seemed that the xargs/parallel was not completely necessary, but does
provide a measurable boost if we want to go down this road.

We may want to exclude other tests besides ascii. I'll leave that up to
portmgr who knows better.

Diff Detail

Repository
rP FreeBSD ports repository
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

feld retitled this revision from to Significantly speed up Mk/Scripts/qa.sh.
feld updated this object.
feld edited the test plan for this revision. (Show Details)
feld added reviewers: bdrewery, mat, bapt.
bdrewery edited edge metadata.

Please commit. Yes we can probably ignore other types. We only care about elf here.

This revision is now accepted and ready to land.Mar 23 2015, 8:05 PM