bsdconfig packages gains f_package_parallel_install(), which installs a
job set concurrently while honoring inter-package dependencies. The
run-dependency data already collected from pkg-rquery(8) seeds a
topological schedule (Kahn's algorithm): packages with no pending
dependencies dispatch immediately to a worker pool sized by
PACKAGE_INSTALL_JOBS (default hw.ncpu), and each completion, reported
over a FIFO, releases any dependents it was blocking. Workers run
pkg-fetch(8) then pkg-install(8) with LOCK_WAIT/LOCK_RETRIES raised so
siblings tolerate the package database lock, and log per-package output
under a scratch directory for post-mortem. A failure marks all
transitive dependents skipped rather than aborting the whole set, and
the summary names every package that failed or was skipped. Overall
progress renders through bsdpv(1) in line mode, fed one line per
completed package.
Details
- Reviewers
adrian emaste des bapt asiciliano
Prerequisites:
https://reviews.freebsd.org/D58118
https://reviews.freebsd.org/D58133
The scheduler's ordering, failure propagation, and progress plumbing
were exercised with a mock pkg(8): independent packages start together,
dependents start only after their prerequisites complete, and a failed
package skips exactly its transitive dependents while unrelated
installs proceed.
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Skipped - Unit
Tests Skipped - Build Status
Buildable 75171 Build 72054: arc lint + arc unit
Event Timeline
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 55 | nproc is so much nicer to use (and more portable!). | |
| 618 | Where does tmpdir get cleaned up? | |
| 636 | ||
| 675 | I've fallen into the habit of setting variables like USE_XDIALOG to true / false for explicitness when dealing with boolean flags. It's made my shell code cleaner over the years. | |
| 710 | ||
| 731 | ||
| 756 | ||
There are a lot of lines in this package -- I'm guessing the maintenance is tricky.
Given our tech stack requirements, it might be a good idea to consider lua instead of /bin/sh. Are you open to that possibility (given infinite time/resources of course..)?
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 641–651 | Might be worth DRYing _pjob_ndeps_$varpkg to prevent typo-related bugs from sneaking in. | |
Given infinite time and resources, anything is possible. At present, the TUI rewritten in flua(1) -- a subset of Lua in base at /usr/libexec/flua -- would change bsdconfig from 35k lines of shell to 35k lines of Lua that constantly calls out to /bin/sh to implement the TUI via bsddialog (using the os.execute or io.popen shell-out approach).
Also, standard Lua modules such as luaposix are not included in flua by default. So what we would need in base is not yet there. So now we're talking about potentially >40k lines of Lua total because we'd have to implement native bindings to bridge the gap, just to add thinkings like process forks, file descriptor manipulation etc, to our Lua scripts.
As it stands presently, moving bsdconfig away from a pure-shell runtime, we are faced with a fundamental constraint: if I do it in pure flua, it will still technically be an sh orchestration engine. You will just be using Lua's control flow, tables, and string manipulation to generate shell command fragments that get passed down to /bin/sh to handle the actual system configuration plumbing.
To truly eliminate the shell and make Lua do everything, I would need to expand flua itself by implementing a custom, minimal POSIX/FreeBSD C binding library in base. Something that exposes native system forks, stream redirections, environment control, and direct system calls to the Lua state.
How do you view the trade-off between expanding flua's C-side footprint in base versus keeping it purely as a logic engine that delegates the heavy lifting to smaller, focused shell helper blocks?
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 636 | That's not POSIX. Omitting $ from variables inside arithmetic expansion, I checked, was added to POSIX in POSIX.1-2001 and so I could conceivably drop the $ from total inside parens, but I've always programmed with them to support POSIX.2-1992. I'd rather make it a separate commit or series of commits moving the shell into POSIX.1-2001 standards (and ultimately POSIX.1-2024), but presently assignment operators inside arithmetic expansion (=, +=, -=, ++, --) are still not part of the the POSIX definition. Does += work? Yes. Is overloading the : built-in to eat the expansion that is left as a result more efficient than building an assignment? Arguably no. It's arguably less efficient to build positional arguments to the : built-in just to utilize some syntactic sugar that is not POSIX compliant than to use total=$(( ... )) | |
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 55 | It is nicer. Also nicer than the equally portable getconf _NPROCESSORS_ONLN I'll switch this over to nproc | |
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 675 | This would actually be a good optimization. I looked into it and true/false have been shell built-ins since POSIX.1-2001 and I went back to 12.x and confirmed they are indeed built-ins back then too (I don't have anything older to test on). Invoking those built-ins would absolutely be more efficient than the test idiom (which comes from C programming parlance). However, since I have more than 900 instances of [ "$bool" ] and/or [ ! "$bool" ], I think we should make the style change a separate review. | |
| usr.sbin/bsdconfig/share/packages/packages.subr | ||
|---|---|---|
| 710 | Not POSIX. I'd prefer to keep this POSIX compliant shell. I also don't like the bodge of dropping the result onto : to avoid the attempted execution the result | |
| 731 | Not POSIX. I'd prefer to keep this POSIX compliant shell. I also don't like the bodge of dropping the result onto : to avoid the attempted execution the result | |
| 756 | Not POSIX. I'd prefer to keep this POSIX compliant shell. I also don't like the bodge of dropping the result onto : to avoid the attempted execution the result | |