Page MenuHomeFreeBSD

src: fix deps in package build
Needs ReviewPublic

Authored by vladlen on Sun, Jul 19, 10:34 AM.
Referenced Files
Unknown Object (File)
Sat, Aug 8, 9:22 AM
Unknown Object (File)
Thu, Aug 6, 11:26 AM
Unknown Object (File)
Wed, Aug 5, 8:51 PM
Unknown Object (File)
Mon, Aug 3, 7:10 AM
Unknown Object (File)
Sat, Aug 1, 5:43 AM
Unknown Object (File)
Sat, Aug 1, 2:25 AM
Unknown Object (File)
Fri, Jul 31, 3:50 AM
Unknown Object (File)
Sun, Jul 26, 12:32 AM
Subscribers

Details

Reviewers
ivy
Group Reviewers
releng
pkgbase
Summary

I am testing pkgbase upgrade scenarios, using a CURRENT snapshot
as target version to simplify testing. The same issues appear on releng/15.X and
current/15 branches. Source version is 14.4R after pkgbasify.

This patch fixes a problem: post-install triggers fail
because not all packages are updated yet due to missing explicit
dependencies. Specifically, caroot depends on certctl, which
depends on openssl-lib(but certctl's dependency in Makefile
is on openssl, that is obsolete after changing certctl from script
to binary implementation) and openssl-lib is not declared. Thus, during upgrade,
certctl's post-install script tries to load libcrypto.so.35 before
openssl-lib is installed, causing "Shared object not found".

[234/435] Upgrading FreeBSD-openssl from 14.4p6 to 16.snap20260629171155...
[234/435] Extracting FreeBSD-openssl-16.snap20260629171155: 100%
[235/435] Upgrading FreeBSD-certctl from 14.4p6 to 16.snap20260629171155...
[235/435] Extracting FreeBSD-certctl-16.snap20260629171155: 100%
[236/435] Upgrading FreeBSD-caroot from 14.4p6 to 16.snap20260427094521...
[236/435] Extracting FreeBSD-caroot-16.snap20260427094521: 100%
ld-elf.so.1: Shared object "libcrypto.so.35" not found, required by "certctl"
pkg: POST-INSTALL script failed

The fix adds an explicit dependency on openssl-lib to the certctl
package.

This demonstrates that we need to explicitly declare dependencies
for all packages with post-install triggers and their descendants,
contrary to the README's claim that pkg should handle this
automatically.

Further investigation reveals that we should explicitly declare
dependencies for all packages and their descendants.
pkg could not resolve automatically many cases,
especially cases with new packages.

Test Plan

tested on local build, amd64 arch.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

we don't want to add manual dependencies on library packages; these are supposed to be handled automatically by pkg. if that's not working properly, it's a bug in pkg(8) and should be fixed there.

% pkg info -d FreeBSD-certctl
FreeBSD-certctl-16.snap20260720115648:
        FreeBSD-openssl-16.snap20260720115648
        FreeBSD-clibs-16.snap20260720115648 (libc.so.7)
        FreeBSD-openssl-lib-16.snap20260720115648 (libcrypto.so.35)

(my guess would be that perhaps the automatic dependencies are not being used during ordering of packages to install/upgrade.)

In D58339#1341688, @ivy wrote:

we don't want to add manual dependencies on library packages; these are supposed to be handled automatically by pkg. if that's not working properly, it's a bug in pkg(8) and should be fixed there.

I don't believe this is a bug in pkg(8). I reviewed a very long log file (with -dddd)
and confirmed that library dependencies are not used during the scheduling stage.

I found a relevant comment in Bugzilla (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259785#c15)
which suggests that this functionality is not yet implemented. In fact, I suspect it may be impossible to implement
correctly in practice. Library dependencies at the package level can introduce cycles, and pkg currently has no way to resolve them.

I could prepare a long list of technical obstacles explaining why this is difficult
(if not nearly impossible) to implement. However, rather than waiting for a solution
that may never come, I think we need to make a responsible decision and address this case now.

Only the releng team has the authority to implement proper package configuration,
build it, and verify it across the full system. We can do this under full control and
ensure that upgrades to new versions work reliably.

If we look at the ports framework, it has explicit dependencies and
safeguards against circular dependencies at the framework level. pkgbase currently has
minimal dependency settings, as if someone else is responsible for ensuring correct
pkg behavior. But pkg cannot predict all possible issues —
it doesn't know about libraries required by post-install executables,
nor can it prioritize vital packages appropriately.

I thougth about solutions and think we can implement simple script to add all needed
dependencies in Makefiles. It is not endless manual work.

I don't believe this is a bug in pkg(8). I reviewed a very long log file (with -dddd) and confirmed that library dependencies are not used during the scheduling stage.

that's the bug: a package's dependencies should be installed before the package itself. there's no reason for this to be different with shlib dependencies, especially since that will break many/most post-install scripts.

in case of cycles, it's fine to break them by installing all the packages involved first (in any order), then running the package scripts once everything is installed and binaries/shlibs are available. this is how Linux package managers (e.g., dpkg) handle this problem.

@bapt any thoughts here?

However, rather than waiting for a solution that may never come, I think we need to make a responsible decision and address this case now.

if this affected upgrades in a supported release, i would probably agree. but aiui, this is only for upgrading from 14.x, where pkgbase was not supported, and upgrading from 14 to 15 with pkgbase is also not supported. that doesn't mean we should deliberately break it, but it also means there's a higher bar for adding complexity to the build to support this case. in the case of adding manual shlib dependencies to packages to work around deficiencies in pkg itself, i'm not convinced this is worth it. manually specifying things like this is prone to bitrot over time, and may even introduce other issues if packages are renamed or files are moved in the future.

Only the releng team has the authority to implement proper package configuration

i'm pretty sure this is not true, if i understand you; any developer has the same "authority" to work on pkgbase as they do any other part of the codebase, and i did the vast majority of my pkgbase work before i was on releng. there's probably a lot of overlap between people who work on pkgbase and people who are on releng, but that's largely a coincidence.

In D58339#1343137, @ivy wrote:

I don't believe this is a bug in pkg(8). I reviewed a very long log file (with -dddd) and confirmed that library dependencies are not used during the scheduling stage.

that's the bug: a package's dependencies should be installed before the package itself. there's no reason for this to be different with shlib dependencies, especially since that will break many/most post-install scripts.

The reason is fundamental: it is technically impossible for pkg to know which package provides a required library in the new release — that package may not even exist in the old version, or the library may have moved. This information simply doesn't exist at the scheduling stage. So automatic resolution is impossible here — explicit dependencies are the only reliable way.

To illustrate this, I have actually built world and all packages from main, and manually analyzed the dependencies. Without explicit declarations prepared in built target system, the correct upgrade order simply cannot be guaranteed. This is not a theoretical concern — it's a practical finding from real testing.

The recent situation with PAM is a perfect example: it was moved to a separate package, and during upgrades, systems become unable to log in because login(1) cannot find libpam.so.6. The only safe way to handle this is to ensure the dependency is explicitly declared, otherwise pkg does not have information where libpam.so.6 is located.

However, rather than waiting for a solution that may never come, I think we need to make a responsible decision and address this case now.

if this affected upgrades in a supported release, i would probably agree. but aiui, this is only for upgrading from 14.x, where pkgbase was not supported, and upgrading from 14 to 15 with pkgbase is also not supported.

I used 14.4 simply because it was the system I had ready and powerful enough for testing. The issue is not source‑specific — it's about the target OS release.

manually specifying things like this is prone to bitrot over time, and may even introduce other issues if packages are renamed or files are moved in the future.

That bitrot risk was actually my biggest worry too, but I think the current pkgbase packaging setup with deps in Makefile gives us a simple way to handle it.

Only the releng team has the authority to implement proper package configuration

i'm pretty sure this is not true, if i understand you; any developer has the same "authority" to work on pkgbase as they do any other part of the codebase, and i did the vast majority of my pkgbase work before i was on releng. there's probably a lot of overlap between people who work on pkgbase and people who are on releng, but that's largely a coincidence.

That's a very fortunate coincidence — you've been deeply involved in pkgbase both before and after joining releng, so if anyone knows the pitfalls, it's you. I didn't mean to suggest any formal restriction; rather, I was thinking about the practical coordination needed to get changes into official builds. Your experience would be extremely helpful here. I'm just offering my help voluntarily, of course.

P.S. I have vacation now and limited access to Internet.