Page MenuHomeFreeBSD

Remove MK_AUTO_OBJ from env passed to PORTS_MODULES
ClosedPublic

Authored by jah on Jan 31 2018, 7:12 AM.

Details

Summary

Remove MK_AUTO_OBJ from the build environment provided to each port in PORTS_MODULES.
This fixes a failure to resolve object file paths seen when buildkernel (which sets MK_AUTO_OBJ=yes) and installkernel (which sets MK_AUTO_OBJ=no) are run as separate steps. r329232 partially fixed this scenario by removing MAKEOBJDIR, but it seems AUTO_OBJ settings also need to be on the same page for these two steps.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

jah created this revision.Jan 31 2018, 7:12 AM
jah updated this revision to Diff 38714.Jan 31 2018, 10:02 AM

Add dep on share/mk contents

Is this fixing something? It's adding more complexity.

jah added a comment.Feb 1 2018, 5:06 AM

Is this fixing something? It's adding more complexity.

Killing MK_AUTO_OBJ/MKOBJDIR fixes the installkernel failure described at https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068319.html
I wouldn't be surprised if there's a cleaner way to do that though.

The staging part is more of a conceptual thing. We're already setting OSVERSION and SRC_BASE to build the port as though it's being built against the updated OS. It seems like we should also be building it against the system makefiles as they would be installed on the updated OS. Just setting SYSDIR might not be enough; sys/conf/kmod.mk still pulls bsd.init.mk and bsd.compiler.mk from the system makefile path, not to mention any non-kmod components of the port. It seems reasonably likely that mismatches between those makefiles and e.g. the environment created by buildkernel/installkernel could cause problems on something like a major version upgrade. That said, I did not see any problems when testing an 11->12 upgrade either before or after this change.

jah added a comment.Feb 9 2018, 6:36 AM
In D14143#296975, @jah wrote:

Is this fixing something? It's adding more complexity.

Killing MK_AUTO_OBJ/MKOBJDIR fixes the installkernel failure described at https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068319.html
I wouldn't be surprised if there's a cleaner way to do that though.
The staging part is more of a conceptual thing. We're already setting OSVERSION and SRC_BASE to build the port as though it's being built against the updated OS. It seems like we should also be building it against the system makefiles as they would be installed on the updated OS. Just setting SYSDIR might not be enough; sys/conf/kmod.mk still pulls bsd.init.mk and bsd.compiler.mk from the system makefile path, not to mention any non-kmod components of the port. It seems reasonably likely that mismatches between those makefiles and e.g. the environment created by buildkernel/installkernel could cause problems on something like a major version upgrade. That said, I did not see any problems when testing an 11->12 upgrade either before or after this change.

ping...any thoughts on this? I'm not set on doing things this way by any means, if there's something different you'd prefer to do here.

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

jah added a comment.Feb 10 2018, 12:26 AM

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

That would be multimedia/cx88 (which surprisingly, people still use:). I created that port a long time ago, before PORTS_MODULES was even a thing; besides the kmods there's also a userspace utility in there, and that's the part that fails in 11.0+ due to src.libnames.mk.

But like I said in the review for D13053, maybe the real right answer is to just split ports like that up, so the kmods are in a separate port by themselves?
I'm still the maintainer for cx88, so I can make that happen :)

I did all this because I thought maybe I could make PORTS_MODULES more flexible without adding too much complexity, and staging is just an extension of that. But if that's just going to create a maintenance headache that's worse than any problem it might fix, let's not do it that way.

jah added a comment.Feb 10 2018, 4:34 AM
In D14143#299639, @jah wrote:

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

That would be multimedia/cx88 (which surprisingly, people still use:). I created that port a long time ago, before PORTS_MODULES was even a thing; besides the kmods there's also a userspace utility in there, and that's the part that fails in 11.0+ due to src.libnames.mk.
But like I said in the review for D13053, maybe the real right answer is to just split ports like that up, so the kmods are in a separate port by themselves?
I'm still the maintainer for cx88, so I can make that happen :)
I did all this because I thought maybe I could make PORTS_MODULES more flexible without adding too much complexity, and staging is just an extension of that. But if that's just going to create a maintenance headache that's worse than any problem it might fix, let's not do it that way.

Also: to recreate the issue reported on current@, you have to run buildkernel and installkernel as separate steps instead of just 'make kernel'. Doing that I was able to repro the problem with both multimedia/cx88 and x11/nvidia-driver.

I fixed the problem differently in r329232

jah added a comment.Feb 14 2018, 4:17 AM

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

  • Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

jah updated this revision to Diff 39687.Feb 24 2018, 6:50 PM

Get rid of staging, remove MK_AUTO_OBJ

jah added a comment.Feb 24 2018, 6:51 PM
In D14143#301044, @jah wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

  • Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Do you want to fix this or should I?

jah added a comment.Mar 11 2018, 9:58 PM
In D14143#304010, @jah wrote:
In D14143#301044, @jah wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

  • Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Do you want to fix this or should I?

ping

jah edited the summary of this revision. (Show Details)Mar 26 2018, 7:17 AM
jah retitled this revision from Build PORTS_MODULES against staged system makefiles, remove MK_AUTO_OBJ/MKOBJDIR to Remove MK_AUTO_OBJ from env passed to PORTS_MODULES.Mar 29 2018, 4:19 PM
bdrewery accepted this revision.Mar 30 2018, 7:52 PM
This revision is now accepted and ready to land.Mar 30 2018, 7:52 PM
This revision was automatically updated to reflect the committed changes.