Other than reverting r341692, this is the simplest thing that works. But I'm a complete libcasper n00b, so I'm not sure if this is the ideal approach.
Should we instead have something finer-grained (maybe the ability to revoke an individual fd from the service), and/or something like the ability to specify whether stdio/fds should be inherited through the call to cap_service_open() instead of being baked into the service declaration?

ping...does anyone want to review?

In D18664#398479, @kbowling wrote:

You may be able to add something to /usr/src/tests/sys/

Remove unintended README change

In D14143#304010, @jah wrote:

In D14143#301044, @jah wrote:

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

• Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Do you want to fix this or should I?

In D14143#301044, @jah wrote:

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

• Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Get rid of staging, remove MK_AUTO_OBJ

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

In D14143#299639, @jah wrote:

In D14143#299618, @bdrewery wrote:

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

That would be multimedia/cx88 (which surprisingly, people still use:). I created that port a long time ago, before PORTS_MODULES was even a thing; besides the kmods there's also a userspace utility in there, and that's the part that fails in 11.0+ due to src.libnames.mk.

But like I said in the review for D13053, maybe the real right answer is to just split ports like that up, so the kmods are in a separate port by themselves?
I'm still the maintainer for cx88, so I can make that happen :)

I did all this because I thought maybe I could make PORTS_MODULES more flexible without adding too much complexity, and staging is just an extension of that. But if that's just going to create a maintenance headache that's worse than any problem it might fix, let's not do it that way.

In D14143#299618, @bdrewery wrote:

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

In D14143#296975, @jah wrote:

In D14143#296955, @bdrewery wrote:

Is this fixing something? It's adding more complexity.

Killing MK_AUTO_OBJ/MKOBJDIR fixes the installkernel failure described at https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068319.html
I wouldn't be surprised if there's a cleaner way to do that though.

The staging part is more of a conceptual thing. We're already setting OSVERSION and SRC_BASE to build the port as though it's being built against the updated OS. It seems like we should also be building it against the system makefiles as they would be installed on the updated OS. Just setting SYSDIR might not be enough; sys/conf/kmod.mk still pulls bsd.init.mk and bsd.compiler.mk from the system makefile path, not to mention any non-kmod components of the port. It seems reasonably likely that mismatches between those makefiles and e.g. the environment created by buildkernel/installkernel could cause problems on something like a major version upgrade. That said, I did not see any problems when testing an 11->12 upgrade either before or after this change.

In D14143#296955, @bdrewery wrote:

Is this fixing something? It's adding more complexity.

Add dep on share/mk contents

ping

Remove only -m directives from MAKEFLAGS

We could also kill off just the -m directives in MAKEFLAGS. A one-liner like this should do it:

In D13053#271885, @bdrewery wrote:

I think removing MAKEFLAGS may be too much. I think we can just add MAKESYSPATH=/usr/share/mk to the environment and should also set _WITHOUT_SRCCONF=yes.

Style(9) fixes

ping...Marius/Andrew, are you guys ok with the change as it is now?

Use correct include path for arm64 bus_dma_impl.h

In D10729#228579, @marius wrote:

In D10729#228105, @jah wrote:

In D10729#228068, @marius wrote:

In D10729#222257, @jah wrote:

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?

One alternative I'd considered was to let each MD bus_dma.h #define something like WANT_INLINE_DMAMAP.
If set that would change the bus_dmamap* declarations in bus_dma.h to static inline. That would still be a lot
cleaner than what we have now. If we did that, then:
--we could re-inline the sparc64 implementation
--we could inline the x86 and arm64 dmamap calls, which have been sparc64-like function-pointer calls since
Intel DMAR support was added
--for x86 we could move the NULL checks back out to the inlined functions, removing any function call
penalty for no-op maps.

Testing at work with 40 G NICs and a userland IP stack has show that de-inlining such kind of wrappers has a
quite measurable impact on throughput (it's still in the noise with 10 G, though). So given that the performance
of I/O gear also is ever increasing and given that it's not too much hassle, I'd actually prefer your suggested
approach of optional inlining based on a (WANT_)INLINE_DMAMAP macro.

That's a good data point, thank you for testing on that hardware.
How much did the performance penalty as a % end up being for the 40G case?

Inlining alloc_iov() there, i. e. reverting 94210486, results in a ~ 0.4 % improvement of mean performance
with 40 G NICs:
https://github.com/NTAP/warpcore/blob/master/lib/src/warpcore.c
In principle, alloc_iov() is similar to _bus_dmamap_load_buffer() in terms of functionality (get/load a raw
buffer) and usage, i. e. repeatedly called per payload buffer and mbuf respectively. This might also
translate to bus_dmamap_sync() as - although it makes no sense to call that function in a loop -, sending/
receiving one packet requires at least a few invocations (with different DMA maps and/or operations).

Allow MD busdma implementations to provide inline map functions

In D10729#228068, @marius wrote:

In D10729#222257, @jah wrote:

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?

One alternative I'd considered was to let each MD bus_dma.h #define something like WANT_INLINE_DMAMAP.
If set that would change the bus_dmamap* declarations in bus_dma.h to static inline. That would still be a lot
cleaner than what we have now. If we did that, then:
--we could re-inline the sparc64 implementation
--we could inline the x86 and arm64 dmamap calls, which have been sparc64-like function-pointer calls since
Intel DMAR support was added
--for x86 we could move the NULL checks back out to the inlined functions, removing any function call
penalty for no-op maps.

Testing at work with 40 G NICs and a userland IP stack has show that de-inlining such kind of wrappers has a
quite measurable impact on throughput (it's still in the noise with 10 G, though). So given that the performance
of I/O gear also is ever increasing and given that it's not too much hassle, I'd actually prefer your suggested
approach of optional inlining based on a (WANT_)INLINE_DMAMAP macro.

In D10729#223080, @kib wrote:

I gnerally like this simplification.

I do not believe that moving NULL checks into non-inline path would have significant (or measurable) effect on operation of any device.

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?

Whitespace fixes

Replace pcpu_find with get_pcpu()

Return sched_unpin() to original order, rename fields

Use constant for pcpu padding. Collapse sysmaps struct.