Any objections to me submitting this as-is? I got sucked into $work for the last few months, but I did take some time to re-test just now to make sure this hasn't bit-rotted. I'd like to get this in and then look at some of the cache/busdma stuff.

In D18593#440598, @jah wrote:

In D18593#440526, @jmallett wrote:

But it also looks like, on all MIPS configurations, we're reserving TLB entry 0 for KSTACK_TLB_ENTRY. But I don't see any evidence that we're using that TLB slot currently: there's no call to tlb_insert_wired() on that slot that I can find. It also seems like a single TLB entry might not be sufficient for that, depending on the combination of PAGE_SIZE and KSTACK_PAGES.
Can that TLB entry be repurposed, or are we still using it somewhere I've missed?

In D18593#440526, @jmallett wrote:

I don't have any exceptionally helpful comments; the code looks fine, but I haven't given it the review it deserves.

1. I don't think very much about 32-bit MIPS, unfortunately.
2. If you're interested in MALTA, why not use MALTA64? If there's an actual hardware platform you care about, it'd be useful to know which one that is.
3. The general approach of a reusable mapping for this kind of bounce buffering is a good one. Really, you could have a per-thread reusable mapping wired into the TLB for this use, and just update the wiring. That way the update path would be quicker, no TLBL/TLBS/TLBRefill. I never understood the sysmap_lmem code very well because (see point 1 above), but that's how I'd approach it.
4. I would encourage allocating pairs of virtual pages even if you end up wasting one or more, to avoid even possibly splitting the TLB entry for one of these with something else.
5. I agree we should get rid of 32-bit SMP on MIPS barring something really disruptive entering the marketplace. (This seems vanishingly unlikely, much as MIPS seems to be vanishing. People who would have accidentally used MIPS are all using ARM, and people who would have intentionally sought out MIPS are using RISC-V instead.)
6. It's almost 110% certain that the cache handling is not totally right. I don't think very much these days about how to make it right, and it's a subject that requires nontrivial thought given VIVT vs. VIPT. If you're deep into reading about this stuff and have all of it paged in, I'd encourage taking a swing at auditing all of mips/mips for places that caching could be made correct, and giving it a try. You're unlikely to make it worse :)

Thanks for doing this, this is stuff that needs to be happen and looks to be good work!

Designate ignored return values

Avoid reassigning stderr if the dup of the existing descriptor failed

Simplify openlog() control flow, only dup existing stderr if LOG_PERROR
isn't already in use.

Remove cap_close_stdio() and pass stderr within cap_syslog

In D18989#407424, @markj wrote:

In D18989#407047, @jah wrote:

In D18989#406823, @markj wrote:

In D18989#406689, @jah wrote:

I do have one concern with that approach, namely that it's specific to cap_syslog. One of the reasons I started with cap_close_stdio() was that it could be applied to other services that might run into this problem in the future. Do you guys see any added value in an approach like that, or would you prefer that it always be handled in a service-specific way?

I do think cap_close_stdio() would be worth having, but I don't find this particular example particularly compelling... cap_syslog only needs the standard streams if LOG_PERROR is specified, and in this case it's not. We don't have many examples of casper services which request STDIO, so I'm a bit concerned about generalizing from too few examples. That said, I don't object to this patch and am ok with it going in if @oshogbo is ok with it.

BTW, @oshogbo, why does cap_fileargs request the STDIO descriptors? I can't see how they're used.

How about we go forward with your stderr-passing idea for now, and implement something like cap_close_stdio() if/when it ends up being needed? Since passing stderr keeps things completely within cap_syslog, it also has the near-term advantage of not needing a shlib minor version bump or a manpage edit.

That sounds good to me, thanks.

In D18989#406823, @markj wrote:

In D18989#406689, @jah wrote:

I do have one concern with that approach, namely that it's specific to cap_syslog. One of the reasons I started with cap_close_stdio() was that it could be applied to other services that might run into this problem in the future. Do you guys see any added value in an approach like that, or would you prefer that it always be handled in a service-specific way?

I do think cap_close_stdio() would be worth having, but I don't find this particular example particularly compelling... cap_syslog only needs the standard streams if LOG_PERROR is specified, and in this case it's not. We don't have many examples of casper services which request STDIO, so I'm a bit concerned about generalizing from too few examples. That said, I don't object to this patch and am ok with it going in if @oshogbo is ok with it.

BTW, @oshogbo, why does cap_fileargs request the STDIO descriptors? I can't see how they're used.

In D18989#406289, @markj wrote:

Sorry for the delay in looking at this.

This approach should work, but I'm not crazy about it: every cap_syslog consumer needs to consider whether this problem affects them, and add this extra API call if so. Another quick solution would be to revert r341692 and change savecore(8) to not use cap_syslog: it is not a daemon and is quite unlikely to be affected by the problem (reconnecting to syslogd upon an error) which motivates cap_syslog in the first place. All it really needs to do is call openlog(3) before entering capability mode, which it already does.

A more robust solution might be to revert r341692 and instead change cap_openlog(LOG_PERROR) to pass the caller's STDERR_FILENO descriptor to the cap_syslog service. The service can dup() the received descriptor onto STDERR_FILENO, and then I believe cap_syslog(3) will do the right thing. Some care would be needed to ensure that the cap_syslog process doesn't allocate STDERR_FILENO for some other purpose.

Other than reverting r341692, this is the simplest thing that works. But I'm a complete libcasper n00b, so I'm not sure if this is the ideal approach.
Should we instead have something finer-grained (maybe the ability to revoke an individual fd from the service), and/or something like the ability to specify whether stdio/fds should be inherited through the call to cap_service_open() instead of being baked into the service declaration?

ping...does anyone want to review?

In D18664#398479, @kbowling wrote:

You may be able to add something to /usr/src/tests/sys/

Remove unintended README change

In D14143#304010, @jah wrote:

In D14143#301044, @jah wrote:

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

• Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Do you want to fix this or should I?

In D14143#301044, @jah wrote:

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

I don't think just getting rid of MAKEOBJDIR is enough.
With PORTS_MODULES=x11/nvidia-driver in an otherwise empty src.conf, and an empty src.env.conf, I get this during installkernel when I try to build HEAD using 'make buildkernel; make installkernel':

> src (install)

> src/nvidia (install)

install -T release -o root -g wheel -m 555 nvidia.ko /usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/x11/nvidia-driver/work/stage/boot/modules/
install: nvidia.ko: No such file or directory

• Error code 71

buildkernel sets MK_AUTO_OBJ=yes while installkernel sets MK_AUTO_OBJ=no.
It looks like this produces some disagreement in where the build output goes. That's why I also removed MK_AUTO_OBJ in this review.
I think you have to do that or something like it to get everything on the same page; adding MK_AUTO_OBJ=yes to PORTSMODULESENV would also work but seems riskier and less correct.

Get rid of staging, remove MK_AUTO_OBJ

In D14143#300791, @bdrewery wrote:

I fixed the problem differently in r329232

In D14143#299639, @jah wrote:

In D14143#299618, @bdrewery wrote:

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

That would be multimedia/cx88 (which surprisingly, people still use:). I created that port a long time ago, before PORTS_MODULES was even a thing; besides the kmods there's also a userspace utility in there, and that's the part that fails in 11.0+ due to src.libnames.mk.

But like I said in the review for D13053, maybe the real right answer is to just split ports like that up, so the kmods are in a separate port by themselves?
I'm still the maintainer for cx88, so I can make that happen :)

I did all this because I thought maybe I could make PORTS_MODULES more flexible without adding too much complexity, and staging is just an extension of that. But if that's just going to create a maintenance headache that's worse than any problem it might fix, let's not do it that way.

In D14143#299618, @bdrewery wrote:

Which port inspired the need for D13053 due to the LDADD/LIBADD problem?
I cannot recreate the issue being reported on current@. I'm still not fond of staging file like this as it is a ~4th way of system files being read. I'd rather go the other direction and always install everything.
Anyway, since I cannot recreate the newly reported issue I want to look at the port you ran into a problem with and see if I can rework D13053.

In D14143#296975, @jah wrote:

In D14143#296955, @bdrewery wrote:

Is this fixing something? It's adding more complexity.

Killing MK_AUTO_OBJ/MKOBJDIR fixes the installkernel failure described at https://lists.freebsd.org/pipermail/freebsd-current/2018-January/068319.html
I wouldn't be surprised if there's a cleaner way to do that though.

The staging part is more of a conceptual thing. We're already setting OSVERSION and SRC_BASE to build the port as though it's being built against the updated OS. It seems like we should also be building it against the system makefiles as they would be installed on the updated OS. Just setting SYSDIR might not be enough; sys/conf/kmod.mk still pulls bsd.init.mk and bsd.compiler.mk from the system makefile path, not to mention any non-kmod components of the port. It seems reasonably likely that mismatches between those makefiles and e.g. the environment created by buildkernel/installkernel could cause problems on something like a major version upgrade. That said, I did not see any problems when testing an 11->12 upgrade either before or after this change.

In D14143#296955, @bdrewery wrote:

Is this fixing something? It's adding more complexity.

Add dep on share/mk contents

ping

Remove only -m directives from MAKEFLAGS

We could also kill off just the -m directives in MAKEFLAGS. A one-liner like this should do it:

In D13053#271885, @bdrewery wrote:

I think removing MAKEFLAGS may be too much. I think we can just add MAKESYSPATH=/usr/share/mk to the environment and should also set _WITHOUT_SRCCONF=yes.

Style(9) fixes

ping...Marius/Andrew, are you guys ok with the change as it is now?

Use correct include path for arm64 bus_dma_impl.h

In D10729#228579, @marius wrote:

In D10729#228105, @jah wrote:

In D10729#228068, @marius wrote:

In D10729#222257, @jah wrote:

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?

One alternative I'd considered was to let each MD bus_dma.h #define something like WANT_INLINE_DMAMAP.
If set that would change the bus_dmamap* declarations in bus_dma.h to static inline. That would still be a lot
cleaner than what we have now. If we did that, then:
--we could re-inline the sparc64 implementation
--we could inline the x86 and arm64 dmamap calls, which have been sparc64-like function-pointer calls since
Intel DMAR support was added
--for x86 we could move the NULL checks back out to the inlined functions, removing any function call
penalty for no-op maps.

Testing at work with 40 G NICs and a userland IP stack has show that de-inlining such kind of wrappers has a
quite measurable impact on throughput (it's still in the noise with 10 G, though). So given that the performance
of I/O gear also is ever increasing and given that it's not too much hassle, I'd actually prefer your suggested
approach of optional inlining based on a (WANT_)INLINE_DMAMAP macro.

That's a good data point, thank you for testing on that hardware.
How much did the performance penalty as a % end up being for the 40G case?

Inlining alloc_iov() there, i. e. reverting 94210486, results in a ~ 0.4 % improvement of mean performance
with 40 G NICs:
https://github.com/NTAP/warpcore/blob/master/lib/src/warpcore.c
In principle, alloc_iov() is similar to _bus_dmamap_load_buffer() in terms of functionality (get/load a raw
buffer) and usage, i. e. repeatedly called per payload buffer and mbuf respectively. This might also
translate to bus_dmamap_sync() as - although it makes no sense to call that function in a loop -, sending/
receiving one packet requires at least a few invocations (with different DMA maps and/or operations).

Allow MD busdma implementations to provide inline map functions

In D10729#228068, @marius wrote:

In D10729#222257, @jah wrote:

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?

One alternative I'd considered was to let each MD bus_dma.h #define something like WANT_INLINE_DMAMAP.
If set that would change the bus_dmamap* declarations in bus_dma.h to static inline. That would still be a lot
cleaner than what we have now. If we did that, then:
--we could re-inline the sparc64 implementation
--we could inline the x86 and arm64 dmamap calls, which have been sparc64-like function-pointer calls since
Intel DMAR support was added
--for x86 we could move the NULL checks back out to the inlined functions, removing any function call
penalty for no-op maps.

Testing at work with 40 G NICs and a userland IP stack has show that de-inlining such kind of wrappers has a
quite measurable impact on throughput (it's still in the noise with 10 G, though). So given that the performance
of I/O gear also is ever increasing and given that it's not too much hassle, I'd actually prefer your suggested
approach of optional inlining based on a (WANT_)INLINE_DMAMAP macro.

In D10729#223080, @kib wrote:

I gnerally like this simplification.

I do not believe that moving NULL checks into non-inline path would have significant (or measurable) effect on operation of any device.

Would there be any concern about performance implications of un-inlining the NULL checks for x86 and the map calls for sparc64?
Maybe network adapters that need high-frequency mbuf operations?