FYI, I have now committed a man page for DPCPU(9) in r316003. It includes some (safe?) synchronisation patterns in its example code.

In D10048#209661, @markm wrote:

RWatson: Not to get picky or anything, but there was already a malloc() in that place.

In D10048#209659, @markm wrote:

In D10048#209647, @rwatson wrote:

Your commit introduces per-CPU memory allocation.

Where?

M

This is correct: you must make sure that you continue to access state on the CPU for which you acquired a mutex -- e.g., by caching a pointer to the per-CPU state you are accessing, in case migration takes place.

But that is racey. Preemption can in theory occur straight after I have verified that it hasn't. Looks like I need to use critical regions for now. I can live with that if you can?

In D10048#209275, @markm wrote:

• I feel that using sleepable mutexes here is fine -- the difference in performance is negligible on most contemporary microarchitectures, and there is an argument for moving some of our other critical sections to being mutexes (e.g., per-CPU UMA caches).

I'm concerned about cpu migration. Mutexes don't guarantee that a thread will stay on the same cpu, right?

Just a few quick comments:

How does this play out with non-native ABIs (e.g., the Linux emulator) -- I thought SYS_fork (etc) were ABI-specific system-call numbers?

It would be more tempting to add the systrace_probe_func invocation at the end of fork_return() where the similar KTRACE probe fires (for similar reasons). Take a look at the call to ktrsysret(SYS_fork, 0, 0); for details.

This seems like a sensible general change. I'm quite surprised it wasn't this way already (.. and sort of misremembered that it was -- IPSEC should always have been using the netisr...).

I like the idea, and encourage you to proceed, but be aware that struct tcpcb is part of the user-visible ABI for monitoring tools (sigh). Someone should restock our supplies of padding someday.

Adding "-R" support is a good idea.

I'm not sure if consumers of m_pulldown() make assumptions about writability or not. The man page doesn't mention that they should (or not) but this is more of an empirical question. As I recall, m_pulldown() is particularly popular in IPv6, so tagging Bjoern to perhaps take a look at this and see what he thinks.

Seems reasonable to me!

Mentor approval granted. (NB: not a technical review, but existing technical reviews here look good to go!)

Overall I like this approach, but there's an important experimental question as to whether this enables all the use cases we care about -- and, more generally, whether there are visible failure modes that might surprise application programmers. We also need to think quite hard to convince ourselves this maintains safe operation. Getting Jon Anderson, Ben Laurie, and David Drysdale to review the approach would be very useful.

In D8110#168999, @kib wrote:

Implement Jonathan Anderson suggestion of checking the result of dotdot lookup against the recorded list of traversed vnodes. Drop rename notifications. Check for dotdot vnodes living on local fs.

In general, this seems like a good idea. A bit of wordsmithing does help, and reviewing an updated commit candidate before it goes into the tree wouldn't hurt if you can tolerate another RTT with reviewers :-).

I'm fine with exposing the hostname here -- the goal of Capsicum has always been to be pragmatic about getting software running where it doesn't violate isolation properties. You could argue that this is an information leak and/or might cause problems for deterministic replay-style applications of Capsicum -- but I'd rather we had more code working in a sandboxing. :-)

High-level comments rather than a detailed code review:

Should we also be ditching M_COPY() and/or switching it to M_COPYM() for consistency..?

The comments could have to do with the au_qctrl structure, which uses "int", whereas the au_qctrl64 type uses "uint64_t". You can see the code handling the older structure a bit below this point, which likely has to do with compatibility with older Solaris/XNU versions rather than FreeBSD per se.

Looks good to me!

I'd suggest avoiding any style changes in the initial copy of code to the new locations, so diffs can more easily be checked, and changes can be more easily merged. Apply style/comment/etc changes in a separate commit.

I think using panic() here would be preferable to KASSERT().

This change seems sensible to me, but I believe John has worked with this code most recently, so I've added him as a reviewer as well.

This seems like a good idea, but fixing the style bugs (e.g., local variable definitions should be at the start of the function before any executable code) is necessary.

CheriBSD change for reference: https://github.com/CTSRD-CHERI/cheribsd/commit/7ed51f1f4ff2ea4c7ba1cebc101d9dd6e26f3844

Hi Warner:

In D4964#105406, @theraven wrote:

Hi Jim,

I don't think that this approach will work, because noinline does not do what you might expect. The compiler will not inline the function, but it can still look inside it. In particular, it will determine that it does not read or write any memory other than the arguments and propagate these attributes to anything that calls it. This will happen even with the ThinLTO model that Google is slowly upstreaming to LLVM.

In particular, if this function is called immediately before a free(), then the compiler is free to elide it and leave the sensitive data in the free list.

Note that the version using memory barriers can be made portable by using the relevant memory barrier function (memory order: sequentially consistent) from <stdatomic.h>.

The simplest way to prevent the compiler eliding the stores would be to have an extern volatile void* variable (defined in an assembly file, so that the compiler can never see it) and so assign the buf variable to it before doing the bzero. The compiler is not permitted to elide volatile stores to variables that it can not guarantee that it can see the entire lifetime of, because they may be device memory. This will guarantee that the compiler believes that the pointer has escaped and so it then may not elide any stores to it. This does not establish a happens-before relationship with any other threads, so the compiler should still be able to emit efficient code for everything.

Seems generally sensible. Better documenting the locking protocol would be useful -- why various locks are required, and what the implications of the unlock flag are (they are more broad than suggested by the name). Addition of a flags argument that is conflated with a local flags field is confusing.

On net.inet.tcp.experimental.initcwnd10: if it's not shipped in a release, we can remove it as a bit of interface stability in 11-CURRENT. If it's not a tunable, then it will generate a warning on boot if set in sysctl.conf after kernel support is removed, as the sysctl won't be present. If it's also a tunable, there won't be a warning, and it will silently fail to operate. So, hopefully it's not in a release? If it is, we probably need something in the release notes about its removal, etc.

Just to quickly comment on sysctl naming: I don't like putting things like "nonstandard" or "experimental" in sysctl names, because what is standardised or experimental frequently changes. Sysctl names are effectively ABIs, since the names get put in loader.conf, sysctl.conf, etc, and if you change the sysctl name in the kernel, you may break those configuration lines. For sysctl.conf, you at least get an error, but if a loader tunable changes names, it just silently stops working. There are other ways to document whether something is standardised -- e.g., source-code commands and in the man page -- that are probably better. We already have a TCP man page documenting a number of sysctls (I think?) and really the information should just be there, with suitable caveats. So I'd vote (if I were asked to vote) for just naming the sysctl the most descriptive thing.

Just to quickly chime in with a few high-level points:

Should be fine for SDT, but I've not tried FBT, and as you point out, stack layout may differ a bit. I will try to give that a test run in isolation from the unupstreamed dtrace_invop_jump_addr change. We really do need to upstream the latter -- is that in your court?

Updated diff from svn contains further context; no functional change.

Updated patch generated against Subversion, and contains more context. No functional change.

I admit I substantially prefer this approach to the prior approach, given that much of the fast-forwarding win was from direct dispatch -- something we now do by default for local-destination traffic as well (and have done for ten years).

I don't currently have views on the DTrace-related parts themselves, but some quick comments on other details.

Although I have read through it and believe that the principle of this patch is a very good one, I have not, myself, tested it.

This patch seems reasonable to me.

In D473#64974, @lattera-gmail.com wrote:

In D473#64973, @rwatson wrote:

In D473#59442, @rwatson wrote:

That seems a sensible strategy.

Any luck?

$WORK has been tough lately. I've been working 110 hours a week for around a month now.
Zero time for the patch until around October. Sorry for the delays.

In D473#59442, @rwatson wrote:

That seems a sensible strategy.

Seems reasonable to me -- probably long overdue!

La la la.

That seems a sensible strategy.

Hi Shawn. Hope you are feeling better post-travel. Just wanted to check on the status of this patch -- will there be an updated version fixing the credential transition issue soon?

In D2025#55036, @markm wrote:

In D2025#55026, @rwatson wrote:

My reason for removing the pluggability is fewer locks. There is currently the need for a long-term-sleepable read-many-write-seldom lock. We don't have such a thing and I want to get rid of the need for it.

[ The need is for a lock to be held to prevent reconfiguration while pluggable modules come, go and/or are exchanged in priority ]

I think RM_SLEEPABLE will do what you wanted -- although you want in fact to have two rm locks, one non-sleepable for non-sleepable contexts, and one sleepable for sleepable contexts, to prevent priority inversion. During a reconfiguration event, you'd acquire first the latter, and then the former, before committing any changes. The MAC Framework does this using an sx lock and an rm lock, since there weren't sleepable rm locks when it was written -- but perhaps it should be moved over now.

According to the man page, this won't work: "Writers are permitted to sleep while holding a read-mostly lock, but readers are not." I need a sleeping reader.

My reason for removing the pluggability is fewer locks. There is currently the need for a long-term-sleepable read-many-write-seldom lock. We don't have such a thing and I want to get rid of the need for it.

[ The need is for a lock to be held to prevent reconfiguration while pluggable modules come, go and/or are exchanged in priority ]

If the goal is to have an MFC candidate, then having pluggable support for Yarrow remain seems sensible so that we can leave Yarrow the default in older branches while starting to support Fortuna there. Also, it would be nice to have a fallback option available in the event that a problem is found with the new Fortuna implementation -- i.e., "Here is your nice security advisory on a new break in Fortuna published at $conference -- the workaround is to use Yarrow instead, with the following tradeoffs". That said, there is much benefit to less complexity.

Getting close, but a couple more comments :-).

Sounds reasonable with regard to fchdir() and friends -- and, indeed -- right now, CAPMODE forces O_BENEATH, but it's true we don't have a variant on O_BENEATH that forces it to be used for open()'d directories via other open()'d directories with O_BENEATH set. I think this is fine. There's a slight confused deputy problem if directories are passed from CAPMODE processes to non-CAPMODE processes anyway -- and we don't make it worse here.