@imp thank for looking into this! And cherry-picking stuff to reduce this are totally cool with me. :) I forgot to mention that the "whole-archive" was an empirical workaround for some weird linking "symbol not found issues" with LTO enabled that I could not overcome or explain otherwise. I am not 100% sure if this some deficiency of LTO in lld's version that we have in 13.1 that I was experimenting with and if it fixed in -current. I was planning to setup full fledged -current build environment to see if it's really needed, but it got boggled down by some 3-weeks travelling spree going at the moment (still ongoing). It's effects are offset by the symbol versioning, so anything not referenced is simply dropped by the linker, which I actually verified. I should be back to my office in about 2 weeks.

@imp I think at one point you would be pulling just the devpath.c from the libsa, but then you gave up and started to pull the whole circus. But it continued to pull that one still. Fixed in the new rev.

• devpath.c needs not to be pulled in explicitly any longer.

In D44872#1023204, @imp wrote:

impressive gains if true... However, the 'no-whole-archive' stuff makes me ask the question "doesn't that eliminate all the commands that we build on linker sets?"

Also wonder how making things weak actually does anything useful... We don't include these in multiple places, and the efi ones certainly aren't included twice...

I'll have to look at this in more detail, but making sure all the commands and other things we build on linker sets are still around would be good to test in the mean time.

@glebius dully noted. 🙉🧐

In D43776#999307, @markj wrote:

In D43776#999148, @sobomax wrote:

@markj @glebius I suppose looking at the crash pattern, disabling LRO would probably workaround it, correct? We have other issues with LRO in this setup so maybe this is the easiest way out for now.

Maybe? I don't know anything about your network configuration. I don't see anything in the backtraces which specifically implicates LRO.

@markj @glebius I suppose looking at the crash pattern, disabling LRO would probably workaround it, correct? We have other issues with LRO in this setup so maybe this is the easiest way out for now.

In D43776#998976, @glebius wrote:

You need to enable INVARIANTS in the kernel config. If you anticipate that performance impact would be too big to run production normally, try using vm.debug.divisor=10 in your loader.conf. That will reduce impact of memory trashing/checking in UMA by factor of 10. This trashing is the biggest CPU hog enabled by INVARIANTS. With this tunable we are running production with up to 100 Gbit/s.

In D43776#998975, @glebius wrote:

Looks like I'm able to turn INVARIANTS just in that specific TU, let's see if we deploy it into production and narrow it down:

You are able of course, but that isn't going to work.

Looks like I'm able to turn INVARIANTS just in that specific TU, let's see if we deploy it into production and narrow it down:

Crash trace seems to be misleading, based on the crash IP (0xffffffff80664190), the crash is inside m_apply():

Dump of assembler code for function m_apply:
   0xffffffff80664100 <+0>:     push   %rbp
   0xffffffff80664101 <+1>:     mov    %rsp,%rbp
   0xffffffff80664104 <+4>:     push   %r15
   0xffffffff80664106 <+6>:     push   %r14
   0xffffffff80664108 <+8>:     push   %r13
   0xffffffff8066410a <+10>:    push   %r12
   0xffffffff8066410c <+12>:    push   %rbx
   0xffffffff8066410d <+13>:    sub    $0x18,%rsp
   0xffffffff80664111 <+17>:    mov    %r8,-0x38(%rbp)
   0xffffffff80664115 <+21>:    mov    %rcx,-0x30(%rbp)
   0xffffffff80664119 <+25>:    mov    %edx,%r12d
   0xffffffff8066411c <+28>:    mov    %rdi,%rbx
   0xffffffff8066411f <+31>:    test   %esi,%esi
   0xffffffff80664121 <+33>:    jle    0xffffffff80664140 <m_apply+64>
   0xffffffff80664123 <+35>:    mov    %esi,%eax
   0xffffffff80664125 <+37>:    cs nopw 0x0(%rax,%rax,1)
   0xffffffff8066412f <+47>:    nop
   0xffffffff80664130 <+48>:    sub    0x18(%rbx),%eax
   0xffffffff80664133 <+51>:    jl     0xffffffff80664149 <m_apply+73>
   0xffffffff80664135 <+53>:    mov    (%rbx),%rbx
   0xffffffff80664138 <+56>:    mov    %eax,%esi
   0xffffffff8066413a <+58>:    test   %eax,%eax
   0xffffffff8066413c <+60>:    jg     0xffffffff80664130 <m_apply+48>
   0xffffffff8066413e <+62>:    jmp    0xffffffff80664142 <m_apply+66>
   0xffffffff80664140 <+64>:    mov    %esi,%eax
   0xffffffff80664142 <+66>:    test   %r12d,%r12d
   0xffffffff80664145 <+69>:    jg     0xffffffff80664190 <m_apply+144>
   0xffffffff80664147 <+71>:    jmp    0xffffffff80664150 <m_apply+80>
   0xffffffff80664149 <+73>:    mov    %esi,%eax
   0xffffffff8066414b <+75>:    test   %r12d,%r12d
   0xffffffff8066414e <+78>:    jg     0xffffffff80664190 <m_apply+144>
   0xffffffff80664150 <+80>:    xor    %eax,%eax
   0xffffffff80664152 <+82>:    add    $0x18,%rsp
   0xffffffff80664156 <+86>:    pop    %rbx
   0xffffffff80664157 <+87>:    pop    %r12
   0xffffffff80664159 <+89>:    pop    %r13
   0xffffffff8066415b <+91>:    pop    %r14
   0xffffffff8066415d <+93>:    pop    %r15
   0xffffffff8066415f <+95>:    pop    %rbp
   0xffffffff80664160 <+96>:    ret
   0xffffffff80664161 <+97>:    cs nopw 0x0(%rax,%rax,1)
   0xffffffff8066416b <+107>:   nopl   0x0(%rax,%rax,1)
   0xffffffff80664170 <+112>:   cltq
   0xffffffff80664172 <+114>:   add    %rax,%rsi
   0xffffffff80664175 <+117>:   mov    -0x38(%rbp),%rdi
   0xffffffff80664179 <+121>:   mov    %r14d,%edx
   0xffffffff8066417c <+124>:   call   *-0x30(%rbp)
   0xffffffff8066417f <+127>:   test   %eax,%eax
   0xffffffff80664181 <+129>:   jne    0xffffffff80664152 <m_apply+82>
   0xffffffff80664183 <+131>:   sub    %r14d,%r12d
   0xffffffff80664186 <+134>:   mov    (%rbx),%rbx
   0xffffffff80664189 <+137>:   xor    %eax,%eax
   0xffffffff8066418b <+139>:   test   %r12d,%r12d
   0xffffffff8066418e <+142>:   jle    0xffffffff80664152 <m_apply+82>
   0xffffffff80664190 <+144>:   mov    0x18(%rbx),%r14d
                                ^^^^^^^^^^^^^^
End of assembler dump.

Different calls chain though, @markj any ideas? 👀

Looks like some other issue at play, I am seeing those crashes even with my patch applied.

In D43776#998736, @glebius wrote:

Withdraw as I was wrong with my quick conclusions.

o sorted alphabetically;
o updated .Dd.

P.S. popen(3) under the hood uses /bin/sh, so it needs tons of stuff to be present and working. Not to mention having fstat around is not guranteed.

Looks OKish (I never used that feature myself, only for testing), but the fact that this is now runs by default and tries to do bunch of unnecessary magic under the hood is of a big concern. I already have a problem with pkg starting to open /dev/null just in case, I don't want this to cause my builds to start suddenly failing right and left.

In D40025#911514, @kevans wrote:

In D40025#911513, @emaste wrote:

mfctracker already matches dashes re.match('^\s*mfc(-|\s+)after\s*:', line, flags=re.IGNORECASE)

https://github.com/gonzoua/mfctracker/blob/77800ee5ab17e4efefb12bfbd547bff1ed94c6a1/mfctracker/management/commands/importcommits.py#L181

yeah, I did both of those at the same time... a bit unusual that @sobomax never picked that one up, actually

In D40025#911506, @emaste wrote:

@kevans has a pull request open to add - support to the MFC notification service https://github.com/sobomax/mfc_notifications/pull/3

Looks good.

termcap.small is still useful I think for the case when /usr is not mounted (i.e. single-user/recovery mode in a traditional FS layout). These days of course most people are using monolithic root+usr esp with ZFS, however we for example have 200+ systems in the field partitioned in a traditional way, so I am wondering how many there may be people like us out there.