Page MenuHomeFreeBSD

gallatin (Andrew Gallatin)
User

Projects

User Details

User Since
Jun 22 2015, 5:21 PM (180 w, 6 d)

Recent Activity

Tue, Nov 27

gallatin closed D17901: Use busdma unconditionally in iflib.
Tue, Nov 27, 8:02 PM
gallatin committed rS341095: Use busdma unconditionally in iflib.
Use busdma unconditionally in iflib
Tue, Nov 27, 8:02 PM

Fri, Nov 16

gallatin accepted D17998: Drop sbsndptr KPI.

I was about to say that cxgbe still uses it, but it looks like that was just changed in r340465. So yes, lets get rid of it.

Fri, Nov 16, 2:05 PM

Sat, Nov 10

gallatin updated the diff for D17901: Use busdma unconditionally in iflib.
  • Fixed a pre-existing bug where, when receiving small packets, rx clusters were not unmapped, yet they were re-mapped when refilling the ring. This was fixed by keeping track of the bus address in ifsd_ba, and simply re-using them rather than re-doing the virt to bus mapping
Sat, Nov 10, 2:27 PM

Nov 9 2018

gallatin added a comment to D17901: Use busdma unconditionally in iflib.

Thanks Olivier. 5% is about what I had expected. I think I see a way to improve that though. We seem to be doing repeated virt to phys translation on clusters where we have copied out a small mbuf on the rx side in iflib_rxd_pkt_get(). We call rxd_frag_to_sd() with a FALSE arg, to prevent unmapping. Yet in _iflib_fl_refill(), we seem to always do the mapping, even when we do not re-allocate clusters. This is one bit that's going to be more expensive on low-end boxes, since it will now result in a busdma callback function being called. I think this is an actual bug that would impact systems w/IOMMUs

Nov 9 2018, 3:09 PM

Nov 8 2018

gallatin created D17901: Use busdma unconditionally in iflib.
Nov 8 2018, 12:06 AM

Oct 26 2018

gallatin accepted D17704: Handle an empty NUMA domain 0..
Oct 26 2018, 1:41 PM

Oct 23 2018

gallatin added a comment to D17665: Ensure mbuf send tags don't trigger on ifnets.
In D17665#377526, @np wrote:
Oct 23 2018, 8:26 PM

Oct 19 2018

gallatin accepted D17416: Initialize global domainsets during boot..
Oct 19 2018, 12:37 PM
gallatin added inline comments to D17418: Make malloc_domain(9) fall back to other domains..
Oct 19 2018, 12:30 PM

Oct 11 2018

gallatin added a comment to D17419: Use malloc_domainset(9) instead of malloc_domain(M_WAITOK)..

Stepping back for a second, are we going to get to a point where we can make malloc() and uma_zalloc() use the current thread's domainset policy, rather than one derived from the kernel object?

Oct 11 2018, 1:21 PM
gallatin added inline comments to D17416: Initialize global domainsets during boot..
Oct 11 2018, 1:09 PM

Oct 10 2018

gallatin accepted D17503: Run epoch calls sooner and more reliably.

FWIW, i have patched both of these changes (this + D17492) into the upstream ck git repo, and run them through the CK regression tests for epoch in userpace on my 32-core/64-thread amd, and they passed.

Oct 10 2018, 7:16 PM
gallatin accepted D17492: Fix early epoch calls.
Oct 10 2018, 7:15 PM

Oct 8 2018

gallatin accepted D17439: Make it possible to disable NUMA support with a tunable..
Oct 8 2018, 8:37 PM
gallatin accepted D17416: Initialize global domainsets during boot..
Oct 8 2018, 8:36 PM
gallatin accepted D17474: A couple of style fixes in recent TCP changes..
Oct 8 2018, 7:34 PM
gallatin added a comment to D17416: Initialize global domainsets during boot..

You mention kstacks. I have not gone through the rest of your reviews, but I was wondering if you were working on affinitizing kstacks. Did you plan to break the cache out by domain, or drop it for affinitized kstack allocations? The default is so undersized these days, we're mostly not getting the benefit of the cache anyway (and the cache might be pessimal, since it is protected by a global lock).

I wasn't working on affinity. Rather, I want to ensure that the kstack allocation policy allows allocations from any domain, ensuring that we can permit kstack swapins so long as at least one domain has some free pages. See D17304; it's really just a bug fix for 12.0.

I haven't thought much about the problems you listed. Do you have a proposal?

Oct 8 2018, 5:59 PM
gallatin added inline comments to D17416: Initialize global domainsets during boot..
Oct 8 2018, 5:55 PM
gallatin added inline comments to D17417: Refactor domainset iterators in preparation for use by malloc() and UMA..
Oct 8 2018, 2:44 PM
gallatin accepted D17416: Initialize global domainsets during boot..

You mention kstacks. I have not gone through the rest of your reviews, but I was wondering if you were working on affinitizing kstacks. Did you plan to break the cache out by domain, or drop it for affinitized kstack allocations? The default is so undersized these days, we're mostly not getting the benefit of the cache anyway (and the cache might be pessimal, since it is protected by a global lock).

Oct 8 2018, 12:58 PM

Oct 6 2018

gallatin accepted D17450: Fix three epoch leaks.
Oct 6 2018, 8:11 PM

Oct 5 2018

gallatin added a comment to D17418: Make malloc_domain(9) fall back to other domains..

Do we really need a new malloc variant? Couldn't this be done with a new malloc flag that was specific to malloc_domain(). Something like M_ALTOK or something? (or maybe M_EXACT if you want the default to be the other way around, and have malloc_domain() fall back by default)

Oct 5 2018, 4:51 PM

Oct 1 2018

gallatin closed D16836: Allow empty NUMA domains to support AMD 2990WX.

This was committed as r339043. Due to a cut and paste error, the wrong review was linked from the commit message, so the review was not auto-closed.

Oct 1 2018, 3:09 PM
gallatin committed rS339043: Allow empty NUMA memory domains to support Threadripper2.
Allow empty NUMA memory domains to support Threadripper2
Oct 1 2018, 2:14 PM

Sep 24 2018

gallatin added inline comments to D16836: Allow empty NUMA domains to support AMD 2990WX.
Sep 24 2018, 9:20 PM
gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

kmem_back() now also needs updating.

Sep 24 2018, 1:12 PM
gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.
  • Fixed spelling errors pointed out by alc
  • Updated to account for recent VM changes:
    • added empty domain guard in kmm_back()
    • fixed conflict in keg_fetch_slab() after r338755
Sep 24 2018, 1:11 PM

Sep 19 2018

gallatin accepted D17242: Add IFCAP_TSO6 for igb.
Sep 19 2018, 6:05 PM

Sep 17 2018

gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.

Added check for empty domains in uma_large_malloc_domain(), and removed added blank line, as pointed out by Mark

Sep 17 2018, 4:55 PM

Sep 15 2018

gallatin accepted D17158: Fix capabilities handling for iflib drivers.
Sep 15 2018, 3:32 PM

Sep 13 2018

gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

We already have code to renumber NUMA domains to avoid holes in the domain ID space (renumber_domains() in srat.c). Wouldn't it be simpler to apply a similar approach to empty domains, or is there some reason that won't work?

Sep 13 2018, 1:34 PM

Sep 10 2018

gallatin accepted D17109: Style fixes for in6_pcblookup_lbgroup()..
Sep 10 2018, 6:41 PM
gallatin added a comment to D17031: Fix synchronization of lbgroup access..

FWIW, as I mentioned in private email, the testing went fine. I verified that there were no apparent mem leaks, and that in_pcblbgroup_free_deferred() was called when I bounced nginx several times on a heavily loaded (> 90Gb/s) box.

Sep 10 2018, 6:40 PM

Sep 6 2018

gallatin accepted D17059: Enable options NUMA on amd64 GENERIC/MINIMAL:.

We've had it in our config at Netflix for so long that I forgot that it was not in GENERIC..

Sep 6 2018, 1:00 PM · NUMA

Aug 29 2018

gallatin accepted D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

For what it is, Van's birthday might be a better choice. Eg, some non-valid pointer that would cause free or mfree to panic rather than accept it and corrupt data.

Aug 29 2018, 10:08 AM

Aug 28 2018

gallatin added a comment to D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

Sorry, having the ring pointer in the ring is just so odd.. Can you explain why it is there? Maybe it would be better to just not have the ring pointer in the ring and not have to have all these special cases.

Aug 28 2018, 10:58 PM
gallatin added a comment to D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

What keeps the normal, non-IFC_QFLUSH from hitting this condition? Is the txq pointer included in r->size?

Aug 28 2018, 6:20 PM

Aug 27 2018

gallatin committed rS338341: Reject IPv4 SO_REUSEPORT_LB groups when looking up an IPv6 listening socket.
Reject IPv4 SO_REUSEPORT_LB groups when looking up an IPv6 listening socket
Aug 27 2018, 6:13 PM
gallatin closed D16899: SO_REUSEPORT_LB critical bug fix: avoid IPv4 listen group in IPv6.
Aug 27 2018, 6:13 PM

Aug 25 2018

gallatin added inline comments to D16855: Fix temperature reporting on AMD 2990WX.
Aug 25 2018, 11:10 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 25 2018, 11:09 PM
gallatin created D16899: SO_REUSEPORT_LB critical bug fix: avoid IPv4 listen group in IPv6.
Aug 25 2018, 2:43 PM

Aug 23 2018

gallatin accepted D16875: Possible fix for "desc avail = " dmesg spam and igb breakage.
Aug 23 2018, 11:57 PM
gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

I haven't had much of a chance to review this yet, but I wanted to test an "options NUMA" kernel on a system with an empty domain (dual socket Haswell xeon with only one DIMM installed at the moment). This patch worked fine for me. Thanks!

Aug 23 2018, 9:37 PM
gallatin added inline comments to D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 23 2018, 6:25 PM
gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 23 2018, 6:23 PM
gallatin added a comment to D16855: Fix temperature reporting on AMD 2990WX.

Thanks for the clarification on the non-offset range!

Aug 23 2018, 6:19 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 23 2018, 6:18 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 23 2018, 2:29 PM
gallatin added a comment to D16855: Fix temperature reporting on AMD 2990WX.

Thank you so much for the pointer to the docs!!! It is much nicer to write from real docs, rather than crib from undocumented changes in another project.

Aug 23 2018, 2:22 PM
gallatin accepted D16733: iflib sysctl cleanup.
Aug 23 2018, 12:46 PM
gallatin created D16855: Fix temperature reporting on AMD 2990WX.
Aug 23 2018, 12:06 AM

Aug 22 2018

gallatin added inline comments to D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 22 2018, 5:45 PM

Aug 21 2018

gallatin created D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 21 2018, 8:39 PM

Aug 17 2018

gallatin added inline comments to D16429: ixlv(4): Update to use iflib; change name to iavf(4).
Aug 17 2018, 7:35 PM · Intel Networking

Aug 16 2018

gallatin added inline comments to D16733: iflib sysctl cleanup.
Aug 16 2018, 1:03 PM

Aug 15 2018

gallatin added a reviewer for D16731: if_media: Add new 2.5G/5G/25G/40G/50G/100G/200G/400G media types: np.
Aug 15 2018, 7:03 PM

Aug 13 2018

gallatin committed rS337709: lagg: allow lacp to manage the link state.
lagg: allow lacp to manage the link state
Aug 13 2018, 2:13 PM

Aug 10 2018

gallatin accepted D16429: ixlv(4): Update to use iflib; change name to iavf(4).
Aug 10 2018, 3:41 PM · Intel Networking

Jul 30 2018

gallatin added a comment to D16429: ixlv(4): Update to use iflib; change name to iavf(4).

In general, I looked around at ixlv.c, and it looks like there are many functions for the main pf which are identical, or nearly identical for the VF. Can we please work on collapsing them down to common code?

Jul 30 2018, 4:32 PM · Intel Networking

Jul 17 2018

gallatin accepted D16302: Add knob to control tx ring abdication..

Thank for doing this!

Jul 17 2018, 7:10 PM

Jul 5 2018

gallatin committed rS335973: mxge: replace 65536 with IP_MAXPACKET in tso settings..
mxge: replace 65536 with IP_MAXPACKET in tso settings.
Jul 5 2018, 2:43 AM

Jul 4 2018

gallatin committed rS335967: mxge: choose appropriate values for hw tso.
mxge: choose appropriate values for hw tso
Jul 4 2018, 7:29 PM
gallatin committed rS335966: mxge: Add SIOCGI2C support for devices with SFP/XFP cages.
mxge: Add SIOCGI2C support for devices with SFP/XFP cages
Jul 4 2018, 6:54 PM
gallatin committed rS335957: mxge: fix panic at module unload.
mxge: fix panic at module unload
Jul 4 2018, 2:26 PM

Jun 27 2018

gallatin added a comment to D15985: Reduce unnecessary preemption, add a preemption knob for timeshare, fix missing NEEDRESCHED.

I ran this on a Netflix 100g box. I observed no measurable difference in CPU time. So I think this patch is "neutral" for the perspective of our (mostly kernel) workload.

Jun 27 2018, 1:32 PM

Jun 20 2018

gallatin added a comment to D15937: Optimize the TSO and copy paths to use the new tcp_m_copy routine.

I've deployed a variant of this widely at LLNW on the default stack as well. I think you can garbage collect sbsndptr() from the tree.

Jun 20 2018, 9:30 PM
gallatin accepted D15937: Optimize the TSO and copy paths to use the new tcp_m_copy routine.

I added Hans since it is mostly his code we are factoring out.

Jun 20 2018, 8:40 PM
gallatin added a reviewer for D15937: Optimize the TSO and copy paths to use the new tcp_m_copy routine: hselasky.
Jun 20 2018, 8:39 PM

Jun 14 2018

gallatin added a comment to D15686: convert inpcbhash rlock to epoch.

if @jch cannot run his real workload, what about running a synthetic test that creates / tears down many connections per second? Eg, something like many copies of netperf -t TCP_CC ?

Jun 14 2018, 4:23 PM
gallatin accepted D15577: Update ixl(4) to use iflib..
Jun 14 2018, 2:53 AM
gallatin added a comment to D15577: Update ixl(4) to use iflib..

You might want to go for some middle ground here. 1ms is a bit much (even for us).

Jun 14 2018, 2:52 AM

Jun 11 2018

gallatin added inline comments to D15577: Update ixl(4) to use iflib..
Jun 11 2018, 11:41 PM

Jun 4 2018

gallatin accepted D15558: iflib: Record TCP checksum info in iflib for ixl(4).

Sure.. Those checks for TCP in the TSO case should probably turn into asserts eventually..

Jun 4 2018, 8:13 PM · Intel Networking

May 30 2018

gallatin accepted D15526: reduce overhead of entropy collection.

If people want to use ethernet entropy harvesting, can we do something to make it more useful in a separate change? I was going to initially suggest the ether dst addr, but that's not very random either.

May 30 2018, 9:22 PM
gallatin added a reviewer for D15526: reduce overhead of entropy collection: gallatin.
May 30 2018, 9:18 PM

May 29 2018

gallatin added a comment to D15577: Update ixl(4) to use iflib..

This is the sparse detect I'm using. Note how it walks the header, and then the TSO by packet on the wire.

May 29 2018, 9:28 PM
gallatin added a comment to D15577: Update ixl(4) to use iflib..

Somehow my last comment got lost. With real Netflix traffic, the sparse detection logic is not sufficient. When we have kernel TLS, which inserts 13 and 16 byte mbufs around TLS records, we hit MDDs regularly. I think you need to walk the packet stream, not the DMA descriptors, so that you keep the same segmentation used by the hardware. I re-wrote the sparse detection to do this, and our MDDs went away.

May 29 2018, 9:27 PM
gallatin added inline comments to D15577: Update ixl(4) to use iflib..
May 29 2018, 9:25 PM

May 25 2018

gallatin requested changes to D15558: iflib: Record TCP checksum info in iflib for ixl(4).
May 25 2018, 8:27 PM · Intel Networking
gallatin added a comment to D15558: iflib: Record TCP checksum info in iflib for ixl(4).

Was this the patch that we discussed on our call? Does this fix your TX checksum problems on IXL?

Yes and Yes.

Unfortunately, I've now proceeded to the "entire box hangs" portion of the game. Even serial console break-to-debugger is ignored. I'm going to try NMIs next..

May 25 2018, 8:18 PM · Intel Networking
gallatin accepted D15575: iflib: Add new shared flag to iflib: IFLIB_ADMIN_ALWAYS_RUN.
May 25 2018, 8:16 PM
gallatin added inline comments to D15575: iflib: Add new shared flag to iflib: IFLIB_ADMIN_ALWAYS_RUN.
May 25 2018, 6:32 PM

May 24 2018

gallatin added a comment to D15558: iflib: Record TCP checksum info in iflib for ixl(4).

Was this the patch that we discussed on our call? Does this fix your TX checksum problems on IXL?

May 24 2018, 6:20 PM · Intel Networking

May 21 2018

gallatin accepted D15510: Defer inpcb deletion until after a grace period has elapsed.
May 21 2018, 10:04 PM

May 18 2018

gallatin accepted D15409: Protect global ifnet list and short lived ifaddr references with epoch.
May 18 2018, 11:14 PM
gallatin committed rS333793: Teach pmcannotate about $TMPDIR and _PATH_TMP.
Teach pmcannotate about $TMPDIR and _PATH_TMP
May 18 2018, 2:14 PM

May 16 2018

gallatin accepted D15366: Replace if_addr_lock rwlock with epoch + mutex.
May 16 2018, 12:12 AM

May 15 2018

gallatin committed rS333655: Unhook DEBUG_BUFRING from INVARIANTS.
Unhook DEBUG_BUFRING from INVARIANTS
May 15 2018, 11:55 PM

May 13 2018

gallatin accepted D15419: epoch(9) man page.

Looks good for content to me.

May 13 2018, 11:04 PM

May 11 2018

gallatin added a comment to D15366: Replace if_addr_lock rwlock with epoch + mutex.

Under a heavy UDP packet flood this helps quite a bit.

May 11 2018, 6:36 PM

May 10 2018

gallatin committed rS333462: Fix a panic in the IPv6 multicast code..
Fix a panic in the IPv6 multicast code.
May 10 2018, 4:19 PM
gallatin committed rS333459: Fix the build after r333457.
Fix the build after r333457
May 10 2018, 1:19 PM

May 9 2018

gallatin added a reviewer for D15366: Replace if_addr_lock rwlock with epoch + mutex: jtl.
May 9 2018, 3:02 PM
gallatin added a comment to D15366: Replace if_addr_lock rwlock with epoch + mutex.

Looks awesome. My only issue is that I'd strongly prefer that the STAILQ_HEAD / STAILQ_INIT / STAILQ_ENTRY macros be prefixed with CK_ so that readers realize that these lists are used with epochs, and cannot be safely used with the normal STAILQ macros. I tried to point out most of them, but I may have missed a few.

May 9 2018, 3:02 PM
gallatin added inline comments to D15365: simple preempt safe epoch API.
May 9 2018, 1:56 PM
gallatin accepted D15354: MFC iflib bugfixes.
May 9 2018, 1:45 PM

May 8 2018

gallatin added a comment to D15345: Add support for packet batching in ifnet.

It would be nice to have this patch together with at least one example of a caller and one example of a NIC driver implementing the API.

May 8 2018, 2:09 PM
gallatin added a reviewer for D15345: Add support for packet batching in ifnet: hselasky.
May 8 2018, 1:25 PM