Page MenuHomeFreeBSD

gallatin (Andrew Gallatin)
User

Projects

User Details

User Since
Jun 22 2015, 5:21 PM (199 w, 4 d)

Recent Activity

Thu, Apr 18

gallatin updated the diff for D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.

Removed redundant parens around bus_get_domain() call as pointed out by markj

Thu, Apr 18, 3:18 PM
gallatin added a reviewer for D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory: scottl.
Thu, Apr 18, 12:43 PM

Wed, Apr 17

gallatin added inline comments to D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.
Wed, Apr 17, 5:54 PM
gallatin updated the diff for D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.

Address kib's feedback

  • remove unneeded _domainset.h include
  • re-write if_alloc_dev() to be more concise
Wed, Apr 17, 5:53 PM
gallatin updated the diff for D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.
  • Added if_alloc_dev() and changed example mlx5en and cxgbe code to use them, as suggested by kib.
  • Added links to if_alloc_domain and if_alloc_dev for man pages
  • Fixed style issues pointed out by kib
Wed, Apr 17, 3:34 PM
gallatin added a comment to D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.
In D19930#428329, @kib wrote:

I wonder if a KPI of the format

struct ifnet *if_alloc_dev(u_char type, dev_t device);

would be more useful. It would hide all NUMA/non-NUMA/bus_get_domain() failed details that driver authors will copy/paste ad infinitum.
I do not propose to not add if_alloc_domain(), just think that if_alloc_dev() should be the primari KPI. Also it might allow to get more device_t context in if_alloc().

Wed, Apr 17, 12:21 PM
gallatin added a reviewer for D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory: glebius.
Wed, Apr 17, 12:09 AM
gallatin created D19930: Track device's NUMA domain in ifnet & allocate ifnet from NUMA local memory.
Wed, Apr 17, 12:09 AM

Tue, Apr 16

gallatin closed D19862: replace cosqos field in pkthdr with a numa domain tag.
Tue, Apr 16, 4:49 PM
gallatin committed rS346281: Replace cosqos with numa_domain in mbuf pkthdr.
Replace cosqos with numa_domain in mbuf pkthdr
Tue, Apr 16, 4:49 PM

Mon, Apr 15

gallatin committed rS346247: mlx5en: Enable new pfil(9) KPI ethernet filtering hooks.
mlx5en: Enable new pfil(9) KPI ethernet filtering hooks
Mon, Apr 15, 5:15 PM
gallatin closed D19063: mlx5en: add pfil ethernet hook.
Mon, Apr 15, 5:15 PM

Tue, Apr 9

gallatin updated the summary of D19862: replace cosqos field in pkthdr with a numa domain tag.
Tue, Apr 9, 3:46 PM
gallatin created D19862: replace cosqos field in pkthdr with a numa domain tag.
Tue, Apr 9, 3:43 PM

Thu, Mar 28

gallatin added inline comments to D19645: add pfil hooks to iflib.
Thu, Mar 28, 2:26 PM
gallatin updated the diff for D19645: add pfil hooks to iflib.

Address review feedback from marius & shurd

Thu, Mar 28, 2:26 PM

Wed, Mar 27

gallatin accepted D19719: Use a dedicated malloc type for lagg(4)'s structures..

YES! Death to M_DEVBUF!

Wed, Mar 27, 1:15 PM
gallatin accepted D19718: Remove nested epochs from lagg(4)..
Wed, Mar 27, 1:14 PM

Mar 19 2019

gallatin updated the diff for D19645: add pfil hooks to iflib.

Fixed comments, as pointed out by ejg

Mar 19 2019, 6:27 PM
gallatin created D19645: add pfil hooks to iflib.
Mar 19 2019, 2:56 PM

Mar 18 2019

gallatin committed rS345273: Fix a typo introduced in r344133.
Fix a typo introduced in r344133
Mar 18 2019, 12:42 PM

Mar 16 2019

gallatin accepted D19577: iflib: mark isc_driver_version as constant.
Mar 16 2019, 10:20 PM

Mar 14 2019

gallatin accepted D19468: iflib: prevent possible infinite loop in iflib_encap.
Mar 14 2019, 5:56 PM

Mar 13 2019

gallatin accepted D19468: iflib: prevent possible infinite loop in iflib_encap.

I think this is a mostly theoretical issue, as you'll have had a chain which was impossible to send (eg, far too large, etc) even after defrag.

Mar 13 2019, 6:42 PM

Mar 10 2019

gallatin updated the diff for D19063: mlx5en: add pfil ethernet hook.
  • Use new pfil_mem2mbuf() to avoid unaligned access of mbuf pointer
  • Remove unneeded breaks
Mar 10 2019, 6:21 PM

Feb 25 2019

gallatin accepted D19315: Dynamically allocate per-cpu scheduler state.

Much cleaner, as usual, than the hack I gave you to start with.

Feb 25 2019, 4:48 PM

Feb 21 2019

gallatin updated the diff for D19063: mlx5en: add pfil ethernet hook.

Guard against null pfil at device detach by deregistering pfil hook later. While here, check for a null pfil hook and cache it in a local to make the code easier to read.

Feb 21 2019, 5:40 PM

Feb 15 2019

gallatin accepted D19201: iflib: Return the correct filter result in the interrupt handler..
Feb 15 2019, 12:52 PM

Feb 14 2019

gallatin added a comment to D19201: iflib: Return the correct filter result in the interrupt handler..

I understand the missing FILTER_STRAY.

Feb 14 2019, 8:21 PM

Feb 8 2019

gallatin updated the diff for D19063: mlx5en: add pfil ethernet hook.

Updated to truncate the length passed to pfil for multi-segment jumbo frames to just the length of the first segment.

Feb 8 2019, 3:13 PM

Feb 7 2019

gallatin added a comment to D19063: mlx5en: add pfil ethernet hook.

Any high performance interface will look similar to this and be similarly invasive.

Feb 7 2019, 9:27 PM
gallatin added a comment to D19063: mlx5en: add pfil ethernet hook.

The reason to have this in the driver is for performance. The plan for pfil is to eventually just use a pointer to a memory blob for filtering. The current fake mbuf stuff is a step in that direction.

Feb 7 2019, 6:35 PM

Feb 6 2019

gallatin added inline comments to D19063: mlx5en: add pfil ethernet hook.
Feb 6 2019, 3:58 PM
gallatin updated the diff for D19063: mlx5en: add pfil ethernet hook.
  • moved rv declaration to top of function
  • entered CURVNET to prevent panic from null curvnet when VIMAGE is compiled in Note placing this at the top of the rx function is a trade off between tiny overhead in the common case (nothing hooked) and a bit larger overhead in the case when something is hooked, and entering/restoring the VNET each time we call into the filter.
  • Addressed all possible return values from pfil_run_hooks Note that PFIL_REALLOCED was tested by hacking pfil.c to copy packets
  • Added a label to jump to when PFIL_REALLOCED is returned, and we need to send up an mbuf allocated by the filter.
Feb 6 2019, 3:56 PM

Feb 2 2019

gallatin created D19063: mlx5en: add pfil ethernet hook.
Feb 2 2019, 3:29 PM

Jan 31 2019

gallatin accepted D19040: Don't set IFCAP_TXRTLMT during lagg_clone_create()..
Jan 31 2019, 12:15 AM

Jan 28 2019

gallatin committed rS343526: MFC r343430.
MFC r343430
Jan 28 2019, 2:35 PM

Jan 25 2019

gallatin committed rS343430: Fix an iflib driver unload panic introduced in r343085.
Fix an iflib driver unload panic introduced in r343085
Jan 25 2019, 3:02 PM

Jan 22 2019

gallatin committed rS343304: MFC r341095:.
MFC r341095:
Jan 22 2019, 5:35 PM

Jan 4 2019

gallatin committed rS342774: Limit git history searches in newvers.sh.
Limit git history searches in newvers.sh
Jan 4 2019, 6:38 PM
gallatin closed D18745: Limit git history searches in newvers.sh.
Jan 4 2019, 6:38 PM
gallatin updated the diff for D18745: Limit git history searches in newvers.sh.

Remove XXX as per wlosh

Jan 4 2019, 5:02 PM
gallatin updated the diff for D18745: Limit git history searches in newvers.sh.
Jan 4 2019, 4:25 PM
gallatin created D18745: Limit git history searches in newvers.sh.
Jan 4 2019, 4:23 PM

Jan 1 2019

gallatin accepted D18690: Reduce overhead of per-packet processing by ipfw(4).
Jan 1 2019, 9:10 PM

Dec 30 2018

gallatin accepted D18690: Reduce overhead of per-packet processing by ipfw(4).

I really like collapsing all those pointers, and making the NULL checks flags. Great work.

Dec 30 2018, 9:10 PM

Dec 15 2018

gallatin added a comment to D18532: Use byte-counting rather than packet counting for TX batch size.

I like this much better; hopefully the performance tests will show that it is an improvement.

Dec 15 2018, 1:39 PM

Dec 14 2018

gallatin added inline comments to D18532: Use byte-counting rather than packet counting for TX batch size.
Dec 14 2018, 1:26 AM

Nov 27 2018

gallatin closed D17901: Use busdma unconditionally in iflib.
Nov 27 2018, 8:02 PM
gallatin committed rS341095: Use busdma unconditionally in iflib.
Use busdma unconditionally in iflib
Nov 27 2018, 8:02 PM

Nov 16 2018

gallatin accepted D17998: Drop sbsndptr KPI.

I was about to say that cxgbe still uses it, but it looks like that was just changed in r340465. So yes, lets get rid of it.

Nov 16 2018, 2:05 PM

Nov 10 2018

gallatin updated the diff for D17901: Use busdma unconditionally in iflib.
  • Fixed a pre-existing bug where, when receiving small packets, rx clusters were not unmapped, yet they were re-mapped when refilling the ring. This was fixed by keeping track of the bus address in ifsd_ba, and simply re-using them rather than re-doing the virt to bus mapping
Nov 10 2018, 2:27 PM

Nov 9 2018

gallatin added a comment to D17901: Use busdma unconditionally in iflib.

Thanks Olivier. 5% is about what I had expected. I think I see a way to improve that though. We seem to be doing repeated virt to phys translation on clusters where we have copied out a small mbuf on the rx side in iflib_rxd_pkt_get(). We call rxd_frag_to_sd() with a FALSE arg, to prevent unmapping. Yet in _iflib_fl_refill(), we seem to always do the mapping, even when we do not re-allocate clusters. This is one bit that's going to be more expensive on low-end boxes, since it will now result in a busdma callback function being called. I think this is an actual bug that would impact systems w/IOMMUs

Nov 9 2018, 3:09 PM

Nov 8 2018

gallatin created D17901: Use busdma unconditionally in iflib.
Nov 8 2018, 12:06 AM

Oct 26 2018

gallatin accepted D17704: Handle an empty NUMA domain 0..
Oct 26 2018, 1:41 PM

Oct 23 2018

gallatin added a comment to D17665: Ensure mbuf send tags don't trigger on ifnets.
In D17665#377526, @np wrote:
Oct 23 2018, 8:26 PM

Oct 19 2018

gallatin accepted D17416: Initialize global domainsets during boot..
Oct 19 2018, 12:37 PM
gallatin added inline comments to D17418: Make malloc_domain(9) fall back to other domains..
Oct 19 2018, 12:30 PM

Oct 11 2018

gallatin added a comment to D17419: Use malloc_domainset(9) instead of malloc_domain(M_WAITOK)..

Stepping back for a second, are we going to get to a point where we can make malloc() and uma_zalloc() use the current thread's domainset policy, rather than one derived from the kernel object?

Oct 11 2018, 1:21 PM
gallatin added inline comments to D17416: Initialize global domainsets during boot..
Oct 11 2018, 1:09 PM

Oct 10 2018

gallatin accepted D17503: Run epoch calls sooner and more reliably.

FWIW, i have patched both of these changes (this + D17492) into the upstream ck git repo, and run them through the CK regression tests for epoch in userpace on my 32-core/64-thread amd, and they passed.

Oct 10 2018, 7:16 PM
gallatin accepted D17492: Fix early epoch calls.
Oct 10 2018, 7:15 PM

Oct 8 2018

gallatin accepted D17439: Make it possible to disable NUMA support with a tunable..
Oct 8 2018, 8:37 PM
gallatin accepted D17416: Initialize global domainsets during boot..
Oct 8 2018, 8:36 PM
gallatin accepted D17474: A couple of style fixes in recent TCP changes..
Oct 8 2018, 7:34 PM
gallatin added a comment to D17416: Initialize global domainsets during boot..

You mention kstacks. I have not gone through the rest of your reviews, but I was wondering if you were working on affinitizing kstacks. Did you plan to break the cache out by domain, or drop it for affinitized kstack allocations? The default is so undersized these days, we're mostly not getting the benefit of the cache anyway (and the cache might be pessimal, since it is protected by a global lock).

I wasn't working on affinity. Rather, I want to ensure that the kstack allocation policy allows allocations from any domain, ensuring that we can permit kstack swapins so long as at least one domain has some free pages. See D17304; it's really just a bug fix for 12.0.

I haven't thought much about the problems you listed. Do you have a proposal?

Oct 8 2018, 5:59 PM
gallatin added inline comments to D17416: Initialize global domainsets during boot..
Oct 8 2018, 5:55 PM
gallatin added inline comments to D17417: Refactor domainset iterators in preparation for use by malloc() and UMA..
Oct 8 2018, 2:44 PM
gallatin accepted D17416: Initialize global domainsets during boot..

You mention kstacks. I have not gone through the rest of your reviews, but I was wondering if you were working on affinitizing kstacks. Did you plan to break the cache out by domain, or drop it for affinitized kstack allocations? The default is so undersized these days, we're mostly not getting the benefit of the cache anyway (and the cache might be pessimal, since it is protected by a global lock).

Oct 8 2018, 12:58 PM

Oct 6 2018

gallatin accepted D17450: Fix three epoch leaks.
Oct 6 2018, 8:11 PM

Oct 5 2018

gallatin added a comment to D17418: Make malloc_domain(9) fall back to other domains..

Do we really need a new malloc variant? Couldn't this be done with a new malloc flag that was specific to malloc_domain(). Something like M_ALTOK or something? (or maybe M_EXACT if you want the default to be the other way around, and have malloc_domain() fall back by default)

Oct 5 2018, 4:51 PM

Oct 1 2018

gallatin closed D16836: Allow empty NUMA domains to support AMD 2990WX.

This was committed as r339043. Due to a cut and paste error, the wrong review was linked from the commit message, so the review was not auto-closed.

Oct 1 2018, 3:09 PM
gallatin committed rS339043: Allow empty NUMA memory domains to support Threadripper2.
Allow empty NUMA memory domains to support Threadripper2
Oct 1 2018, 2:14 PM

Sep 24 2018

gallatin added inline comments to D16836: Allow empty NUMA domains to support AMD 2990WX.
Sep 24 2018, 9:20 PM
gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

kmem_back() now also needs updating.

Sep 24 2018, 1:12 PM
gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.
  • Fixed spelling errors pointed out by alc
  • Updated to account for recent VM changes:
    • added empty domain guard in kmm_back()
    • fixed conflict in keg_fetch_slab() after r338755
Sep 24 2018, 1:11 PM

Sep 19 2018

gallatin accepted D17242: Add IFCAP_TSO6 for igb.
Sep 19 2018, 6:05 PM

Sep 17 2018

gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.

Added check for empty domains in uma_large_malloc_domain(), and removed added blank line, as pointed out by Mark

Sep 17 2018, 4:55 PM

Sep 15 2018

gallatin accepted D17158: Fix capabilities handling for iflib drivers.
Sep 15 2018, 3:32 PM

Sep 13 2018

gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

We already have code to renumber NUMA domains to avoid holes in the domain ID space (renumber_domains() in srat.c). Wouldn't it be simpler to apply a similar approach to empty domains, or is there some reason that won't work?

Sep 13 2018, 1:34 PM

Sep 10 2018

gallatin accepted D17109: Style fixes for in6_pcblookup_lbgroup()..
Sep 10 2018, 6:41 PM
gallatin added a comment to D17031: Fix synchronization of lbgroup access..

FWIW, as I mentioned in private email, the testing went fine. I verified that there were no apparent mem leaks, and that in_pcblbgroup_free_deferred() was called when I bounced nginx several times on a heavily loaded (> 90Gb/s) box.

Sep 10 2018, 6:40 PM

Sep 6 2018

gallatin accepted D17059: Enable options NUMA on amd64 GENERIC/MINIMAL:.

We've had it in our config at Netflix for so long that I forgot that it was not in GENERIC..

Sep 6 2018, 1:00 PM · NUMA

Aug 29 2018

gallatin accepted D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

For what it is, Van's birthday might be a better choice. Eg, some non-valid pointer that would cause free or mfree to panic rather than accept it and corrupt data.

Aug 29 2018, 10:08 AM

Aug 28 2018

gallatin added a comment to D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

Sorry, having the ring pointer in the ring is just so odd.. Can you explain why it is there? Maybe it would be better to just not have the ring pointer in the ring and not have to have all these special cases.

Aug 28 2018, 10:58 PM
gallatin added a comment to D16882: Handle case where mbuf == &txq when IFC_QFLUSH is set.

What keeps the normal, non-IFC_QFLUSH from hitting this condition? Is the txq pointer included in r->size?

Aug 28 2018, 6:20 PM

Aug 27 2018

gallatin committed rS338341: Reject IPv4 SO_REUSEPORT_LB groups when looking up an IPv6 listening socket.
Reject IPv4 SO_REUSEPORT_LB groups when looking up an IPv6 listening socket
Aug 27 2018, 6:13 PM
gallatin closed D16899: SO_REUSEPORT_LB critical bug fix: avoid IPv4 listen group in IPv6.
Aug 27 2018, 6:13 PM

Aug 25 2018

gallatin added inline comments to D16855: Fix temperature reporting on AMD 2990WX.
Aug 25 2018, 11:10 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 25 2018, 11:09 PM
gallatin created D16899: SO_REUSEPORT_LB critical bug fix: avoid IPv4 listen group in IPv6.
Aug 25 2018, 2:43 PM

Aug 23 2018

gallatin accepted D16875: Possible fix for "desc avail = " dmesg spam and igb breakage.
Aug 23 2018, 11:57 PM
gallatin added a comment to D16836: Allow empty NUMA domains to support AMD 2990WX.

I haven't had much of a chance to review this yet, but I wanted to test an "options NUMA" kernel on a system with an empty domain (dual socket Haswell xeon with only one DIMM installed at the moment). This patch worked fine for me. Thanks!

Aug 23 2018, 9:37 PM
gallatin added inline comments to D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 23 2018, 6:25 PM
gallatin updated the diff for D16836: Allow empty NUMA domains to support AMD 2990WX.
Aug 23 2018, 6:23 PM
gallatin added a comment to D16855: Fix temperature reporting on AMD 2990WX.

Thanks for the clarification on the non-offset range!

Aug 23 2018, 6:19 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 23 2018, 6:18 PM
gallatin updated the diff for D16855: Fix temperature reporting on AMD 2990WX.
Aug 23 2018, 2:29 PM
gallatin added a comment to D16855: Fix temperature reporting on AMD 2990WX.

Thank you so much for the pointer to the docs!!! It is much nicer to write from real docs, rather than crib from undocumented changes in another project.

Aug 23 2018, 2:22 PM
gallatin accepted D16733: iflib sysctl cleanup.
Aug 23 2018, 12:46 PM