cxgbe netmap: Directly flushes batches of 64 packets in lazy_tx_credit_flush

[bwicht_verisign.com]

Summary

By default, lazy_tx_credit_flush will process packets by batch of 15
packets or less. With this change, batches or 64 packets are directly
processed. Under high load, this prevents to fill up completely the TX
ring and then to flush it entirely.

Submitted by: Marc De La Gueronniere <[e-mail]>
MFC after: 1 week
Sponsored by: Verisign, Inc.

[brd]

@jhb Can you take a look at this?

[np]

@brd what kind of workload do you see the improvements with?

There are no updates here but I discussed this (and a couple of other t4_netmap.c changes) with the author and submitter at vBSDCon 2019 and we thought it best to leave this as a private change in their repo at that time.

If this helps other users too then we could replace the on/off style lazy_tx_credit_flush with a threshold at which to flush the credits instead of hardcoding it to 64.

[markj]

In D17883#869748, @np wrote:

@brd what kind of workload do you see the improvements with?

I believe Brad was just hoping to see this committed on behalf of the submitter. Apparently this patch has been used in production for several years now.

There are no updates here but I discussed this (and a couple of other t4_netmap.c changes) with the author and submitter at vBSDCon 2019 and we thought it best to leave this as a private change in their repo at that time.

If this helps other users too then we could replace the on/off style lazy_tx_credit_flush with a threshold at which to flush the credits instead of hardcoding it to 64.

Is your suggestion to add a new lazy_tx_credit_flush_thresh sysctl, used as follows:

1033                 if (npkt == 0 && npkt_remaining == 0) {                                                                                                                                                                                                                                                                  
1034                         /* All done. */                                                                                                                                                                                                                                                                                  
1035                         if (lazy_tx_credit_flush == 0) {                                                                                                                                                                                                                                                                 
1036                                 wr->equiq_to_len16 |= htobe32(F_FW_WR_EQUEQ |                                                                                                                                                                                                                                            
1037                                     F_FW_WR_EQUIQ);                                                                                                                                                                                                                                                                      
1038                                 nm_txq->equeqidx = nm_txq->pidx;                                                                                                                                                                                                                                                         
1039                                 nm_txq->equiqidx = nm_txq->pidx;                                                                                                                                                                                                                                                         
1040                         } else if (NMIDXDIFF(nm_txq, equeqidx) >=                                                                                                                                                                                                                                                        
1041                             lazy_tx_credit_flush_thresh) {                                                                                                                                                                                                                                                               
1042                                 wr->equiq_to_len16 |= htobe32(F_FW_WR_EQUEQ);                                                                                                                                                                                                                                            
1043                                 nm_txq->equeqidx = nm_txq->pidx;                                                                                                                                                                                                                                                         
1044                         }                                                                                                                                                                                                                                                                                                
1045                         ring_nm_txq_db(sc, nm_txq);                                                                                                                                                                                                                                                                      
1046                         return;                                                                                                                                                                                                                                                                                          
1047                 }

?