Implement fast path for malloc and free

[mjg]

Summary

Avoids func calls and branches in the common case.

malloc+free throughput per second: ~19323918 to ~27418181.

lock1 from will-it-scale (single-threaded): 2196902 -> 2345506

Test Plan

make universe. running with DEBUG_REDZONE and DEBUG_MEMGUARD

[jeff]

Does this not include the malloc.h changes for M_ZERO?

Can we do constant size at the same time?

sys/kern/kern_malloc.c

656–662

This should be a uma_int.h inline.

867–871

Should also be an inline. Possibly with some of the code above.

[mjg]

• rebased

[mjg]

The M_ZERO stuff went in quite some time ago.

I plan to deal with static one lookup later.

I don't think the inlined cache bump matters. But if you insist i can move it to a dedicated routine.

[mjg]

Apart from addressing this review request this needs some reworking due to other changes in uma.

sys/kern/kern_malloc.c

if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) {
                 }
         } else
 #endif
         if (sin6 && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) {
                 /*
                  * Given this is either an IPv6-only socket or no INET is
                  * supported we will fail the send if the given destination
                  * address is a v4mapped address.
-                 *
-                 * XXXGL: do we leak m and control?
                  */
                 INP_UNLOCK(inp);
+                m_freem(m);
+                m_freem(control);
                 return (EINVAL);
         }
 
         NET_EPOCH_ENTER(et);
         if (control) {
                 if ((error = ip6_setpktopts(control, &opt,
                     inp->in6p_outputopts, td->td_ucred, nxt)) != 0) {
                         goto release;

sys/vm/uma_core.c

if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) {
                 }
         } else
 #endif
         if (sin6 && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) {
                 /*
                  * Given this is either an IPv6-only socket or no INET is
                  * supported we will fail the send if the given destination
                  * address is a v4mapped address.
-                 *
-                 * XXXGL: do we leak m and control?
                  */
                 INP_UNLOCK(inp);
+                m_freem(m);
+                m_freem(control);
                 return (EINVAL);
         }
 
         NET_EPOCH_ENTER(et);
         if (control) {
                 if ((error = ip6_setpktopts(control, &opt,
                     inp->in6p_outputopts, td->td_ucred, nxt)) != 0) {
                         goto release;

sys/vm/uma_dbg.h

if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) {
                 }
         } else
 #endif
         if (sin6 && IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) {
                 /*
                  * Given this is either an IPv6-only socket or no INET is
                  * supported we will fail the send if the given destination
                  * address is a v4mapped address.
-                 *
-                 * XXXGL: do we leak m and control?
                  */
                 INP_UNLOCK(inp);
+                m_freem(m);
+                m_freem(control);
                 return (EINVAL);
         }
 
         NET_EPOCH_ENTER(et);
         if (control) {
                 if ((error = ip6_setpktopts(control, &opt,
                     inp->in6p_outputopts, td->td_ucred, nxt)) != 0) {
                         goto release;