Swap in WKILLED processes.

[kib]

Summary

Swapped-out process that is WKILLED must be swapped in as soon as possible. The reason is that such process can be killed by OOM and its pages can be only freed if the process exits. To exit, the kernel stack of the process must be mapped.

When allocating pages for the stack of the WKILLED process on swap in, use VM_ALLOC_SYSTEM requests to increase the chance of the allocation to succeed.

Add counter of the swapped out processes to avoid unneeded iteration over the allprocs list when there is no work to do, reducing the allproc_lock ownership.

Test Plan

Peter, could you, please, test this change ?

[markj]

sys/vm/vm_swapout.c

650

This can't be done with the non-sleepable proc lock held.

[kib]

sys/vm/vm_swapout.c

650

Yes. Also, the decrement can be done only if P_INMEM was clear.

[kib]

Fix bugs with the decrement of the swapped out counter.

[alc]

sys/kern/kern_sig.c

3081

Wouldn't it make sense to test INMEM, and perhaps SWAPPINGIN, before calling wakeup? Only rarely would the wakeup be needed.

[alc]

sys/vm/vm_swapout.c

607

Do we actually need the oom_swapin flag? Can't this function simply test for the WKILLED flag?

[kib]

Remove faultin1().
Avoid some wakeups in proc_wkilled().
Pass the vm_page_alloc() flag to vm_thread_swapin() instead of oom indicator.

[alc]

sys/vm/vm_swapout.c

692

Now that there is no extra argument to faultin(), can't you eliminate this call, and let swapper() perform it? Also, at the same time, I'm tempted to suggest that the return value be the struct proc * rather than a bool, and that struct proc * could already be locked when you return to swapper().

736–737

Aren't we missing initialization and reset of p to NULL?

[kib]

Restructure the code, hiding all swap in selection in swapper_selector().

[alc]

sys/vm/vm_swapout.c

576–577

Was the insertion of a space after the cast intentional?

[alc]

sys/vm/vm_swapout.c

687

Please add another space after the period.

[kib]

sys/vm/vm_swapout.c

576–577

Yes (almost). It happen, and then I decided to keep it because there seems to be more cases with the space, then without, in sys/vm and sys/ufs. What do you prefer ?

Hm, there is an example in style(9) without space. I will remove it in the next version of the patch.

[alc]

sys/vm/vm_swapout.c

576–577

It was my recollection that style(9) said no space. That's why I mentioned it.

[kib]

Style fixes.

[alc]

This comment applies to both the existing and this new version. I do wonder if it really makes sense to swap in more than one non-killed process at a time. It will take a while for a non-killed process to page in its working set, and until that happens we really don't know if there will be enough free memory for yet another process to be swapped in.

[kib]

In D16489#352169, @alc wrote:

This comment applies to both the existing and this new version. I do wonder if it really makes sense to swap in more than one non-killed process at a time. It will take a while for a non-killed process to page in its working set, and until that happens we really don't know if there will be enough free memory for yet another process to be swapped in.

I will code the serialization of swap in for !WKILLED processes, after this patch is finalized.

Since faultin() is synchronous, we only should prevent swapper from starting swapin until any other thread does it, right ?

[alc]

In D16489#352181, @kib wrote:

In D16489#352169, @alc wrote:

This comment applies to both the existing and this new version. I do wonder if it really makes sense to swap in more than one non-killed process at a time. It will take a while for a non-killed process to page in its working set, and until that happens we really don't know if there will be enough free memory for yet another process to be swapped in.

I will code the serialization of swap in for !WKILLED processes, after this patch is finalized.

Sounds good.

Since faultin() is synchronous, we only should prevent swapper from starting swapin until any other thread does it, right ?

Checking ...

[alc]

In D16489#352342, @alc wrote:

In D16489#352181, @kib wrote:

Since faultin() is synchronous, we only should prevent swapper from starting swapin until any other thread does it, right ?

Checking ...

I'm not 100% sure that I actually understand this question. Can you please elaborate?

[kib]

In D16489#352344, @alc wrote:

In D16489#352342, @alc wrote:

In D16489#352181, @kib wrote:

Since faultin() is synchronous, we only should prevent swapper from starting swapin until any other thread does it, right ?

Checking ...

I'm not 100% sure that I actually understand this question. Can you please elaborate?

I mean that faultin() blocks the calling thread until all stack pages are allocated, read and mapped. So all calls to faultin() from the swapper thread are already serialized. We only need to care about faultin() calls outside the swapper (the only caller is PHOLD()).

So I think that a mechanism that prevents swapper from starting faultin() on non-WKILLED process while there is a thread doing faultin() elsewere, would be good enough.

[pho]

I ran all of the stress2 tests on amd64 and a few on i386.
No problems seen.

sys/kern/kern_sig.c

 /*      $OpenBSD: if_trunk.c,v 1.30 2007/01/31 06:20:19 reyk Exp $      */
 
 /*
  * Copyright (c) 2005, 2006 Reyk Floeter <[e-mail]>
  * Copyright (c) 2007 Andrew Thompson <[e-mail]>
- * Copyright (c) 2014 Marcelo Araujo <[e-mail]>
+ * Copyright (c) 2014, 2015 Marcelo Araujo <[e-mail]>
  *
  * Permission to use, copy, modify, and distribute this software for any
  * purpose with or without fee is hereby granted, provided that the above
  * copyright notice and this permission notice appear in all copies.
  *
  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.pr_num = LAGG_PROTO_NONE
     },
     {
         .pr_num = LAGG_PROTO_ROUNDROBIN,
         .pr_attach = lagg_rr_attach,
         .pr_start = lagg_rr_start,
         .pr_input = lagg_rr_input,
     },
     {
         .pr_num = LAGG_PROTO_FAILOVER,
         .pr_start = lagg_fail_start,
         .pr_input = lagg_fail_input,
     },
     {
         .pr_num = LAGG_PROTO_LOADBALANCE,
         .pr_attach = lagg_lb_attach,
         .pr_detach = lagg_lb_detach,
         .pr_start = lagg_lb_start,
if (sc->sc_proto == LAGG_PROTO_LACP) {
                                 ro->ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
                         ro->ro_active = sc->sc_active;
                 } else {
                         ro->ro_active = 0;
                         SLIST_FOREACH(lp, &sc->sc_ports, lp_entries)
                                 ro->ro_active += LAGG_PORTACTIVE(lp);
                 }
+                ro->ro_bkt = sc->sc_bkt;
                 ro->ro_flapping = sc->sc_flapping;
                 ro->ro_flowid_shift = sc->flowid_shift;
                 break;
         case SIOCSLAGGOPTS:
+                if (sc->sc_proto == LAGG_PROTO_ROUNDROBIN) {
+                        if (ro->ro_bkt == 0)
+                                sc->sc_bkt = 1; // Minimum 1 packet per iface.
+                        else
+                                sc->sc_bkt = ro->ro_bkt;
+                }
                 error = priv_check(td, PRIV_NET_LAGG);
                 if (error)
                         break;
                 if (ro->ro_opts == 0)
                         break;
                 /*
                  * Set options.  LACP options are stored in sc->sc_psc,
                  * not in sc_opts.
case -LAGG_OPT_LACP_TIMEOUT:
                         valid = lacp = 1;
                         break;
                 default:
                         valid = lacp = 0;
                         break;
                 }
 
                 LAGG_WLOCK(sc);
+
                 if (valid == 0 ||
                     (lacp == 1 && sc->sc_proto != LAGG_PROTO_LACP)) {
                         /* Invalid combination of options specified. */
                         error = EINVAL;
                         LAGG_WUNLOCK(sc);
                         break;  /* Return from SIOCSLAGGOPTS. */ 
                 }
                 /*
lagg_transmit(struct ifnet *ifp, struct mbuf *m)
 
         if (error != 0)
                 if_inc_counter(ifp, IFCOUNTER_OERRORS, 1);
 
         return (error);
 }
 
 /*
  * The ifp->if_qflush entry point for lagg(4) is no-op.
  */
 static void
 lagg_qflush(struct ifnet *ifp __unused)
 {
 }
 
 static struct mbuf *
 lagg_input(struct ifnet *ifp, struct mbuf *m)
 {
         struct lagg_port *lp = ifp->if_lagg;
         struct lagg_softc *sc = lp->lp_softc;
         struct ifnet *scifp = sc->sc_ifp;
         struct rm_priotracker tracker;
 
         LAGG_RLOCK(sc, &tracker);
         if ((scifp->if_drv_flags & IFF_DRV_RUNNING) == 0 ||
             (lp->lp_flags & LAGG_PORT_DISABLED) ||
             sc->sc_proto == LAGG_PROTO_NONE) {

 /*
  * Simple round robin aggregation
  */
 static void
 lagg_rr_attach(struct lagg_softc *sc)
 {
         sc->sc_capabilities = IFCAP_LAGG_FULLDUPLEX;
         sc->sc_seq = 0;
+        sc->sc_bkt_count = sc->sc_bkt;
 }
 
 static int
 lagg_rr_start(struct lagg_softc *sc, struct mbuf *m)
 {
         struct lagg_port *lp;
         uint32_t p;
 
+        if (sc->sc_bkt_count == 0 && sc->sc_bkt > 0)
+                sc->sc_bkt_count = sc->sc_bkt;
+
+        if (sc->sc_bkt > 0) {
+                atomic_subtract_int(&sc->sc_bkt_count, 1);
+        if (atomic_cmpset_int(&sc->sc_bkt_count, 0, sc->sc_bkt))
                 p = atomic_fetchadd_32(&sc->sc_seq, 1);
+        else
+                p = sc->sc_seq; 
+        } else
+                p = atomic_fetchadd_32(&sc->sc_seq, 1);
+
         p %= sc->sc_count;
         lp = SLIST_FIRST(&sc->sc_ports);
+
         while (p--)
                 lp = SLIST_NEXT(lp, lp_entries);
 
         /*
          * Check the port's link state. This will return the next active
          * port if the link is down or the port is NULL.
          */
         if ((lp = lagg_link_active(sc, lp)) == NULL) {
SLIST_FOREACH(lp, &sc->sc_ports, lp_entries) {
                 if (!LAGG_PORTACTIVE(lp))
                         continue;
 
                 active_ports++;
 
                 if (last != NULL) {
                         m0 = m_copym(m, 0, M_COPYALL, M_NOWAIT);
                         if (m0 == NULL) {
                                 ret = ENOBUFS;
                                 errors++;
                                 break;
                         }
 
                         ret = lagg_enqueue(last->lp_ifp, m0);
                         if (ret != 0)
                                 errors++;
                 }

sys/kern/sys_process.c

 /*      $OpenBSD: if_trunk.c,v 1.30 2007/01/31 06:20:19 reyk Exp $      */
 
 /*
  * Copyright (c) 2005, 2006 Reyk Floeter <[e-mail]>
  * Copyright (c) 2007 Andrew Thompson <[e-mail]>
- * Copyright (c) 2014 Marcelo Araujo <[e-mail]>
+ * Copyright (c) 2014, 2015 Marcelo Araujo <[e-mail]>
  *
  * Permission to use, copy, modify, and distribute this software for any
  * purpose with or without fee is hereby granted, provided that the above
  * copyright notice and this permission notice appear in all copies.
  *
  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.pr_num = LAGG_PROTO_NONE
     },
     {
         .pr_num = LAGG_PROTO_ROUNDROBIN,
         .pr_attach = lagg_rr_attach,
         .pr_start = lagg_rr_start,
         .pr_input = lagg_rr_input,
     },
     {
         .pr_num = LAGG_PROTO_FAILOVER,
         .pr_start = lagg_fail_start,
         .pr_input = lagg_fail_input,
     },
     {
         .pr_num = LAGG_PROTO_LOADBALANCE,
         .pr_attach = lagg_lb_attach,
         .pr_detach = lagg_lb_detach,
         .pr_start = lagg_lb_start,
if (sc->sc_proto == LAGG_PROTO_LACP) {
                                 ro->ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
                         ro->ro_active = sc->sc_active;
                 } else {
                         ro->ro_active = 0;
                         SLIST_FOREACH(lp, &sc->sc_ports, lp_entries)
                                 ro->ro_active += LAGG_PORTACTIVE(lp);
                 }
+                ro->ro_bkt = sc->sc_bkt;
                 ro->ro_flapping = sc->sc_flapping;
                 ro->ro_flowid_shift = sc->flowid_shift;
                 break;
         case SIOCSLAGGOPTS:
+                if (sc->sc_proto == LAGG_PROTO_ROUNDROBIN) {
+                        if (ro->ro_bkt == 0)
+                                sc->sc_bkt = 1; // Minimum 1 packet per iface.
+                        else
+                                sc->sc_bkt = ro->ro_bkt;
+                }
                 error = priv_check(td, PRIV_NET_LAGG);
                 if (error)
                         break;
                 if (ro->ro_opts == 0)
                         break;
                 /*
                  * Set options.  LACP options are stored in sc->sc_psc,
                  * not in sc_opts.
case -LAGG_OPT_LACP_TIMEOUT:
                         valid = lacp = 1;
                         break;
                 default:
                         valid = lacp = 0;
                         break;
                 }
 
                 LAGG_WLOCK(sc);
+
                 if (valid == 0 ||
                     (lacp == 1 && sc->sc_proto != LAGG_PROTO_LACP)) {
                         /* Invalid combination of options specified. */
                         error = EINVAL;
                         LAGG_WUNLOCK(sc);
                         break;  /* Return from SIOCSLAGGOPTS. */ 
                 }
                 /*
lagg_transmit(struct ifnet *ifp, struct mbuf *m)
 
         if (error != 0)
                 if_inc_counter(ifp, IFCOUNTER_OERRORS, 1);
 
         return (error);
 }
 
 /*
  * The ifp->if_qflush entry point for lagg(4) is no-op.
  */
 static void
 lagg_qflush(struct ifnet *ifp __unused)
 {
 }
 
 static struct mbuf *
 lagg_input(struct ifnet *ifp, struct mbuf *m)
 {
         struct lagg_port *lp = ifp->if_lagg;
         struct lagg_softc *sc = lp->lp_softc;
         struct ifnet *scifp = sc->sc_ifp;
         struct rm_priotracker tracker;
 
         LAGG_RLOCK(sc, &tracker);
         if ((scifp->if_drv_flags & IFF_DRV_RUNNING) == 0 ||
             (lp->lp_flags & LAGG_PORT_DISABLED) ||
             sc->sc_proto == LAGG_PROTO_NONE) {

 /*
  * Simple round robin aggregation
  */
 static void
 lagg_rr_attach(struct lagg_softc *sc)
 {
         sc->sc_capabilities = IFCAP_LAGG_FULLDUPLEX;
         sc->sc_seq = 0;
+        sc->sc_bkt_count = sc->sc_bkt;
 }
 
 static int
 lagg_rr_start(struct lagg_softc *sc, struct mbuf *m)
 {
         struct lagg_port *lp;
         uint32_t p;
 
+        if (sc->sc_bkt_count == 0 && sc->sc_bkt > 0)
+                sc->sc_bkt_count = sc->sc_bkt;
+
+        if (sc->sc_bkt > 0) {
+                atomic_subtract_int(&sc->sc_bkt_count, 1);
+        if (atomic_cmpset_int(&sc->sc_bkt_count, 0, sc->sc_bkt))
                 p = atomic_fetchadd_32(&sc->sc_seq, 1);
+        else
+                p = sc->sc_seq; 
+        } else
+                p = atomic_fetchadd_32(&sc->sc_seq, 1);
+
         p %= sc->sc_count;
         lp = SLIST_FIRST(&sc->sc_ports);
+
         while (p--)
                 lp = SLIST_NEXT(lp, lp_entries);
 
         /*
          * Check the port's link state. This will return the next active
          * port if the link is down or the port is NULL.
          */
         if ((lp = lagg_link_active(sc, lp)) == NULL) {
SLIST_FOREACH(lp, &sc->sc_ports, lp_entries) {
                 if (!LAGG_PORTACTIVE(lp))
                         continue;
 
                 active_ports++;
 
                 if (last != NULL) {
                         m0 = m_copym(m, 0, M_COPYALL, M_NOWAIT);
                         if (m0 == NULL) {
                                 ret = ENOBUFS;
                                 errors++;
                                 break;
                         }
 
                         ret = lagg_enqueue(last->lp_ifp, m0);
                         if (ret != 0)
                                 errors++;
                 }

sys/sys/proc.h

 /*      $OpenBSD: if_trunk.c,v 1.30 2007/01/31 06:20:19 reyk Exp $      */
 
 /*
  * Copyright (c) 2005, 2006 Reyk Floeter <[e-mail]>
  * Copyright (c) 2007 Andrew Thompson <[e-mail]>
- * Copyright (c) 2014 Marcelo Araujo <[e-mail]>
+ * Copyright (c) 2014, 2015 Marcelo Araujo <[e-mail]>
  *
  * Permission to use, copy, modify, and distribute this software for any
  * purpose with or without fee is hereby granted, provided that the above
  * copyright notice and this permission notice appear in all copies.
  *
  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.pr_num = LAGG_PROTO_NONE
     },
     {
         .pr_num = LAGG_PROTO_ROUNDROBIN,
         .pr_attach = lagg_rr_attach,
         .pr_start = lagg_rr_start,
         .pr_input = lagg_rr_input,
     },
     {
         .pr_num = LAGG_PROTO_FAILOVER,
         .pr_start = lagg_fail_start,
         .pr_input = lagg_fail_input,
     },
     {
         .pr_num = LAGG_PROTO_LOADBALANCE,
         .pr_attach = lagg_lb_attach,
         .pr_detach = lagg_lb_detach,
         .pr_start = lagg_lb_start,
if (sc->sc_proto == LAGG_PROTO_LACP) {
                                 ro->ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
                         ro->ro_active = sc->sc_active;
                 } else {
                         ro->ro_active = 0;
                         SLIST_FOREACH(lp, &sc->sc_ports, lp_entries)
                                 ro->ro_active += LAGG_PORTACTIVE(lp);
                 }
+                ro->ro_bkt = sc->sc_bkt;
                 ro->ro_flapping = sc->sc_flapping;
                 ro->ro_flowid_shift = sc->flowid_shift;
                 break;
         case SIOCSLAGGOPTS:
+                if (sc->sc_proto == LAGG_PROTO_ROUNDROBIN) {
+                        if (ro->ro_bkt == 0)
+                                sc->sc_bkt = 1; // Minimum 1 packet per iface.
+                        else
+                                sc->sc_bkt = ro->ro_bkt;
+                }
                 error = priv_check(td, PRIV_NET_LAGG);
                 if (error)
                         break;
                 if (ro->ro_opts == 0)
                         break;
                 /*
                  * Set options.  LACP options are stored in sc->sc_psc,
                  * not in sc_opts.
case -LAGG_OPT_LACP_TIMEOUT:
                         valid = lacp = 1;
                         break;
                 default:
                         valid = lacp = 0;
                         break;
                 }
 
                 LAGG_WLOCK(sc);
+
                 if (valid == 0 ||
                     (lacp == 1 && sc->sc_proto != LAGG_PROTO_LACP)) {
                         /* Invalid combination of options specified. */
                         error = EINVAL;
                         LAGG_WUNLOCK(sc);
                         break;  /* Return from SIOCSLAGGOPTS. */ 
                 }
                 /*
lagg_transmit(struct ifnet *ifp, struct mbuf *m)
 
         if (error != 0)
                 if_inc_counter(ifp, IFCOUNTER_OERRORS, 1);
 
         return (error);
 }
 
 /*
  * The ifp->if_qflush entry point for lagg(4) is no-op.
  */
 static void
 lagg_qflush(struct ifnet *ifp __unused)
 {
 }
 
 static struct mbuf *
 lagg_input(struct ifnet *ifp, struct mbuf *m)
 {
         struct lagg_port *lp = ifp->if_lagg;
         struct lagg_softc *sc = lp->lp_softc;
         struct ifnet *scifp = sc->sc_ifp;
         struct rm_priotracker tracker;
 
         LAGG_RLOCK(sc, &tracker);
         if ((scifp->if_drv_flags & IFF_DRV_RUNNING) == 0 ||
             (lp->lp_flags & LAGG_PORT_DISABLED) ||
             sc->sc_proto == LAGG_PROTO_NONE) {

 /*
  * Simple round robin aggregation
  */
 static void
 lagg_rr_attach(struct lagg_softc *sc)
 {
         sc->sc_capabilities = IFCAP_LAGG_FULLDUPLEX;
         sc->sc_seq = 0;
+        sc->sc_bkt_count = sc->sc_bkt;
 }
 
 static int
 lagg_rr_start(struct lagg_softc *sc, struct mbuf *m)
 {
         struct lagg_port *lp;
         uint32_t p;
 
+        if (sc->sc_bkt_count == 0 && sc->sc_bkt > 0)
+                sc->sc_bkt_count = sc->sc_bkt;
+
+        if (sc->sc_bkt > 0) {
+                atomic_subtract_int(&sc->sc_bkt_count, 1);
+        if (atomic_cmpset_int(&sc->sc_bkt_count, 0, sc->sc_bkt))
                 p = atomic_fetchadd_32(&sc->sc_seq, 1);
+        else
+                p = sc->sc_seq; 
+        } else
+                p = atomic_fetchadd_32(&sc->sc_seq, 1);
+
         p %= sc->sc_count;
         lp = SLIST_FIRST(&sc->sc_ports);
+
         while (p--)
                 lp = SLIST_NEXT(lp, lp_entries);
 
         /*
          * Check the port's link state. This will return the next active
          * port if the link is down or the port is NULL.
          */
         if ((lp = lagg_link_active(sc, lp)) == NULL) {
SLIST_FOREACH(lp, &sc->sc_ports, lp_entries) {
                 if (!LAGG_PORTACTIVE(lp))
                         continue;
 
                 active_ports++;
 
                 if (last != NULL) {
                         m0 = m_copym(m, 0, M_COPYALL, M_NOWAIT);
                         if (m0 == NULL) {
                                 ret = ENOBUFS;
                                 errors++;
                                 break;
                         }
 
                         ret = lagg_enqueue(last->lp_ifp, m0);
                         if (ret != 0)
                                 errors++;
                 }

sys/vm/vm_swapout.c

 /*      $OpenBSD: if_trunk.c,v 1.30 2007/01/31 06:20:19 reyk Exp $      */
 
 /*
  * Copyright (c) 2005, 2006 Reyk Floeter <[e-mail]>
  * Copyright (c) 2007 Andrew Thompson <[e-mail]>
- * Copyright (c) 2014 Marcelo Araujo <[e-mail]>
+ * Copyright (c) 2014, 2015 Marcelo Araujo <[e-mail]>
  *
  * Permission to use, copy, modify, and distribute this software for any
  * purpose with or without fee is hereby granted, provided that the above
  * copyright notice and this permission notice appear in all copies.
  *
  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.pr_num = LAGG_PROTO_NONE
     },
     {
         .pr_num = LAGG_PROTO_ROUNDROBIN,
         .pr_attach = lagg_rr_attach,
         .pr_start = lagg_rr_start,
         .pr_input = lagg_rr_input,
     },
     {
         .pr_num = LAGG_PROTO_FAILOVER,
         .pr_start = lagg_fail_start,
         .pr_input = lagg_fail_input,
     },
     {
         .pr_num = LAGG_PROTO_LOADBALANCE,
         .pr_attach = lagg_lb_attach,
         .pr_detach = lagg_lb_detach,
         .pr_start = lagg_lb_start,
if (sc->sc_proto == LAGG_PROTO_LACP) {
                                 ro->ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
                         ro->ro_active = sc->sc_active;
                 } else {
                         ro->ro_active = 0;
                         SLIST_FOREACH(lp, &sc->sc_ports, lp_entries)
                                 ro->ro_active += LAGG_PORTACTIVE(lp);
                 }
+                ro->ro_bkt = sc->sc_bkt;
                 ro->ro_flapping = sc->sc_flapping;
                 ro->ro_flowid_shift = sc->flowid_shift;
                 break;
         case SIOCSLAGGOPTS:
+                if (sc->sc_proto == LAGG_PROTO_ROUNDROBIN) {
+                        if (ro->ro_bkt == 0)
+                                sc->sc_bkt = 1; // Minimum 1 packet per iface.
+                        else
+                                sc->sc_bkt = ro->ro_bkt;
+                }
                 error = priv_check(td, PRIV_NET_LAGG);
                 if (error)
                         break;
                 if (ro->ro_opts == 0)
                         break;
                 /*
                  * Set options.  LACP options are stored in sc->sc_psc,
                  * not in sc_opts.
case -LAGG_OPT_LACP_TIMEOUT:
                         valid = lacp = 1;
                         break;
                 default:
                         valid = lacp = 0;
                         break;
                 }
 
                 LAGG_WLOCK(sc);
+
                 if (valid == 0 ||
                     (lacp == 1 && sc->sc_proto != LAGG_PROTO_LACP)) {
                         /* Invalid combination of options specified. */
                         error = EINVAL;
                         LAGG_WUNLOCK(sc);
                         break;  /* Return from SIOCSLAGGOPTS. */ 
                 }
                 /*
lagg_transmit(struct ifnet *ifp, struct mbuf *m)
 
         if (error != 0)
                 if_inc_counter(ifp, IFCOUNTER_OERRORS, 1);
 
         return (error);
 }
 
 /*
  * The ifp->if_qflush entry point for lagg(4) is no-op.
  */
 static void
 lagg_qflush(struct ifnet *ifp __unused)
 {
 }
 
 static struct mbuf *
 lagg_input(struct ifnet *ifp, struct mbuf *m)
 {
         struct lagg_port *lp = ifp->if_lagg;
         struct lagg_softc *sc = lp->lp_softc;
         struct ifnet *scifp = sc->sc_ifp;
         struct rm_priotracker tracker;
 
         LAGG_RLOCK(sc, &tracker);
         if ((scifp->if_drv_flags & IFF_DRV_RUNNING) == 0 ||
             (lp->lp_flags & LAGG_PORT_DISABLED) ||
             sc->sc_proto == LAGG_PROTO_NONE) {

 /*
  * Simple round robin aggregation
  */
 static void
 lagg_rr_attach(struct lagg_softc *sc)
 {
         sc->sc_capabilities = IFCAP_LAGG_FULLDUPLEX;
         sc->sc_seq = 0;
+        sc->sc_bkt_count = sc->sc_bkt;
 }
 
 static int
 lagg_rr_start(struct lagg_softc *sc, struct mbuf *m)
 {
         struct lagg_port *lp;
         uint32_t p;
 
+        if (sc->sc_bkt_count == 0 && sc->sc_bkt > 0)
+                sc->sc_bkt_count = sc->sc_bkt;
+
+        if (sc->sc_bkt > 0) {
+                atomic_subtract_int(&sc->sc_bkt_count, 1);
+        if (atomic_cmpset_int(&sc->sc_bkt_count, 0, sc->sc_bkt))
                 p = atomic_fetchadd_32(&sc->sc_seq, 1);
+        else
+                p = sc->sc_seq; 
+        } else
+                p = atomic_fetchadd_32(&sc->sc_seq, 1);
+
         p %= sc->sc_count;
         lp = SLIST_FIRST(&sc->sc_ports);
+
         while (p--)
                 lp = SLIST_NEXT(lp, lp_entries);
 
         /*
          * Check the port's link state. This will return the next active
          * port if the link is down or the port is NULL.
          */
         if ((lp = lagg_link_active(sc, lp)) == NULL) {
SLIST_FOREACH(lp, &sc->sc_ports, lp_entries) {
                 if (!LAGG_PORTACTIVE(lp))
                         continue;
 
                 active_ports++;
 
                 if (last != NULL) {
                         m0 = m_copym(m, 0, M_COPYALL, M_NOWAIT);
                         if (m0 == NULL) {
                                 ret = ENOBUFS;
                                 errors++;
                                 break;
                         }
 
                         ret = lagg_enqueue(last->lp_ifp, m0);
                         if (ret != 0)
                                 errors++;
                 }