I'm a little torn on how to handle the extension in the interface to userspace. I've added a new ioctl for it, but we could also extend the struct (and then teach the existing ioctl to cope with two sizes of structure), or we could convert the whole thing to using netlink, to make future extensions easier.

Thank you for working on this!
The unicast option is a good addition indeed.
My only concern is that we're adding global carp-specific ioctls that doesn't look easy to extend afterwards. I'd prefer to create an easily-extenable API, such as netlink.
I'm happy to work with you (or implement a variation of this patch myself) that uses netlink as the control mechanism.

In D38940#886682, @melifaro wrote:

I'm happy to work with you (or implement a variation of this patch myself) that uses netlink as the control mechanism.

I'm going to use this as an opportunity to get to know netlink a bit more, so I'll take a stab at a first version myself, and then we'll go from there.
It probably makes sense to extend the existing code to use netlink and then rebase these patches (now using netlink) on top.

In D38940#886690, @kp wrote:

I'm going to use this as an opportunity to get to know netlink a bit more, so I'll take a stab at a first version myself, and then we'll go from there.

The first thing I'm running into is "The library does not currently offer any wrappers for writing netlink messages.". I can compose it directly, but if I understand things correctly that means we're not doing the TLV thing, and we're back to painting ourselves into a corner w.r.t. extensibility.
I can push onwards for now, but I think that should be addressed before we commit to a conversion to netlink.

I'm also not sure I understand the distinction between fields and attributes. Oh. Wait, are fields always there (i.e. the things we can easily compose directly with a struct before the snl_send() call), and attributes are optional? So for carp it'd probably make sense to have at least 'ifname' be a field, and things like vhid be attributes (so we can optionally filter on vhid)?

In D38940#886729, @kp wrote:

In D38940#886690, @kp wrote:

I'm going to use this as an opportunity to get to know netlink a bit more, so I'll take a stab at a first version myself, and then we'll go from there.

The first thing I'm running into is "The library does not currently offer any wrappers for writing netlink messages.". I can compose it directly, but if I understand things correctly that means we're not doing the TLV thing, and we're back to painting ourselves into a corner w.r.t. extensibility.

I'll have it working for route(8) conversion - will publish writers diff either later today or tomorrow.

I can push onwards for now, but I think that should be addressed before we commit to a conversion to netlink.

Sure, I don't want anyone to write raw message composing code.

I'm also not sure I understand the distinction between fields and attributes. Oh. Wait, are fields always there (i.e. the things we can easily compose directly with a struct before the snl_send() call), and attributes are optional? So for carp it'd probably make sense to have at least 'ifname' be a field, and things like vhid be attributes (so we can optionally filter on vhid)?

Yes, fields are mandatory (in a sense that they need to be present in the wire), while attributes are not _mandatory_ per protocol, but the handler can of course require certain number of them to be present.

For the network interfaces API, netlink has a concept of sharing custom per-interface data in the RTM_NEWLINK messages. Nested attributes IFLA_LINKINFO / IFLA_INFO_DATA are used for that.
I was thinking of using cloners variation to provide this additional data: https://cgit.freebsd.org/src/tree/sys/netlink/route/iface_drivers.c#n207 . This is a creation example for vlan, along with the modification / dump callbacks.

Published the writer code in D38947 (will update with the documentation tomorrow).

This allows carp to be used in unicast mode, which is useful in certain virtual configurations (e.g. AWS, VMWare ESXi, ...)

I use carp in my production environment (ESXi), and I have to enable promisc mode for virtual ports so that multicast frames can forward between VMs.

I do not familiar with AWS, so suppose AWS does not provide mechanism like ESXi. If AWS is willing to provide ( configuration of promisc mode ) then this feature is not that useful.

After dig into D38941 I'm even get confused about the design of unicast mode.
Is it two boxes share same carp group but exchange CARP messages across routers ? That seems to be an anycast setup. For anycast we have BGP so it looks an overkill for CARP protocol.

And how about on carp group with three or more boxes ? So we need peers instead of peer ?

In D38940#887150, @zlei wrote:

This allows carp to be used in unicast mode, which is useful in certain virtual configurations (e.g. AWS, VMWare ESXi, ...)

I use carp in my production environment (ESXi), and I have to enable promisc mode for virtual ports so that multicast frames can forward between VMs.

I do not familiar with AWS, so suppose AWS does not provide mechanism like ESXi. If AWS is willing to provide ( configuration of promisc mode ) then this feature is not that useful.

It doesn't. Yesterday I discovered that the current version of the patch doesn't work in AWS, because it changes the source MAC address to a multicast address. I've got a fix for that (i.e. don't change the MAC address in unicast mode) that allows it to work in AWS. Even in promiscuous mode the multicast traffic never arrives, so we do need this at least for AWS, and it would be useful for ESXi to avoid needing to run interfaces in promiscuous mode.

After dig into D38941 I'm even get confused about the design of unicast mode.
Is it two boxes share same carp group but exchange CARP messages across routers ? That seems to be an anycast setup. For anycast we have BGP so it looks an overkill for CARP protocol.

That's the intention, yes. The idea is to allow an elastic IP (in AWS parlance) to be a failed over between different instances, without making assumptions about the virtual network layout.

And how about on carp group with three or more boxes ? So we need peers instead of peer ?

Interesting point. I hadn't considered that, mostly because this mirrors the functionality in pfsync, where unicast mode is also limited to two peers. My inclination is to start off with the smaller change (i.e. supporting two boxes in the carp group) before we worry about other scenarios.

In D38940#887195, @kp wrote:

In D38940#887150, @zlei wrote:

This allows carp to be used in unicast mode, which is useful in certain virtual configurations (e.g. AWS, VMWare ESXi, ...)

I use carp in my production environment (ESXi), and I have to enable promisc mode for virtual ports so that multicast frames can forward between VMs.

I do not familiar with AWS, so suppose AWS does not provide mechanism like ESXi. If AWS is willing to provide ( configuration of promisc mode ) then this feature is not that useful.

It doesn't. Yesterday I discovered that the current version of the patch doesn't work in AWS, because it changes the source MAC address to a multicast address. I've got a fix for that (i.e. don't change the MAC address in unicast mode) that allows it to work in AWS.

I think AWS and ESXi by default block those forge packets with CARP(VRRP) multicast source MAC address.

Even in promiscuous mode the multicast traffic never arrives, so we do need this at least for AWS, and it would be useful for ESXi to avoid needing to run interfaces in promiscuous mode.

That is a little tricky. I see carp_multicast_setup() calls in_joingroup() / in6_joingroup() , then the interface will join CARP multicast group, so promiscuous mode is not needed. Am I missing something?

After dig into D38941 I'm even get confused about the design of unicast mode.
Is it two boxes share same carp group but exchange CARP messages across routers ? That seems to be an anycast setup. For anycast we have BGP so it looks an overkill for CARP protocol.

That's the intention, yes. The idea is to allow an elastic IP (in AWS parlance) to be a failed over between different instances, without making assumptions about the virtual network layout.

That may be workaround by CARP over VXLAN and devd, or net/ucarp .

And how about on carp group with three or more boxes ? So we need peers instead of peer ?

Interesting point. I hadn't considered that, mostly because this mirrors the functionality in pfsync, where unicast mode is also limited to two peers. My inclination is to start off with the smaller change (i.e. supporting two boxes in the carp group) before we worry about other scenarios.

In D38940#887365, @zlei wrote:

In D38940#887195, @kp wrote:

In D38940#887150, @zlei wrote:

This allows carp to be used in unicast mode, which is useful in certain virtual configurations (e.g. AWS, VMWare ESXi, ...)

I use carp in my production environment (ESXi), and I have to enable promisc mode for virtual ports so that multicast frames can forward between VMs.

I do not familiar with AWS, so suppose AWS does not provide mechanism like ESXi. If AWS is willing to provide ( configuration of promisc mode ) then this feature is not that useful.

It doesn't. Yesterday I discovered that the current version of the patch doesn't work in AWS, because it changes the source MAC address to a multicast address. I've got a fix for that (i.e. don't change the MAC address in unicast mode) that allows it to work in AWS.

I think AWS and ESXi by default block those forge packets with CARP(VRRP) multicast source MAC address.

I wouldn't describe the traffic as forged. (And I was wrong, it's not a multicast MAC address, but a "virtual router MAC address"). The VRRP standard at least explicitly requires that address to be used. CARP isn't formally standardised (at least that I could find), but it behaves very like VRRP.
In any event, we need to do things slightly differently to make them work in cloud setups.

Even in promiscuous mode the multicast traffic never arrives, so we do need this at least for AWS, and it would be useful for ESXi to avoid needing to run interfaces in promiscuous mode.

That is a little tricky. I see carp_multicast_setup() calls in_joingroup() / in6_joingroup() , then the interface will join CARP multicast group, so promiscuous mode is not needed. Am I missing something?

Cloud networks tend to be a bit special. At least in the first tests the multicast traffic simply didn't make it between VMs (in AWS). I don't currently have access to an ESXi instance, but I understood from your comments that it needs workarounds for multicast. Again something we'd like to avoid having to deal with.

After dig into D38941 I'm even get confused about the design of unicast mode.
Is it two boxes share same carp group but exchange CARP messages across routers ? That seems to be an anycast setup. For anycast we have BGP so it looks an overkill for CARP protocol.

That's the intention, yes. The idea is to allow an elastic IP (in AWS parlance) to be a failed over between different instances, without making assumptions about the virtual network layout.

That may be workaround by CARP over VXLAN and devd, or net/ucarp .

I'm insufficiently familiar with VXLAN to say if it'll accept multicast traffic or not, but in any event, we'd also like to keep the carp code as similar as possible between the physical hardware and cloud instance case.

In D38940#887375, @kp wrote:

In D38940#887365, @zlei wrote:

In D38940#887195, @kp wrote:

In D38940#887150, @zlei wrote:

This allows carp to be used in unicast mode, which is useful in certain virtual configurations (e.g. AWS, VMWare ESXi, ...)

I use carp in my production environment (ESXi), and I have to enable promisc mode for virtual ports so that multicast frames can forward between VMs.

I do not familiar with AWS, so suppose AWS does not provide mechanism like ESXi. If AWS is willing to provide ( configuration of promisc mode ) then this feature is not that useful.

It doesn't. Yesterday I discovered that the current version of the patch doesn't work in AWS, because it changes the source MAC address to a multicast address. I've got a fix for that (i.e. don't change the MAC address in unicast mode) that allows it to work in AWS.

I think AWS and ESXi by default block those forge packets with CARP(VRRP) multicast source MAC address.

I wouldn't describe the traffic as forged. (And I was wrong, it's not a multicast MAC address, but a "virtual router MAC address"). The VRRP standard at least explicitly requires that address to be used. CARP isn't formally standardised (at least that I could find), but it behaves very like VRRP.

Sorry I did not mean they (packets with CARP(VRRP) multicast source MAC address) are forged, but AWS / ESXi might treat them as forged [1].

In any event, we need to do things slightly differently to make them work in cloud setups.

Even in promiscuous mode the multicast traffic never arrives, so we do need this at least for AWS, and it would be useful for ESXi to avoid needing to run interfaces in promiscuous mode.

That is a little tricky. I see carp_multicast_setup() calls in_joingroup() / in6_joingroup() , then the interface will join CARP multicast group, so promiscuous mode is not needed. Am I missing something?

Cloud networks tend to be a bit special. At least in the first tests the multicast traffic simply didn't make it between VMs (in AWS). I don't currently have access to an ESXi instance, but I understood from your comments that it needs workarounds for multicast. Again something we'd like to avoid having to deal with.

After dig into D38941 I'm even get confused about the design of unicast mode.
Is it two boxes share same carp group but exchange CARP messages across routers ? That seems to be an anycast setup. For anycast we have BGP so it looks an overkill for CARP protocol.

That's the intention, yes. The idea is to allow an elastic IP (in AWS parlance) to be a failed over between different instances, without making assumptions about the virtual network layout.

That may be workaround by CARP over VXLAN and devd, or net/ucarp .

I'm insufficiently familiar with VXLAN to say if it'll accept multicast traffic or not,

I can confirm it will.

but in any event, we'd also like to keep the carp code as similar as possible between the physical hardware and cloud instance case.

I believe that's a good approach, though I can not tell which one is better, a workaround or more features to support non-standard-compliant devices.

[1] https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-7DC6486F-5400-44DF-8A62-6273798A2F80.html

In D38940#886682, @melifaro wrote:

I'm happy to work with you (or implement a variation of this patch myself) that uses netlink as the control mechanism.

I've got a first draft of netlink for carp here: D39048

Generally, LGTM, please see some comments (mostly related to the IP address handling) inline.