epair: add VLAN_HWTAGGING

[timo.voelker_fh-muenster.de]

Summary

Add capability VLAN_HWTAGGING to the epair interface and enable it by default.

When sending a packet over a VLAN interface that uses an epair interface, the flag M_VLANTAG and the ether_vtag (which contains the VLAN ID and/or PCP) are set in the mbuf to inform the hardware that the VLAN header has to be added. The sending epair end does not need to actually add a VLAN header. It can just pass the mbuf with this setting to the other epair end, which receives the packet. The receiving epair end can just pass the mbuf with this setting to the upper layer. Due to this setting, the upper layer believes that there was a VLAN header that has been removed by the interface.

If the packet later leaves the host, the outgoing physical interface can add the VLAN header in hardware if it supports VLAN_HWTAGGING. If not, the implementation of Ethernet or bridge adds the VLAN header in software.

Test Plan

Create epair interface, and VLAN interface with IP address.

ifconfig epair0 create
ifconfig epair0a.3 create vlan 3 vlandev epair0a
ifconfig epair0a.3 inet 192.168.3.1/24
ifconfig epair0a up

Create jail and add the epair b-side to it.

jail -c name=jvnet host.hostname=jvnet persist vnet vnet.interface=epair0b

Add VLAN interface with IP address in the jail

jexec jvnet ifconfig epair0b.3 create vlan 3 vlandev epair0b
jexec jvnet ifconfig epair0b.3 inet 192.168.3.2/24
jexec jvnet ifconfig epair0b up

The epair end in the jail is reachable even with VLAN_HWTAGGING enabled on epair0a.

ping 192.168.3.2

[timo.voelker_fh-muenster.de]

sys/net/if_epair.c

155

This code was added to fix a bug D39482. Later, code was added in if_ethersubr.c to clear the M_VLANTAG flag D39499, making this code no longer necessary.

For VLAN_HWTAGGING the VLAN information must be preserved.

[zlei]

sys/net/if_epair.c

503

the other end has to have it enabled too

I'm a little confused by this statement. Does it hint that a combination of IFCAP_VLAN_HWTAGGING on epair0a and no IFCAP_VLAN_HWTAGGING on epair0b is not supposed to work ?

[timo.voelker_fh-muenster.de]

sys/net/if_epair.c

503

the other end has to have it enabled too

I'm a little confused by this statement. Does it hint that a combination of IFCAP_VLAN_HWTAGGING on epair0a and no IFCAP_VLAN_HWTAGGING on epair0b is not supposed to work ?

My understanding of IFCAP_VLAN_HWTAGGING is that it affects an interface in both directions.

• On output, the interface adds a VLAN header based ether_vtag if the M_VLANTAG flag is set.
• On input, the interface removes the VLAN header, copies the VLAN tag in ether_vtag and sets the M_VLANTAG flag.
  
  If that's correct, it might not work if the other end does not have IFCAP_VLAN_HWTAGGING enabled, since it is not supposed to pass a packet to the upper layer with the M_VLANTAG set.

[zlei]

sys/net/if_epair.c

503

the other end has to have it enabled too

I'm a little confused by this statement. Does it hint that a combination of IFCAP_VLAN_HWTAGGING on epair0a and no IFCAP_VLAN_HWTAGGING on epair0b is not supposed to work ?

My understanding of IFCAP_VLAN_HWTAGGING is that it affects an interface in both directions.

• On output, the interface adds a VLAN header based ether_vtag if the M_VLANTAG flag is set.
• On input, the interface removes the VLAN header, copies the VLAN tag in ether_vtag and sets the M_VLANTAG flag.

That is true for a real interface, aka the hardware, since then the packets are sent to wire they do not have M_VLANTAG flag. For inbound the hardware sees frames with vlan header but no M_VLANTAG ;)

If that's correct, it might not work if the other end does not have IFCAP_VLAN_HWTAGGING enabled, since it is not supposed to pass a packet to the upper layer with the M_VLANTAG set.

if_epair(4) is not a real interface, so no *HARD* wire between the two ends, then IFCAP_VLAN_HWTAGGING is more a hint to the net stack to tell it stop manipulating vlan header but instead the driver will do. Well, to be *efficient*, the driver does not actually insert any vlan header for outbound.

For the outbound, the behavior should be same with real interfaces. For inbound, it will be a little complicated. Let's do a matrix, you can read the traffic flow goes from left to right.

epair A net stack out	epair A driver out	epair B driver in	epair B net stack in
no HWTAGGING, net stack insert vlan header	keep vlan header as is	no HWTAGGING, keep vlan header as is	net stack extract vlan header and add M_VLANTAG to mbuf
no HWTAGGING, net stack insert vlan header	keep vlan header as is	HWTAGGING, should extrace vlan header and add M_VLANTAG to mbuf, but can be lazy let the the net stack do	receive M_VLANTAG as is, or extract vlan header if the driver is lazy and does not honors HWTAGGING
HWTAGGING let A do the "offload"	keep M_VLANTAG as is	no HWTAGGING, should receive as is, keep M_VLANTAG as is	is fooled, driver does not do HWTAGGING but pass in mbuf with M_VLANTAG, should accept M_VLANTAG as is
HWTAGGING let A do the "offload"	keep M_VLANTAG as is	HWTAGGING, receive as is, keep M_VLANTAG as is	receive M_VLANTAG as is

So the logic becomes clear. For if_epair(4), IFCAP_VLAN_HWTAGGING is most useful for the out direction but not for the in direction.

[tuexen]

sys/net/if_epair.c

503

Is there any benefit to not sync the IFCAP_VLAN_HWTAGGING? It makes the handling pretty straightforward and covers (in my view) all relevant use cases.

[zlei]

sys/net/if_epair.c

503

sync the IFCAP_VLAN_HWTAGGING between two ends should work, but it may expose more concurrency issue and it will also introduce some conflicts from the point of management.

Say in most case, the epair has one end in a vnet, and the other in another, process in different jail / vnet toggle the cap. That is rare but possible. If we can avoid that, I think we should.

[tuexen]

sys/net/if_epair.c

503

sync the IFCAP_VLAN_HWTAGGING between two ends should work, but it may expose more concurrency issue and it will also introduce some conflicts from the point of management.

I would be interested to learn what the concurrency issue and management conflicts are. This syncing is already done to transmit checksum offloading.

Say in most case, the epair has one end in a vnet, and the other in another, process in different jail / vnet toggle the cap. That is rare but possible. If we can avoid that, I think we should.

I think the stack would automatically adopt. Am I wrong?

[zlei]

sys/net/if_epair.c

503

sync the IFCAP_VLAN_HWTAGGING between two ends should work, but it may expose more concurrency issue and it will also introduce some conflicts from the point of management.

I would be interested to learn what the concurrency issue and management conflicts are. This syncing is already done to transmit checksum offloading.

For the concurrency issue , I think that is a long due. Lots of drivers lack proper lock sync in the ioctl() path. Toggling the if_flags is a typical one of such flaw. I figured out that while overhauling the lifecycle of ifnet. @glebius may have more insight in that.

The management conflicts is from the perspective of isolation of net stack, aka the VNET. Unless it is explained ( in the man page or somewhere ) with a good reason why the sync of capabilities is mandatory, maybe user will get confused, since that can not be easily concluded by user experience.

Say in most case, the epair has one end in a vnet, and the other in another, process in different jail / vnet toggle the cap. That is rare but possible. If we can avoid that, I think we should.

I think the stack would automatically adopt. Am I wrong?

Probably right now is not the right time to fix all the concurrency issue. I'm not strongly against this approach.

[zlei]

The code looks good to me.

Well I think the document should be revised a little. Indeed for inbound traffic, if the mbuf has flag M_VLANTAG, the epair(4) end should accept but not try to discard or trim the flag, since the input path of the driver does nothing.

Also the net stack would expect the driver to do vlan header extracting, but current implementation does *NOTHING* but just pass in the mbuf as is. So IFCAP_VLAN_HWTAGGING for the input patch is more false capability.

[timo.voelker_fh-muenster.de]

In D52465#1198897, @zlei wrote:

The code looks good to me.

Well I think the document should be revised a little. Indeed for inbound traffic, if the mbuf has flag M_VLANTAG, the epair(4) end should accept but not try to discard or trim the flag, since the input path of the driver does nothing.

Also the net stack would expect the driver to do vlan header extracting, but current implementation does *NOTHING* but just pass in the mbuf as is. So IFCAP_VLAN_HWTAGGING for the input patch is more false capability.

That's true. When sending a packet with a VLAN header over an epair, the receiving epair end won't extract the vlan header. This may happen (even with syncing IFCAP_VLAN_HWTAGGING between the epair ends) if the host receives a packet over an interface that does not extract the vlan header and forwards this packet (via a bridge) to an epair interface.

The question is, what does IFCAP_VLAN_HWTAGGING on an interface mean for the receiving side? If it is a guarantee that the interface will never pass a packet to the upper layer with a vlan header, then this patch should not go in. If it is instead an indication that it may extract the vlan header for the upper layer (as I understand it), then the patch is OK, I guess.

[tuexen]

In D52465#1199673, @timo.voelker_fh-muenster.de wrote:

In D52465#1198897, @zlei wrote:

The code looks good to me.

Well I think the document should be revised a little. Indeed for inbound traffic, if the mbuf has flag M_VLANTAG, the epair(4) end should accept but not try to discard or trim the flag, since the input path of the driver does nothing.

Also the net stack would expect the driver to do vlan header extracting, but current implementation does *NOTHING* but just pass in the mbuf as is. So IFCAP_VLAN_HWTAGGING for the input patch is more false capability.

That's true. When sending a packet with a VLAN header over an epair, the receiving epair end won't extract the vlan header. This may happen (even with syncing IFCAP_VLAN_HWTAGGING between the epair ends) if the host receives a packet over an interface that does not extract the vlan header and forwards this packet (via a bridge) to an epair interface.

The question is, what does IFCAP_VLAN_HWTAGGING on an interface mean for the receiving side? If it is a guarantee that the interface will never pass a packet to the upper layer with a vlan header, then this patch should not go in. If it is instead an indication that it may extract the vlan header for the upper layer (as I understand it), then the patch is OK, I guess.

I think answering such a question makes sense for physical interfaces. For logical interfaces, I think we should take into account two scenarios:

1. The other end is on a host. In this case computations which do not make sense might no be performed.
2. The other end is in a bridge and the packet goes out via a physical interface. It makes sense to be able to use hardware capabilities of a physical NIC, if possible.

I think this original intention of this patch is to allow checksum offloading in combination with vlan tagging. This requires support for hardware tagging.