Summary

When one creates interfaces in VPP that carry ethernet frames, default values for their creation result in an interface to which can both be plugged into a pure L2 context (like a bridge domain or l2xc) or have an IP address assigned and routed to from a FIB. This is true for af-packet, tap, vhost, etc. This had been true previously for VXLAN.

After https://gerrit.fd.io/r/c/vpp/+/34721 this is no longer true for VXLAN.

Adding an IP address to a VXLAN interface (as one would to any other ethernet interface) results in the interface operating in 'L3' mode in VPP. When a packet is routed out that VXLAN interface, VPP to crash due to arp failing because the VXLAN interface lacks a mac address because it is no longer by default registered with ethernet_register_interface.

This crash is very confusing to most API consumers, who are not equipped to debug why VPP crashed, they just know their previously working code that was using the VPP API suddenly results in VPP crashing when they upgrade VPP.

History

VXLAN is and always has been an ethernet interface. RFC 7348 defines VXLAN as carrying a single type of payload: an ethernet frame.

VXLAN had mistakenly been using vnet_register_interface instead of ethernet_register_interface leading to VPP-1962 which was fixed by https://gerrit.fd.io/r/c/vpp/+/30806

Subsequently, it was discovered that a novel use of VXLAN by Calico-VPP that ignored the ethernet headers required non-standard treatment in VPP (the use of vnet_register_interface). sknat submitted https://gerrit.fd.io/r/c/vpp/+/32085 added 'is_l3' in order to provide a flag for use to trigger this non-standard treatment. 'is_l3' was always a flag for the Calico-VPP ignore-the-ethernet-headers case.

The Geneve plugin had been largely copied from the VXLAN plugin. The original (and current) implementation of Geneve only supported transporting ethernet frames. Unlike VXLAN however, Geneve can in principle be used to transport other non-ethernet payloads, like IP. See RFC8926 'Protocol Type'. Geneve copied the mistaken use of vnet_register_interface for an ethernet interface at it's creation.

Subsequently, otroan https://gerrit.fd.io/r/c/vpp/+/27586 made the decision to use a flag 'l3_mode' to establish a case for registering using ethernet_register_interface for Geneve. Currently, Geneve 'l3_mode' is still using ethernet payloads. Downstream users have become accustomed in Geneve to using 'l3_mode' if they do not want Geneve to crash when assigning an IP to a Geneve interface, such as in the BVI case.

Downstream users who had become accustomed to using 'l3_mode' in order to be able to assign an IP address to a Geneve interface without crashing, attempted to use 'is_l3' with VXLAN to assign an IP address. This inadvertent configuring of the Calico-VPP special ignore-the-ethernet-headers case involved the use of vnet_register_interface, which resulted in VPP crashing when 'is_l3' was used for VXLAN and an IP address is assigned. They quite naturally reported this as a bug.

sluong, nverted the meaning of 'is_l3' in VXLAN in https://gerrit.fd.io/r/c/vpp/+/34721 in an attempt to fix the reported bug.

This broke downstream consumers who had been relying on the standard VXLAN behavior for the default (being an ethernet interface). That breakage results in a difficult to debug VPP crash. API users who are not VPP coders tend to find such crashes utterly mystifying.

This also resulted in issues for the Calico-VPP case, which is the origin of the 'is_l3' flag in VPP. It is unclear if there are any other users relying on the 'is_l3' flag.

VPP and 'L3' vs 'L2' mode

VPP does have a notion of interfaces being in L2 or L3 mode. VPP's 'L3' mode has nothing to do with the existing 'is_l3' in VXLAN. An interface that carries ethernet is set into 'L3' mode when an IP address is assigned to it, otherwise it defaults 'L2' mode.

For existing interface types that carry ethernet other than VXLAN and Geneve (for example af-packet, tap, vhost, etc) there is no need to configure the 'interface' at creation time with an flag indicating 'l3-ness'. Simply configuring an IP address does so.

Prior to https://gerrit.fd.io/r/c/vpp/+/34721 VXLAN followed this standard behavior for ethernet interfaces in VPP: by default, if you created one, you could assign an IP address to it and it would function correctly as an ethernet interface in 'L3' mode. Now VPP crashes if a user follows the same pattern that works for all other ethernet interfaces in VPP.

Recommended VXLAN resolution

1. Revert https://gerrit.fd.io/r/c/vpp/+/34721 to restore the behavior in previous releases and re-align with how most other interfaces (other than Geneve) work in VPP.
2. The Calico-VPP folks provide a fix so that 'is_l3' does not crash if an IP address is assigned to the interface and a user attempts to forward traffic over it.

The revert should happen immediately for integration into a dot release of VPP so as to restore previous behavior and avoid VPP crashes by existing API usage. The fix so 'is_l3' is not dangerous may optionally come later.

Note on Geneve

Geneve is a somewhat different case in that it could be used to carry non-ethernet payloads even though currently VPP does not support that. One could argue that the current Geneve implementation should also behave as a proper ethernet interface in VPP in its default configuration. However, as there is ongoing work exploring supporting other payload protocols like IP in Geneve, it is probably best to defer a decision on such questions until the shape of that work has become clearer.