UPDATE FROM Damjan:

VPP uses VLAN v2 offload APIs which the i40e driver doesn't contain. Utilizing a driver with the v2 API will resolve the issue, since it "allows us to know if stripping is supported or not on the specific interface". V1 version of the API returns an error and there's not much more we can do about it.

Damjan will not work on a workaround - we'll need a driver with the new v2 API.

Until we have that driver, the only course is to use an outdated driver (which doesn't procude the error in question).

----------------------------------

We're been seeing this since upgrading to 2.15.9 i40e driver (and 4.1.1 iavf, but I think this is just i40e related) using this VF configuration:

localadmin@s55-t13-sut1:~$ ip link show enp145s0f0
8: enp145s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:e6:a1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether ba:dc:0f:fe:02:00 brd ff:ff:ff:ff:ff:ff, vlan 1100, spoof checking off, link-state auto, trust on
vf 1 link/ether ba:dc:0f:fe:02:01 brd ff:ff:ff:ff:ff:ff, vlan 1101, spoof checking off, link-state auto, trust on
vf 2 link/ether ba:dc:0f:fe:02:02 brd ff:ff:ff:ff:ff:ff, vlan 1102, spoof checking off, link-state auto, trust on

...

up to vf 63

VPP error:
2021/08/30 09:15:27:343 debug avf 0000:91:04.1: request_queues: num_queue_pairs 1
2021/08/30 09:15:27:434 debug avf 0000:91:04.1: version: major 1 minor 1
2021/08/30 09:15:27:444 debug avf 0000:91:04.1: get_vf_resources: bitmap 0x180b80a1 (l2 wb-on-itr adv-link-speed vlan-v2 vlan rx-polling rss-pf offload-adv-rss-pf offload-fdir-pf)
2021/08/30 09:15:27:445 debug avf 0000:91:04.1: get_vf_resources: num_vsis 1 num_queue_pairs 1 max_vectors 5 max_mtu 0 vf_cap_flags 0xb0081 (l2 adv-link-speed vlan rx-polling rss-pf) rss_key_size 52 rss_lut_size 64
2021/08/30 09:15:27:445 debug avf 0000:91:04.1: get_vf_resources_vsi[0]: vsi_id 27 num_queue_pairs 1 vsi_type 6 qset_handle 21 default_mac_addr ba:dc:0f:fe:02:11
2021/08/30 09:15:27:445 debug avf 0000:91:04.1: disable_vlan_stripping
2021/08/30 09:15:27:559 error avf 0000:00:00.0: error: avf_send_to_pf: error [v_opcode = 28, v_retval -5]
from avf_create_if: pci-addr 0000:91:04.1

Syslog:
Aug 30 09:15:27 s55-t13-sut1 kernel: [352169.781206] vfio-pci 0000:91:04.1: enabling device (0000 -> 0002)
Aug 30 09:15:27 s55-t13-sut1 kernel: [352170.140729] i40e 0000:91:00.0: Cannot disable vlan stripping when port VLAN is set
Aug 30 09:15:27 s55-t13-sut1 kernel: [352170.140737] i40e 0000:91:00.0: VF 17 failed opcode 28, retval: -5

It looks like the new i40e driver doesn't allow disabling VLAN stripping on VFs with configured VLANs.