-
Bug
-
Resolution: Cannot Reproduce
-
Medium
-
24.02, 24.06
-
None
-
Ubuntu 22.04 running on Triton Datacenter using bhyve. Here is the configuration manifest.
{ "zonename": "d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "autoboot": true, "brand": "bhyve", "limit_priv": "default,-file_link_any,-net_access,-proc_fork,-proc_info,-proc_session", "v": 1, "create_timestamp": "2024-05-22T00:01:23.240Z", "cpu_shares": 64, "max_lwps": 2000, "max_msg_ids": 4096, "max_sem_ids": 4096, "max_shm_ids": 4096, "max_shm_memory": 8192, "zfs_io_priority": 2000, "max_physical_memory": 8448, "max_locked_memory": 8448, "max_swap": 16384, "cpu_cap": 400, "billing_id": "85987143-901b-4da3-b058-96ab955b05a3", "archive_on_delete": true, "disk_driver": "virtio", "nic_driver": "virtio", "resolvers": [ "10.10.60.69" ], "alias": "vpp-test-build", "ram": 8192, "vcpus": 4, "flexible_disk_size": 204800, "device": null, "disks": [ { "path": "/dev/zvol/rdsk/zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4/disk0", "boot": true, "model": "virtio", "media": "disk", "image_size": 10240, "image_uuid": "cccbdd29-adc0-4231-ac2c-26b4a762df5f", "pci_slot": "0:4:0", "uuid": "83662ed3-7447-4d12-908a-0715ed6dac77", "zfs_filesystem": "zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4/disk0", "zpool": "zones", "size": 102400, "compression": "off", "refreservation": 139739, "block_size": 8192 } ], "owner_uuid": "d9e86a13-9980-4059-b03e-efb8dca331d4", "nics": [ { "interface": "net0", "mac": "90:b8:d0:7b:57:3f", "vlan_id": 84, "nic_tag": "external", "gateway": "10.10.84.1", "gateways": [ "10.10.84.1" ], "netmask": "255.255.252.0", "ip": "10.10.84.30", "ips": [ "10.10.84.30/22" ], "network_uuid": "6c0fb8bf-9c3f-46db-b7aa-31acabb2f254", "model": "virtio", "mtu": 1500, "primary": true }, { "interface": "net1", "mac": "90:b8:d0:6e:c1:1c", "vlan_id": 82, "nic_tag": "vppnat", "gateway": "10.10.82.1", "gateways": [ "10.10.82.1" ], "netmask": "255.255.255.0", "ip": "10.10.82.12", "ips": [ "10.10.82.12/24" ], "network_uuid": "72768d0d-054b-4362-8092-4912b3560c5b", "model": "virtio", "allow_dhcp_spoofing": true, "allow_ip_spoofing": true, "allow_mac_spoofing": true, "allow_restricted_traffic": true, "mtu": 1500 }, { "interface": "net2", "mac": "90:b8:d0:25:2a:40", "vlan_id": 83, "nic_tag": "vppint", "netmask": "255.255.255.0", "ip": "10.10.83.12", "ips": [ "10.10.83.12/24" ], "network_uuid": "8e91cc0c-f672-478e-bf04-362fa9b82e0d", "model": "virtio", "allow_dhcp_spoofing": true, "allow_ip_spoofing": true, "allow_mac_spoofing": true, "allow_restricted_traffic": true, "mtu": 1500 } ], "com1": "/dev/zconsole", "com2": "socket,/tmp/vm.ttyb", "bootrom": "uefi", "zlog_mode": "g--", "zlog_name": "platform.log", "uuid": "d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "zone_state": "running", "zonepath": "/zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "hvm": true, "zoneid": 22, "zonedid": 16, "last_modified": "2024-05-22T00:11:59.000Z", "firewall_enabled": false, "server_uuid": "00000000-0000-0000-0000-0cc47a1389e0", "datacenter_name": "us-west-1", "platform_buildstamp": "20240111T002438Z", "state": "running", "boot_timestamp": "2024-05-22T00:01:26.000Z", "init_restarts": 1, "pid": 235935, "customer_metadata": { }, "internal_metadata": { }, "routes": {}, "tags": {}, "quota": 1, "zfs_root_recsize": 131072, "zfs_filesystem": "zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "zpool": "zones", "snapshots": [], "free_space": 102400 }Ubuntu 22.04 running on Triton Datacenter using bhyve. Here is the configuration manifest. { "zonename": "d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "autoboot": true, "brand": "bhyve", "limit_priv": "default,-file_link_any,-net_access,-proc_fork,-proc_info,-proc_session", "v": 1, "create_timestamp": "2024-05-22T00:01:23.240Z", "cpu_shares": 64, "max_lwps": 2000, "max_msg_ids": 4096, "max_sem_ids": 4096, "max_shm_ids": 4096, "max_shm_memory": 8192, "zfs_io_priority": 2000, "max_physical_memory": 8448, "max_locked_memory": 8448, "max_swap": 16384, "cpu_cap": 400, "billing_id": "85987143-901b-4da3-b058-96ab955b05a3", "archive_on_delete": true, "disk_driver": "virtio", "nic_driver": "virtio", "resolvers": [ "10.10.60.69" ], "alias": "vpp-test-build", "ram": 8192, "vcpus": 4, "flexible_disk_size": 204800, "device": null, "disks": [ { "path": "/dev/zvol/rdsk/zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4/disk0", "boot": true, "model": "virtio", "media": "disk", "image_size": 10240, "image_uuid": "cccbdd29-adc0-4231-ac2c-26b4a762df5f", "pci_slot": "0:4:0", "uuid": "83662ed3-7447-4d12-908a-0715ed6dac77", "zfs_filesystem": "zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4/disk0", "zpool": "zones", "size": 102400, "compression": "off", "refreservation": 139739, "block_size": 8192 } ], "owner_uuid": "d9e86a13-9980-4059-b03e-efb8dca331d4", "nics": [ { "interface": "net0", "mac": "90:b8:d0:7b:57:3f", "vlan_id": 84, "nic_tag": "external", "gateway": "10.10.84.1", "gateways": [ "10.10.84.1" ], "netmask": "255.255.252.0", "ip": "10.10.84.30", "ips": [ "10.10.84.30/22" ], "network_uuid": "6c0fb8bf-9c3f-46db-b7aa-31acabb2f254", "model": "virtio", "mtu": 1500, "primary": true }, { "interface": "net1", "mac": "90:b8:d0:6e:c1:1c", "vlan_id": 82, "nic_tag": "vppnat", "gateway": "10.10.82.1", "gateways": [ "10.10.82.1" ], "netmask": "255.255.255.0", "ip": "10.10.82.12", "ips": [ "10.10.82.12/24" ], "network_uuid": "72768d0d-054b-4362-8092-4912b3560c5b", "model": "virtio", "allow_dhcp_spoofing": true, "allow_ip_spoofing": true, "allow_mac_spoofing": true, "allow_restricted_traffic": true, "mtu": 1500 }, { "interface": "net2", "mac": "90:b8:d0:25:2a:40", "vlan_id": 83, "nic_tag": "vppint", "netmask": "255.255.255.0", "ip": "10.10.83.12", "ips": [ "10.10.83.12/24" ], "network_uuid": "8e91cc0c-f672-478e-bf04-362fa9b82e0d", "model": "virtio", "allow_dhcp_spoofing": true, "allow_ip_spoofing": true, "allow_mac_spoofing": true, "allow_restricted_traffic": true, "mtu": 1500 } ], "com1": "/dev/zconsole", "com2": "socket,/tmp/vm.ttyb", "bootrom": "uefi", "zlog_mode": "g--", "zlog_name": "platform.log", "uuid": "d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "zone_state": "running", "zonepath": "/zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "hvm": true, "zoneid": 22, "zonedid": 16, "last_modified": "2024-05-22T00:11:59.000Z", "firewall_enabled": false, "server_uuid": "00000000-0000-0000-0000-0cc47a1389e0", "datacenter_name": "us-west-1", "platform_buildstamp": "20240111T002438Z", "state": "running", "boot_timestamp": "2024-05-22T00:01:26.000Z", "init_restarts": 1, "pid": 235935, "customer_metadata": { }, "internal_metadata": { }, "routes": {}, "tags": {}, "quota": 1, "zfs_root_recsize": 131072, "zfs_filesystem": "zones/d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4", "zpool": "zones", "snapshots": [], "free_space": 102400 }
I've sent a couple of messages to the vpp-dev list on this issue. This seems like the right forum, so I'm adding the information here. Since the message I sent below, I also tested with a build from master with the same results:
vpp v24.06-rc0~284-gf03d44814 built by root on d8cdbd8e-d8fe-4c08-8bef-4fb25c345ae4 at 2024-05-22T00:35:46
At this point, I've tried everything I can think of (other than moving to bare metal, which I don't currently have a system to test with) to resolve this. It doesn't seem like NAT64 is usable with recent releases and code in its current state.
Original message below:
I’m working on getting VPP setup to handle NAT64 for a small ISP. I’ve largely followed https://ipng.ch/s/articles/2021/12/23/vpp-playground.html, with some modifications for the current release (24.02) running on Ubuntu 22.04. Interfaces are assigned by PCI address directly to VPP and segmented from the management interface in a ‘dataplane’ namespace. I’m also using linux_cp and linux_nl with FRR. I’ve witnessed this behavior with every configuration of VPP I’ve tried (initially using just VPP without linux_cp/nl and FRR, and with veth interfaces routed through the main system).
NAT64 works, but each IPv6 request (using ping for testing) results in a loop on the local interface on VPP that doesn’t stop until the TTL is exhausted. In testing, this also results in that ICMP6 echo packet being processed as many times as the TTL allows (lots of DUP! reply messages coming back from the IPv4 ping target). Note that requests from a directly connected interface do not show this behavior; it’s only routed client requests that are a problem.
This is the ping request:
https://gist.github.com/justindthomas/aade7d72d7a4aa5bb816a9e14a595365
This is a packet capture on the switch interface connected to the VPP server showing that the loop is not happening outside of VPP:
https://gist.github.com/justindthomas/e1da8c7a0d702a671d1c4a2291e27472
Here is the VPP trace showing the duplicated echo request packets from packet 8 through 37 and then the response packets in 38 and 39 (the reply and a TTL exceeded message).
https://gist.github.com/justindthomas/699f731533cc957449a9a176df6e4246
Here is the VPP IP6 FIB (note that there is not currently a default route since all clients will come from 2604:2940::/32, so I only have a direct route for that – I have also configured this previously with a default route to the same destination with the same behavior):
https://gist.github.com/justindthomas/cb7c56ed25503f9c3aa161ea57630a2a
Here is the VPP IP FIB:
https://gist.github.com/justindthomas/63ced82309c6d00ecfad5792ea52564e
The bulk of the VPP config is set by this script:
https://gist.github.com/justindthomas/cc39cbf5f02e74070150097e88be494a
I’ll be happy if someone tells me that I just screwed something up and a configuration change would correct it, but it looks like a bug to me. Pings to the IPv6 interface of VPP work fine. And pings from directly connected systems also work fine. It’s just systems that are routed.
