-
Bug
-
Resolution: Unresolved
-
High
-
None
-
23.10
-
None
-
show version verbose cmdline:
Version: v23.10.0-22~g6b287b530-dirty
{ nodaemon log /home/ubuntu/vpp/vpp.log cli-listen /run/vpp/cli.sock gid vpp startup-config /home/ubuntu/upf/interface.conf coredump-size unlimited full-coredump }
Compiled by: root
Compile host: ubuntu
Compile date: 2024-03-05T04:06:44
Compile location: /home/ubuntu/vpp
Compiler: Clang/LLVM 14.0.0
Current PID: 1714
Command line arguments:
/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp
unix
api-trace
{ on }
api-segment
{ gid vpp }
socksvr
{ default }
cpu
{ main-core 1 workers 1 }
dpdk
{ name n3 }
{
dev
66847e0e-6e9d-4ff3-950f-8082c80b5cf1
dev
{ name local_n6 }
f22432a8-dda0-4d1c-a34e-8a44c21d4466
uio-driver
{ disable }
uio_hv_generic
log-level
debug
}
plugins
{
path
/home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/lib/x86_64-linux-gnu/vpp_plugins
plugin
default
plugin
{ enable }
dpdk_plugin.so
plugin
ping_plugin.so
{ enable }}
{ default-log-level debug default-syslog-log-level debug }
logging
OS: Ubuntu 22.04.4 LTS.
NIC type: Netvsc. Under it, in the case of the HP server, HPE Ethernet 1Gb 4-port 331FLR and HPE Ethernet 1Gb 4-port 331i. In the case of the HP laptop, just the hypervisor, as the ICMP source is located in the Hyper-V host (internal network virtual switches).
CLI command to setup the interfaces:
set interface ip address n3 172.31.128.133/24
set interface mtu packet 1500 n3
set interface ip address local_n6 172.31.130.3/24
set interface mtu packet 1500 local_n6
set interface state n3 up
set interface state local_n6 upshow ip fib:
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[default-route:1, ]
0.0.0.0/0
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]]
[0] [@0]: dpo-drop ip4
0.0.0.0/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
[0] [@0]: dpo-drop ip4
172.31.128.0/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:11 to:[0:0]]
[0] [@0]: dpo-drop ip4
172.31.128.0/24
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:8 buckets:1 uRPF:10 to:[0:0]]
[0] [@4]: ipv4-glean: [src:172.31.128.0/24] n3: mtu:1500 next:1 flags:[] ffffffffffff00155d0183230806
172.31.128.133/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:15 to:[0:0]]
[0] [@11]: dpo-receive: 172.31.128.133 on n3
172.31.128.255/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:13 to:[8:926]]
[0] [@0]: dpo-drop ip4
172.31.130.0/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:17 to:[0:0]]
[0] [@0]: dpo-drop ip4
172.31.130.0/24
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:16 to:[0:0]]
[0] [@4]: ipv4-glean: [src:172.31.130.0/24] local_n6: mtu:1500 next:2 flags:[] ffffffffffff00155d0183150806
172.31.130.3/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:21 to:[0:0]]
[0] [@11]: dpo-receive: 172.31.130.3 on local_n6
172.31.130.255/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:19 to:[8:926]]
[0] [@0]: dpo-drop ip4
224.0.0.0/4
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
[0] [@0]: dpo-drop ip4
240.0.0.0/4
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
[0] [@0]: dpo-drop ip4
255.255.255.255/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
[0] [@0]: dpo-drop ip4show ip6 fib:
ipv6-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[default-route:1, ]
::/0
unicast-ip6-chain
[@0]: dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:5 to:[0:0]]
[0] [@0]: dpo-drop ip6
fe80::/10
unicast-ip6-chain
[@0]: dpo-load-balance: [proto:ip6 index:7 buckets:1 uRPF:6 to:[0:0]]
[0] [@14]: ip6-link-localshow hardware:
Name Idx Link Hardware
local0 0 down local0
Link speed: unknown
local
local_n6 2 up local_n6
Link speed: 10 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
TX Queues:
TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers]
queue shared thread(s)
0 no 0
1 no 1
Ethernet address 00:15:5d:01:83:15
Microsoft Hyper-V Netvsc
carrier up full duplex max-frame-size 0
flags: admin-up tx-offload rx-ip4-cksum
Devargs:
rx: queues 1 (max 64), desc 1024 (min 0 max 65535 align 1)
tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1)
max rx packet len: 2048
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6
rss active: none
tx burst function: hn_xmit_pkts
rx burst function: hn_recv_pktsrx frames ok 61
rx bytes ok 11943
extended stats:
rx_good_packets 61
rx_good_bytes 11943
rx_q0_packets 61
rx_q0_bytes 11943
rx_q0_good_packets 61
rx_q0_good_bytes 11943
rx_q0_multicast_packets 53
rx_q0_broadcast_packets 8
rx_q0_size_65_127_packets 10
rx_q0_size_128_255_packets 51
n3 1 up n3
Link speed: 10 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
TX Queues:
TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers]
queue shared thread(s)
0 no 0
1 no 1
Ethernet address 00:15:5d:01:83:23
Microsoft Hyper-V Netvsc
carrier up full duplex max-frame-size 0
flags: admin-up tx-offload rx-ip4-cksum
Devargs:
rx: queues 1 (max 64), desc 1024 (min 0 max 65535 align 1)
tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1)
max rx packet len: 2048
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6
rss active: none
tx burst function: hn_xmit_pkts
rx burst function: hn_recv_pktsrx frames ok 61
rx bytes ok 11943
extended stats:
rx_good_packets 61
rx_good_bytes 11943
rx_q0_packets 61
rx_q0_bytes 11943
rx_q0_good_packets 61
rx_q0_good_bytes 11943
rx_q0_multicast_packets 53
rx_q0_broadcast_packets 8
rx_q0_size_65_127_packets 10
rx_q0_size_128_255_packets 51show interface:
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
local0 0 down 0/0/0/0
local_n6 2 up 1500/0/0/0 rx packets 65
rx bytes 12811
drops 65
ip4 65
n3 1 up 1500/0/0/0 rx packets 65
rx bytes 12811
drops 65
ip4 65show interface address:
local0 (dn):
local_n6 (up):
L3 172.31.130.3/24
n3 (up):
L3 172.31.128.133/24show version verbose cmdline: Version: v23.10.0-22~g6b287b530-dirty Compiled by: root Compile host: ubuntu Compile date: 2024-03-05T04:06:44 Compile location: /home/ubuntu/vpp Compiler: Clang/LLVM 14.0.0 Current PID: 1714 Command line arguments: /home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp unix { nodaemon log /home/ubuntu/vpp/vpp.log cli-listen /run/vpp/cli.sock gid vpp startup-config /home/ubuntu/upf/interface.conf coredump-size unlimited full-coredump } api-trace { on } api-segment { gid vpp } socksvr { default } cpu { main-core 1 workers 1 } dpdk { dev 66847e0e-6e9d-4ff3-950f-8082c80b5cf1 { name n3 } dev f22432a8-dda0-4d1c-a34e-8a44c21d4466 { name local_n6 } uio-driver uio_hv_generic log-level debug } plugins { path /home/ubuntu/vpp/build-root/install-vpp_debug-native/vpp/lib/x86_64-linux-gnu/vpp_plugins plugin default { disable } plugin dpdk_plugin.so { enable } plugin ping_plugin.so { enable } } logging { default-log-level debug default-syslog-log-level debug } OS: Ubuntu 22.04.4 LTS. NIC type: Netvsc. Under it, in the case of the HP server, HPE Ethernet 1Gb 4-port 331FLR and HPE Ethernet 1Gb 4-port 331i. In the case of the HP laptop, just the hypervisor, as the ICMP source is located in the Hyper-V host (internal network virtual switches). CLI command to setup the interfaces: set interface ip address n3 172.31.128.133/24 set interface mtu packet 1500 n3 set interface ip address local_n6 172.31.130.3/24 set interface mtu packet 1500 local_n6 set interface state n3 up set interface state local_n6 up show ip fib: ipv4-VRF:0, fib_index:0, flow hash: [src dst sport dport proto flowlabel ] epoch:0 flags:none locks: [default-route:1, ] 0.0.0.0/0 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to: [0:0] ] [0] [@0] : dpo-drop ip4 0.0.0.0/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to: [0:0] ] [0] [@0] : dpo-drop ip4 172.31.128.0/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:11 to: [0:0] ] [0] [@0] : dpo-drop ip4 172.31.128.0/24 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:8 buckets:1 uRPF:10 to: [0:0] ] [0] [@4] : ipv4-glean: [src:172.31.128.0/24] n3: mtu:1500 next:1 flags:[] ffffffffffff00155d0183230806 172.31.128.133/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:15 to: [0:0] ] [0] [@11] : dpo-receive: 172.31.128.133 on n3 172.31.128.255/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:13 to: [8:926] ] [0] [@0] : dpo-drop ip4 172.31.130.0/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:17 to: [0:0] ] [0] [@0] : dpo-drop ip4 172.31.130.0/24 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:16 to: [0:0] ] [0] [@4] : ipv4-glean: [src:172.31.130.0/24] local_n6: mtu:1500 next:2 flags:[] ffffffffffff00155d0183150806 172.31.130.3/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:21 to: [0:0] ] [0] [@11] : dpo-receive: 172.31.130.3 on local_n6 172.31.130.255/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:19 to: [8:926] ] [0] [@0] : dpo-drop ip4 224.0.0.0/4 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to: [0:0] ] [0] [@0] : dpo-drop ip4 240.0.0.0/4 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to: [0:0] ] [0] [@0] : dpo-drop ip4 255.255.255.255/32 unicast-ip4-chain [@0] : dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to: [0:0] ] [0] [@0] : dpo-drop ip4 show ip6 fib: ipv6-VRF:0, fib_index:0, flow hash: [src dst sport dport proto flowlabel ] epoch:0 flags:none locks: [default-route:1, ] ::/0 unicast-ip6-chain [@0] : dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:5 to: [0:0] ] [0] [@0] : dpo-drop ip6 fe80::/10 unicast-ip6-chain [@0] : dpo-load-balance: [proto:ip6 index:7 buckets:1 uRPF:6 to: [0:0] ] [0] [@14] : ip6-link-local show hardware: Name Idx Link Hardware local0 0 down local0 Link speed: unknown local local_n6 2 up local_n6 Link speed: 10 Gbps RX Queues: queue thread mode 0 vpp_wk_0 (1) polling TX Queues: TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers] queue shared thread(s) 0 no 0 1 no 1 Ethernet address 00:15:5d:01:83:15 Microsoft Hyper-V Netvsc carrier up full duplex max-frame-size 0 flags: admin-up tx-offload rx-ip4-cksum Devargs: rx: queues 1 (max 64), desc 1024 (min 0 max 65535 align 1) tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1) max rx packet len: 2048 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash rx offload active: ipv4-cksum tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6 rss active: none tx burst function: hn_xmit_pkts rx burst function: hn_recv_pkts rx frames ok 61 rx bytes ok 11943 extended stats: rx_good_packets 61 rx_good_bytes 11943 rx_q0_packets 61 rx_q0_bytes 11943 rx_q0_good_packets 61 rx_q0_good_bytes 11943 rx_q0_multicast_packets 53 rx_q0_broadcast_packets 8 rx_q0_size_65_127_packets 10 rx_q0_size_128_255_packets 51 n3 1 up n3 Link speed: 10 Gbps RX Queues: queue thread mode 0 vpp_wk_0 (1) polling TX Queues: TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers] queue shared thread(s) 0 no 0 1 no 1 Ethernet address 00:15:5d:01:83:23 Microsoft Hyper-V Netvsc carrier up full duplex max-frame-size 0 flags: admin-up tx-offload rx-ip4-cksum Devargs: rx: queues 1 (max 64), desc 1024 (min 0 max 65535 align 1) tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1) max rx packet len: 2048 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash rx offload active: ipv4-cksum tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6 rss active: none tx burst function: hn_xmit_pkts rx burst function: hn_recv_pkts rx frames ok 61 rx bytes ok 11943 extended stats: rx_good_packets 61 rx_good_bytes 11943 rx_q0_packets 61 rx_q0_bytes 11943 rx_q0_good_packets 61 rx_q0_good_bytes 11943 rx_q0_multicast_packets 53 rx_q0_broadcast_packets 8 rx_q0_size_65_127_packets 10 rx_q0_size_128_255_packets 51 show interface: Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count local0 0 down 0/0/0/0 local_n6 2 up 1500/0/0/0 rx packets 65 rx bytes 12811 drops 65 ip4 65 n3 1 up 1500/0/0/0 rx packets 65 rx bytes 12811 drops 65 ip4 65 show interface address: local0 (dn): local_n6 (up): L3 172.31.130.3/24 n3 (up): L3 172.31.128.133/24
We are trying to run VPP on an Ubuntu 22.04 with worker threads over Hyper-V.
We don't have a DPDK implementing HW NIC yet, nor SR-IOV, so we are using the paravirtualized Hyper-V NIC with DPDK support over the Netvsc PMD.
We have tested it in an HP server and an HP laptop, and both fail.
The setup can be so simple as just loading the DPDK and ping plugins.
If we comment out the cpu section of startup.conf (i.e., we configure VPP to use a single thread), it works.
As soon as we add one worker and send a bunch of ICMP Echo Request to one of the interfaces, VPP aborts:
cpu
{ main-core 1 workers 1 }The coredump always says it is a problem in a DPDK spinlock accessed by Netvsc:
Thread 3 "vpp_wk_0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa7d2a640 (LWP 1296)]
0x00007fffb0190710 in rte_spinlock_trylock (sl=0x20) at ../src-dpdk/lib/eal/x86/include/rte_spinlock.h:63
63 asm volatile (
(gdb) bt
#0 0x00007fffb0190710 in rte_spinlock_trylock (sl=0x20) at ../src-dpdk/lib/eal/x86/include/rte_spinlock.h:63
#1 0x00007fffb01bd531 in hn_process_events (hv=0xac03a7bc0, queue_id=1, tx_limit=0) at ../src-dpdk/drivers/net/netvsc/hn_rxtx.c:1075
#2 0x00007fffb01c55d3 in hn_xmit_pkts (ptxq=0xac08a5040, tx_pkts=0x7fffb7b32e00, nb_pkts=1) at ../src-dpdk/drivers/net/netvsc/hn_rxtx.c:1497
#3 0x00007fffb0b5bbf2 in rte_eth_tx_burst (port_id=0, queue_id=1, tx_pkts=0x7fffb7b32e00, nb_pkts=1)
at /home/ubuntu/vpp/build-root/install-vpp_debug-native/external/include/rte_ethdev.h:6320
#4 0x00007fffb0b57799 in tx_burst_vector_internal (vm=0x7fffb7af8b80, xd=0x7fffb7b37c40, mb=0x7fffb7b32e00, n_left=1, queue_id=1, is_shared=0 '\000')
at /home/ubuntu/vpp/src/plugins/dpdk/device/device.c:173
#5 0x00007fffb0b56ac2 in dpdk_device_class_tx_fn_hsw (vm=0x7fffb7af8b80, node=0x7fffb7b54540, f=0x7fffb7bcfdc0)
at /home/ubuntu/vpp/src/plugins/dpdk/device/device.c:423
#6 0x00007ffff7062a32 in dispatch_node (vm=0x7fffb7af8b80, node=0x7fffb7b54540, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING,
frame=0x7fffb7bcfdc0, last_time_stamp=29933360537300) at /home/ubuntu/vpp/src/vlib/main.c:960
#7 0x00007ffff7063452 in dispatch_pending_node (vm=0x7fffb7af8b80, pending_frame_index=9, last_time_stamp=29933360537300)
at /home/ubuntu/vpp/src/vlib/main.c:1119
#8 0x00007ffff705e7d2 in vlib_main_or_worker_loop (vm=0x7fffb7af8b80, is_main=0) at /home/ubuntu/vpp/src/vlib/main.c:1608
#9 0x00007ffff705ded7 in vlib_worker_loop (vm=0x7fffb7af8b80) at /home/ubuntu/vpp/src/vlib/main.c:1741
#10 0x00007ffff709b050 in vlib_worker_thread_fn (arg=0x7fffb865d600) at /home/ubuntu/vpp/src/vlib/threads.c:1604
#11 0x00007ffff7096236 in vlib_worker_thread_bootstrap_fn (arg=0x7fffb865d600) at /home/ubuntu/vpp/src/vlib/threads.c:418
#12 0x00007ffff6c86ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007ffff6d18850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81