-
Bug
-
Resolution: Cannot Reproduce
-
High
-
19.04
-
None
-
None
-
Ubuntu Bionic
kernel 4.15.0-45-generic
ARM Taishan server
A 19.04-rc0~103-g776e46d77 package built on ThunderX1 with 2 numa nodes (or 1 numa node) used on a Taishan server with 4 numa nodes results in VPP not picking up interfaces assigned to numa node 2 or 3:
testuser@huawei-1:~$ cat /sys/bus/pci/devices/000a\:11\:00.0/numa_node # control against tested iface
-1
testuser@huawei-1:~$ sudo /usr/bin/vpp -c /etc/vpp/startup_heasize4G.conf
vlib_plugin_early_init:361: plugin path /usr/lib/aarch64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins
load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
load_one_plugin:191: Loaded plugin: cdp_plugin.so
load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet)
load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy)
load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GTPv1-U)
load_one_plugin:189: Loaded plugin: igmp_plugin.so (IGMP messaging)
load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound OAM)
load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
load_one_plugin:189: Loaded plugin: l2e_plugin.so (L2 Emulation)
load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol)
load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer)
load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC source-address filter)
load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of address and port (MAP))
load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimental))
load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation)
load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header)
load_one_plugin:189: Loaded plugin: nsim_plugin.so (network delay simulator plugin)
load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so
load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPPoE)
load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy)
load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
load_one_plugin:189: Loaded plugin: svs_plugin.so (Source VRF Select)
load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Plugin)
clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
dpdk_ipsec_process:1010: not enough DPDK crypto resources, default to OpenSSL
_______ _ _ _____ ___
_/ __/ _ \ ()__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
// /__()/_/ |_// /_/
vpp# show int
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
TenGigabitEthernet11/0/0 1 down 9000/0/0/0
local0 0 down 0/0/0/0
vpp# show pci
Address Sock VID:PID Link Speed Driver Product Name Vital Product Data
0002:e9:00.0 15b3:1015 8.0 GT/s x8 mlx5_core CX4121A - ConnectX-4 LX SFP28
0002:e9:00.1 15b3:1015 8.0 GT/s x8 mlx5_core CX4121A - ConnectX-4 LX SFP28
000a:11:00.0 8086:10fb 5.0 GT/s x8 vfio-pci
000a:11:00.1 2 8086:10fb 5.0 GT/s x8 vfio-pci
vpp# show log
2019/ 2/13 07:28:27:588 warn dpdk EAL init args: -c 2 -n 4 --in-memory --file-prefix vpp --master-lcore 1
2019/ 2/13 07:28:28:903 notice dpdk EAL: Detected 64 lcore(s)
2019/ 2/13 07:28:28:903 notice dpdk EAL: Detected 2 NUMA nodes
2019/ 2/13 07:28:28:903 notice dpdk EAL: Some devices want iova as va but pa will be used because.. EAL: vfio-noiommu mode configured
2019/ 2/13 07:28:28:903 notice dpdk EAL: Probing VFIO support...
2019/ 2/13 07:28:28:903 notice dpdk EAL: VFIO support initialized
2019/ 2/13 07:28:28:903 notice dpdk EAL: PCI device 000a:11:00.0 on NUMA socket -1
2019/ 2/13 07:28:28:903 notice dpdk EAL: Invalid NUMA socket, default to 0
2019/ 2/13 07:28:28:903 notice dpdk EAL: probe driver: 8086:10fb net_ixgbe
2019/ 2/13 07:28:28:903 notice dpdk EAL: using IOMMU type 8 (No-IOMMU)
2019/ 2/13 07:28:28:903 notice dpdk EAL: Ignore mapping IO port bar(2)
2019/ 2/13 07:28:28:903 notice dpdk EAL: PCI device 000a:11:00.1 on NUMA socket 2
2019/ 2/13 07:28:28:903 notice dpdk EAL: probe driver: 8086:10fb net_ixgbe
2019/ 2/13 07:28:28:903 notice dpdk EAL: Ignore mapping IO port bar(2)
2019/ 2/13 07:28:28:903 notice dpdk EAL: failed to allocate private dataEAL: Releasing pci mapped resource for 000a:11:00.1
2019/ 2/13 07:28:28:903 notice dpdk EAL: Calling pci_unmap_resource for 000a:11:00.1 at 0x2101404000
2019/ 2/13 07:28:28:903 notice dpdk EAL: Calling pci_unmap_resource for 000a:11:00.1 at 0x2101804000
2019/ 2/13 07:28:28:903 notice dpdk EAL: Requested device 000a:11:00.1 cannot be used
2019/ 2/13 07:28:28:903 notice dpdk Invalid port_id=1
When using local build (i.e. built directly on a Taishan server) vpp detects 4 NUMA nodes:
2019/ 2/13 07:30:13:619 notice dpdk EAL: Detected 4 NUMA nodes
A workaround of manually assigning the interfaces to numa 0 or 1 would reflect negatively on VPP performance in performance tests, since the interfaces we need should be on numa node 3:
lshw -businfo -class network
pci@000a:11:00.0 enP10p17s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection
pci@000a:11:00.1 enP10p17s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection
lstopo
HostBridge L#4
PCIBridge
PCI 8086:10fb
Net L#6 "enP10p17s0f0"
PCI 8086:10fb
Net L#7 "enP10p17s0f1"