Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-1566

Interfaces on numa nodes > 1 don't show up in VPP

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: High High
    • 19.08
    • 19.04
    • None
    • None
    • Ubuntu Bionic

      kernel 4.15.0-45-generic

      ARM Taishan server

      A 19.04-rc0~103-g776e46d77 package built on ThunderX1 with 2 numa nodes (or 1 numa node) used on a Taishan server with 4 numa nodes results in VPP not picking up interfaces assigned to numa node 2 or 3:
      testuser@huawei-1:~$ cat /sys/bus/pci/devices/000a\:11\:00.0/numa_node # control against tested iface
      -1

      testuser@huawei-1:~$ sudo /usr/bin/vpp -c /etc/vpp/startup_heasize4G.conf
      vlib_plugin_early_init:361: plugin path /usr/lib/aarch64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins
      load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
      load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
      load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
      load_one_plugin:191: Loaded plugin: cdp_plugin.so
      load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
      load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet)
      load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy)
      load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GTPv1-U)
      load_one_plugin:189: Loaded plugin: igmp_plugin.so (IGMP messaging)
      load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
      load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound OAM)
      load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
      load_one_plugin:189: Loaded plugin: l2e_plugin.so (L2 Emulation)
      load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol)
      load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer)
      load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC source-address filter)
      load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of address and port (MAP))
      load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimental))
      load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation)
      load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header)
      load_one_plugin:189: Loaded plugin: nsim_plugin.so (network delay simulator plugin)
      load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so
      load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPPoE)
      load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
      load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
      load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy)
      load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
      load_one_plugin:189: Loaded plugin: svs_plugin.so (Source VRF Select)
      load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
      load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
      load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
      load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Plugin)
      clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
      load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
      load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
      dpdk_ipsec_process:1010: not enough DPDK crypto resources, default to OpenSSL
      _______ _ _ _____ ___
      _/ __/ _ \ ()__ | | / / _ \/ _ \
      _/ _// // / / / _ \ | |/ / ___/ ___/
      // /__()/_/ |_// /_/

      vpp# show int
      Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
      TenGigabitEthernet11/0/0 1 down 9000/0/0/0
      local0 0 down 0/0/0/0
      vpp# show pci
      Address Sock VID:PID Link Speed Driver Product Name Vital Product Data
      0002:e9:00.0 15b3:1015 8.0 GT/s x8 mlx5_core CX4121A - ConnectX-4 LX SFP28
      0002:e9:00.1 15b3:1015 8.0 GT/s x8 mlx5_core CX4121A - ConnectX-4 LX SFP28
      000a:11:00.0 8086:10fb 5.0 GT/s x8 vfio-pci
      000a:11:00.1 2 8086:10fb 5.0 GT/s x8 vfio-pci
      vpp# show log
      2019/ 2/13 07:28:27:588 warn dpdk EAL init args: -c 2 -n 4 --in-memory --file-prefix vpp --master-lcore 1
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Detected 64 lcore(s)
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Detected 2 NUMA nodes
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Some devices want iova as va but pa will be used because.. EAL: vfio-noiommu mode configured
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Probing VFIO support...
      2019/ 2/13 07:28:28:903 notice dpdk EAL: VFIO support initialized
      2019/ 2/13 07:28:28:903 notice dpdk EAL: PCI device 000a:11:00.0 on NUMA socket -1
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Invalid NUMA socket, default to 0
      2019/ 2/13 07:28:28:903 notice dpdk EAL: probe driver: 8086:10fb net_ixgbe
      2019/ 2/13 07:28:28:903 notice dpdk EAL: using IOMMU type 8 (No-IOMMU)
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Ignore mapping IO port bar(2)
      2019/ 2/13 07:28:28:903 notice dpdk EAL: PCI device 000a:11:00.1 on NUMA socket 2
      2019/ 2/13 07:28:28:903 notice dpdk EAL: probe driver: 8086:10fb net_ixgbe
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Ignore mapping IO port bar(2)
      2019/ 2/13 07:28:28:903 notice dpdk EAL: failed to allocate private dataEAL: Releasing pci mapped resource for 000a:11:00.1
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Calling pci_unmap_resource for 000a:11:00.1 at 0x2101404000
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Calling pci_unmap_resource for 000a:11:00.1 at 0x2101804000
      2019/ 2/13 07:28:28:903 notice dpdk EAL: Requested device 000a:11:00.1 cannot be used
      2019/ 2/13 07:28:28:903 notice dpdk Invalid port_id=1

       

      When using local build (i.e. built directly on a Taishan server) vpp detects 4 NUMA nodes:
      2019/ 2/13 07:30:13:619 notice dpdk EAL: Detected 4 NUMA nodes

       

      A workaround of manually assigning the interfaces to numa 0 or 1 would reflect negatively on VPP performance in performance tests, since the interfaces we need should be on numa node 3:
      lshw -businfo -class network
      pci@000a:11:00.0 enP10p17s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection
      pci@000a:11:00.1 enP10p17s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection
      lstopo
        HostBridge L#4
          PCIBridge
            PCI 8086:10fb
              Net L#6 "enP10p17s0f0"
            PCI 8086:10fb
              Net L#7 "enP10p17s0f1"

            juraj.linkes Juraj Linkeš
            juraj.linkes Juraj Linkeš
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: