Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-115

VPP crashes during start with RSS enabled for enic

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Medium Medium
    • None
    • None
    • None

      If RSS is enabled using new approach, where number of rx/tx rings can be specified per interface, enic driver segfauts. Stack-trace-1 shows more information. If this exercise is repeated couple of more times, then VPP hangs during start-up. Stack-trace-2 shows where it hangs.

      I'm using VPP debug build, and code is current as of 06/06/2016 9:00AM.

      Below is a snippet of my config file. If I remove enic (0000:09:00.0) and just leave IXGBE (0000:0e:00.0) there, VPP starts up fine. It used to work with "rss 4" earlier.

      dpdk {
      socket-mem 1024
      dev 0000:09:00.0

      { num-rx-queues 4 num-tx-queues 4 }
      dev 0000:0e:00.0 { num-rx-queues 4 num-tx-queues 4 }

      }

      *--------------------------
      STACK-TRACE-1
      -------------------------*

      Program received signal SIGSEGV, Segmentation fault.
      [Switching to Thread 0x7fff9d400700 (LWP 2752)]
      0x000000000055ebbf in vnic_dev_priv (vdev=0x0)
      at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/base/vnic_dev.c:98
      98 return vdev->priv;
      (gdb) where
      #0 0x000000000055ebbf in vnic_dev_priv (vdev=0x0)
      at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/base/vnic_dev.c:98
      #1 0x0000000000559658 in enic_recv_pkts (rx_queue=0x7fff8fbcc6c8, rx_pkts=0x7fffc4db1cc0, nb_pkts=256)
      at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/enic_rxtx.c:248
      #2 0x00007ffff6fbc653 in rte_eth_rx_burst (port_id=0 '\000', queue_id=1, rx_pkts=0x7fffc4db1cc0, nb_pkts=256)
      at /scratch/localadmin/openvpp/vpp/build-root/install-vpp_debug-native/dpdk/include/rte_ethdev.h:2641
      #3 0x00007ffff6fbc8c2 in dpdk_rx_burst (dm=0xb28640 <dpdk_main>, xd=0x7fffc4b97ac0, queue_id=1)
      at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:65
      #4 0x00007ffff6fbdd97 in dpdk_device_input (dm=0xb28640 <dpdk_main>, xd=0x7fffc4b97ac0, node=0x7fffc4e83100, cpu_index=2, queue_id=1,
      use_efd=0) at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/node.c:511
      #5 0x00007ffff6fbee54 in dpdk_input_rss (vm=0x7fffc4e90f14, node=0x7fffc4e83100, f=0x0)
      at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/node.c:822
      #6 0x00007ffff74e4bc5 in dispatch_node (vm=0x7fffc4e90f14, node=0x7fffc4e83100, type=VLIB_NODE_TYPE_INPUT,
      dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, last_time_stamp=2595560569848974)
      at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/main.c:996
      #7 0x00007ffff6fc336d in dpdk_worker_thread_internal (vm=0x7fffc4e90f14, callback=0x0, have_io_threads=0)
      at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:209
      #8 0x00007ffff6fc3598 in dpdk_worker_thread (w=0x7fffc521ca50, io_name=0x7ffff70aae8d "io", callback=0x0)
      at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:265
      #9 0x00007ffff6fc35fa in dpdk_worker_thread_fn (arg=0x7fffc521ca50)
      at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:272
      #10 0x00007ffff6233584 in clib_calljmp () at /scratch/localadmin/openvpp/vpp/build-data/../vppinfra/vppinfra/longjmp.S:110
      #11 0x00007fff9d3ffbb0 in ?? ()
      #12 0x00007ffff75260b0 in vlib_worker_thread_bootstrap_fn (arg=0x7fffc521ca50)
      at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/threads.c:492
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)

      *--------------------------
      STACK-TRACE-2
      -------------------------*

      Starting program: /scratch/localadmin/openvpp/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp -c vppconfigs/vpp_startup.conf
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
      ^C
      Program received signal SIGINT, Interrupt.
      __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
      (gdb) where
      #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1 0x00007ffff5fcb649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007ffff5fcb470 in _GI__pthread_mutex_lock (mutex=0x30008008) at ../nptl/pthread_mutex_lock.c:79
      #3 0x00007ffff68f8577 in region_lock (rp=0x30008000, tag=2) at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:62
      #4 0x00007ffff68fa54b in svm_map_region (a=0x7fffc571cce0) at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:590
      #5 0x00007ffff68fa8b5 in svm_region_init_internal (root_path=0x0, uid=-1, gid=-1)
      at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:653
      #6 0x00007ffff68fac6f in svm_region_init_chroot_uid_gid (root_path=0x0, uid=-1, gid=-1)
      at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:689
      #7 0x0000000000468bac in gmon_init (vm=0xb28840 <vlib_global_main>) at /scratch/localadmin/openvpp/vpp/build-data/../vpp/api/gmon.c:174
      #8 0x00007ffff74d911d in vlib_call_init_exit_functions (vm=0xb28840 <vlib_global_main>, head=0xaecda0 <_vlib_init_function.21287>,
      call_once=1) at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/init.c:57
      #9 0x00007ffff74d91a8 in vlib_call_all_init_functions (vm=0xb28840 <vlib_global_main>)
      at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/init.c:74
      #10 0x00007ffff74e738e in vlib_main (vm=0xb28840 <vlib_global_main>, input=0x7fffc571cfb0)
      at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/main.c:1576
      #11 0x00007ffff7784a89 in thread0 (arg=11700288) at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/unix/main.c:425
      #12 0x00007ffff6233584 in clib_calljmp () at /scratch/localadmin/openvpp/vpp/build-data/../vppinfra/vppinfra/longjmp.S:110
      #13 0x00007fffffffd3d0 in ?? ()
      #14 0x00007ffff7784f13 in vlib_unix_main (argc=46, argv=0xbbab00)
      at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/unix/main.c:485
      #15 0x000000000040b1e2 in main (argc=46, argv=0xbbab00) at /scratch/localadmin/openvpp/vpp/build-data/../vpp/vnet/main.c:246

            shesha Shesha Sreenivasamurthy
            shesha Shesha Sreenivasamurthy
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: