If RSS is enabled using new approach, where number of rx/tx rings can be specified per interface, enic driver segfauts. Stack-trace-1 shows more information. If this exercise is repeated couple of more times, then VPP hangs during start-up. Stack-trace-2 shows where it hangs.
I'm using VPP debug build, and code is current as of 06/06/2016 9:00AM.
Below is a snippet of my config file. If I remove enic (0000:09:00.0) and just leave IXGBE (0000:0e:00.0) there, VPP starts up fine. It used to work with "rss 4" earlier.
dpdk {
socket-mem 1024
dev 0000:09:00.0
dev 0000:0e:00.0 { num-rx-queues 4 num-tx-queues 4 }
}
*--------------------------
STACK-TRACE-1
-------------------------*
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff9d400700 (LWP 2752)]
0x000000000055ebbf in vnic_dev_priv (vdev=0x0)
at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/base/vnic_dev.c:98
98 return vdev->priv;
(gdb) where
#0 0x000000000055ebbf in vnic_dev_priv (vdev=0x0)
at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/base/vnic_dev.c:98
#1 0x0000000000559658 in enic_recv_pkts (rx_queue=0x7fff8fbcc6c8, rx_pkts=0x7fffc4db1cc0, nb_pkts=256)
at /scratch/localadmin/openvpp/vpp/build-root/build-vpp_debug-native/dpdk/dpdk-16.04/drivers/net/enic/enic_rxtx.c:248
#2 0x00007ffff6fbc653 in rte_eth_rx_burst (port_id=0 '\000', queue_id=1, rx_pkts=0x7fffc4db1cc0, nb_pkts=256)
at /scratch/localadmin/openvpp/vpp/build-root/install-vpp_debug-native/dpdk/include/rte_ethdev.h:2641
#3 0x00007ffff6fbc8c2 in dpdk_rx_burst (dm=0xb28640 <dpdk_main>, xd=0x7fffc4b97ac0, queue_id=1)
at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:65
#4 0x00007ffff6fbdd97 in dpdk_device_input (dm=0xb28640 <dpdk_main>, xd=0x7fffc4b97ac0, node=0x7fffc4e83100, cpu_index=2, queue_id=1,
use_efd=0) at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/node.c:511
#5 0x00007ffff6fbee54 in dpdk_input_rss (vm=0x7fffc4e90f14, node=0x7fffc4e83100, f=0x0)
at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/node.c:822
#6 0x00007ffff74e4bc5 in dispatch_node (vm=0x7fffc4e90f14, node=0x7fffc4e83100, type=VLIB_NODE_TYPE_INPUT,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, last_time_stamp=2595560569848974)
at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/main.c:996
#7 0x00007ffff6fc336d in dpdk_worker_thread_internal (vm=0x7fffc4e90f14, callback=0x0, have_io_threads=0)
at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#8 0x00007ffff6fc3598 in dpdk_worker_thread (w=0x7fffc521ca50, io_name=0x7ffff70aae8d "io", callback=0x0)
at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#9 0x00007ffff6fc35fa in dpdk_worker_thread_fn (arg=0x7fffc521ca50)
at /scratch/localadmin/openvpp/vpp/build-data/../vnet/vnet/devices/dpdk/threads.c:272
#10 0x00007ffff6233584 in clib_calljmp () at /scratch/localadmin/openvpp/vpp/build-data/../vppinfra/vppinfra/longjmp.S:110
#11 0x00007fff9d3ffbb0 in ?? ()
#12 0x00007ffff75260b0 in vlib_worker_thread_bootstrap_fn (arg=0x7fffc521ca50)
at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/threads.c:492
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
*--------------------------
STACK-TRACE-2
-------------------------*
Starting program: /scratch/localadmin/openvpp/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp -c vppconfigs/vpp_startup.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
vlib_plugin_early_init:201: plugin path /usr/lib/vpp_plugins
^C
Program received signal SIGINT, Interrupt.
__lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) where
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007ffff5fcb649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007ffff5fcb470 in _GI__pthread_mutex_lock (mutex=0x30008008) at ../nptl/pthread_mutex_lock.c:79
#3 0x00007ffff68f8577 in region_lock (rp=0x30008000, tag=2) at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:62
#4 0x00007ffff68fa54b in svm_map_region (a=0x7fffc571cce0) at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:590
#5 0x00007ffff68fa8b5 in svm_region_init_internal (root_path=0x0, uid=-1, gid=-1)
at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:653
#6 0x00007ffff68fac6f in svm_region_init_chroot_uid_gid (root_path=0x0, uid=-1, gid=-1)
at /scratch/localadmin/openvpp/vpp/build-data/../svm/svm.c:689
#7 0x0000000000468bac in gmon_init (vm=0xb28840 <vlib_global_main>) at /scratch/localadmin/openvpp/vpp/build-data/../vpp/api/gmon.c:174
#8 0x00007ffff74d911d in vlib_call_init_exit_functions (vm=0xb28840 <vlib_global_main>, head=0xaecda0 <_vlib_init_function.21287>,
call_once=1) at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/init.c:57
#9 0x00007ffff74d91a8 in vlib_call_all_init_functions (vm=0xb28840 <vlib_global_main>)
at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/init.c:74
#10 0x00007ffff74e738e in vlib_main (vm=0xb28840 <vlib_global_main>, input=0x7fffc571cfb0)
at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/main.c:1576
#11 0x00007ffff7784a89 in thread0 (arg=11700288) at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/unix/main.c:425
#12 0x00007ffff6233584 in clib_calljmp () at /scratch/localadmin/openvpp/vpp/build-data/../vppinfra/vppinfra/longjmp.S:110
#13 0x00007fffffffd3d0 in ?? ()
#14 0x00007ffff7784f13 in vlib_unix_main (argc=46, argv=0xbbab00)
at /scratch/localadmin/openvpp/vpp/build-data/../vlib/vlib/unix/main.c:485
#15 0x000000000040b1e2 in main (argc=46, argv=0xbbab00) at /scratch/localadmin/openvpp/vpp/build-data/../vpp/vnet/main.c:246