VPP 18.10/19.01 crashes with very simple interface configuration on two DPDK interfaces. Running Ixia RFC2544 tests VPP crashes. Ixia is setup with 4000 src/dst IPs (i.e. 8000 overall) and 4000 flows (1:1 mapping). 

root@7454ee796c9f:/vpp/vpp# git rev-parse HEAD
98d6b61183ecb15932fbdae96a60e0d0fcf97466
root@7454ee796c9f:/vpp/vpp#

The coredump and post_mortem files can be found here: https://cisco.box.com/s/60ul297qeh8ts4l6i97nxxfxsspdi31y

 root@7454ee796c9f:/vpp/vpp# make STARTUP_CONF=/root/starup.conf debug
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /vpp/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp...done.
Signal Stop Print Pass to program Description
SIGUSR1 No No Yes User defined signal 1
(gdb) run
Starting program: /vpp/vpp/build-root/install-vpp_debug-native/vpp/bin/vpp unix { interactive log /var/log/vpp/vpp.log full-coredump cli-listen localhost:5002 } api-trace { on } dpdk { socket-mem 1024,1024 uio-driver igb_uio dev 0000:62:00.0 dev 0000:68:00.0 } cpu { main-core 25 corelist-workers 26-29 }
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
vlib_plugin_early_init:361: plugin path /vpp/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
load_one_plugin:191: Loaded plugin: cdp_plugin.so
load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet)
load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy)
load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GTPv1-U)
load_one_plugin:189: Loaded plugin: igmp_plugin.so (IGMP messaging)
load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound OAM)
load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
load_one_plugin:189: Loaded plugin: l2e_plugin.so (L2 Emulation)
load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol)
load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer)
load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC source-address filter)
load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of address and port (MAP))
load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimental))
load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation)
load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header)
load_one_plugin:189: Loaded plugin: nsim_plugin.so (network delay simulator plugin)
load_one_plugin:189: Loaded plugin: perfmon_plugin.so (Performance monitor plugin)
load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPPoE)
load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy)
load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
load_one_plugin:189: Loaded plugin: svs_plugin.so (Source VRF Select)
load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Plugin)
clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
unix_proc_file_contents: open `/vpp/vpp/build-root/install-vpp_debug-native/vpp/share/vpp/plugins/perfmon/skylakex_core_v1.12.json': No such file or directory
load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
dpdk_config:1407: socket-mem argument is deprecated
[New Thread 0x7ff9442a1700 (LWP 34109)]
[New Thread 0x7ff943aa0700 (LWP 34110)]
[New Thread 0x7ff93b29f700 (LWP 34111)]
[New Thread 0x7ff94329f700 (LWP 34112)]
[New Thread 0x7ff942a9e700 (LWP 34113)]
0: dpdk_ipsec_process:1010: not enough DPDK crypto resources, default to OpenSSL
_______ _ _ _____ ___
_/ __/ _ \ ()__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
// /__()/_/ |_// /_/
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp# set interface ip address TenGigabitEthernet62/0/0 db11::1/64
DBGvpp# set interface state TenGigabitEthernet62/0/0 up
DBGvpp#
DBGvpp# set interface ip address TenGigabitEthernet68/0/0 db22::1/64
DBGvpp# set interface state TenGigabitEthernet68/0/0 up
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp#
DBGvpp# 2: vl_msg_api_alloc_internal:119: garbage collect pool 0 ring 2 index 0
2: vl_msg_api_alloc_internal:124: msg id 8 name rpc_call
1: vl_msg_api_alloc_internal:119: garbage collect pool 0 ring 1 index 0
1: vl_msg_api_alloc_internal:124: msg id 8 name rpc_call
1: vl_msg_api_alloc_internal:119: garbage collect pool 0 ring 2 index 1
1: vl_msg_api_alloc_internal:124: msg id 8 name rpc_call
2: vl_msg_api_alloc_internal:119: garbage collect pool 0 ring 2 index 2
2: vl_msg_api_alloc_internal:124: msg id 8 name rpc_call
Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
0x00007ffff61c3855 in mspace_usable_size (mem=0x301c28b5) at /vpp/vpp/src/vppinfra/dlmalloc.c:4795
4795 if (is_inuse(p))
(gdb) bt
#0 0x00007ffff61c3855 in mspace_usable_size (mem=0x301c28b5) at /vpp/vpp/src/vppinfra/dlmalloc.c:4795
#1 0x00007ffff61c0de6 in mspace_put (msp=0x13004a010, p_arg=0x1301c28b8) at /vpp/vpp/src/vppinfra/dlmalloc.c:4284
#2 0x00007ffff7b90b37 in clib_mem_free (p=0x1301c28b8) at /vpp/vpp/src/vppinfra/mem.h:215
#3 0x00007ffff7b91789 in vl_msg_api_free (a=0x1301c28c8) at /vpp/vpp/src/vlibmemory/memory_shared.c:291
#4 0x00007ffff7bc2e3a in vl_msg_api_handler_with_vm_node (am=0x7ffff7dd3c60 <api_main>, the_msg=0x1301c28c8,
vm=0x7ffff68fc380 <vlib_global_main>, node=0x7fffb5615000) at /vpp/vpp/src/vlibapi/api_shared.c:516
#5 0x00007ffff7b8f2ca in vl_mem_api_handle_rpc (vm=0x7ffff68fc380 <vlib_global_main>, node=0x7fffb5615000)
at /vpp/vpp/src/vlibmemory/memory_api.c:741
#6 0x00007ffff7bad62e in vl_api_clnt_process (vm=0x7ffff68fc380 <vlib_global_main>, node=0x7fffb5615000, f=0x0)
at /vpp/vpp/src/vlibmemory/vlib_api.c:349
#7 0x00007ffff6671779 in vlib_process_bootstrap (_a=140736232160240) at /vpp/vpp/src/vlib/main.c:1438
#8 0x00007ffff61264e4 in clib_calljmp () from /vpp/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01
#9 0x00007fffb51ffbc0 in ?? ()
#10 0x00007ffff66718a4 in vlib_process_startup (vm=0x7ffff667231a <dispatch_suspended_process+862>, p=0x7fffb51ffcc0,
f=0xa4914a5a2bc4db00) at /vpp/vpp/src/vlib/main.c:1460
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)