-
Bug
-
Resolution: Duplicate
-
Medium
-
None
-
None
-
None
-
None
This does not happen right away but occasionally during the process of programming ACLs by the networking-vpp agent.
vpp# show version verbose
Version: v17.04-rc0~81-g655fcc4
Compiled by: root
Compile host: sjo-smf-ubuntu-server-2
Compile date: Tue Jan 10 17:41:27 PST 2017
Compile location: /home/demo/src/vpp
Compiler: GCC 5.4.0 20160609
Current PID: 5845
DPDK Version: DPDK 16.11.0
DPDK EAL init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp -b 0000:08:00.0 -b 0000:09:00.0 --master-lcore 0 --socket-mem 256,256
Here's the console output and backtrace of all the threads:
VPP console output when it is hung:
vpp# 0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 0 ip6: 1
0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 1 ip6: 0
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 2 ip6: 3
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 3 ip6: 2
vpp#
vpp#
vpp# 0: send_acl_details:1538: Sending acl details for ACL index 0
0: send_acl_details:1538: Sending acl details for ACL index 1
0: send_acl_details:1538: Sending acl details for ACL index 2
0: send_acl_details:1538: Sending acl details for ACL index 3
0: send_acl_details:1538: Sending acl details for ACL index 4
0: send_acl_details:1538: Sending acl details for ACL index 5
0: send_acl_details:1538: Sending acl details for ACL index 6
0: send_acl_details:1538: Sending acl details for ACL index 7
0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 8 ip6: 9
0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 9 ip6: 8
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 10 ip6: 11
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 11 ip6: 10
vpp#
vpp# 0: svm_client_scan_this_region_nolock:1139: /vpe-api: cleanup ghost pid 30698
0: svm_client_scan_this_region_nolock:1139: /vpe-api: cleanup ghost pid 30873
0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 8 ip6: 9
0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 9 ip6: 8
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 10 ip6: 11
0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 11 ip6: 10
<VPP HUNG>
GDB trace:
(gdb) info threads
Id Target Id Frame
- 1 Thread 0x7ffff7fe38c0 (LWP 30865) "vpp_main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
2 Thread 0x7fff722a8700 (LWP 30867) "vpp_stats" 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
3 Thread 0x7fff72aa9700 (LWP 30866) "eal-intr-thread" 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007ffff79be5bb in unix_shared_memory_queue_add (q=q@entry=0x305fec80, elem=elem@entry=0x7fffb5a01e68 "<7G0", nowait=nowait@entry=0)
at /home/demo/src/vpp/build-data/../src/vlibmemory/unix_shared_memory_queue.c:216
#2 0x00007ffff79b14dd in vl_msg_api_send_shmem (q=q@entry=0x305fec80, elem=elem@entry=0x7fffb5a01e68 "<7G0")
at /home/demo/src/vpp/build-data/../src/vlibmemory/memory_shared.c:547
#3 0x00000000005d2ba6 in vl_api_control_ping_t_handler (mp=0x3052573c) at /home/demo/src/vpp/build-data/../src/vpp/api/api.c:1277
#4 0x00007ffff7bcc9c3 in vl_msg_api_handler_with_vm_node (am=am@entry=0x7ffff7dd62e0 <api_main>, the_msg=the_msg@entry=0x3052573c,
vm=vm@entry=0x7ffff79aa940 <vlib_global_main>, node=node@entry=0x7fffb59f9000)
at /home/demo/src/vpp/build-data/../src/vlibapi/api_shared.c:510
#5 0x00007ffff79b86b4 in memclnt_process (vm=<optimized out>, node=0x7fffb59f9000, f=<optimized out>)
at /home/demo/src/vpp/build-data/../src/vlibmemory/memory_vlib.c:487
#6 0x00007ffff774e146 in vlib_process_bootstrap (_a=<optimized out>) at /home/demo/src/vpp/build-data/../src/vlib/main.c:1218
#7 0x00007ffff68a1850 in clib_calljmp () at /home/demo/src/vpp/build-data/../src/vppinfra/longjmp.S:110
#8 0x00007fffb5b41e20 in ?? ()
#9 0x00007ffff774f0f9 in vlib_process_startup (f=0x0, p=0x7fffb59f9000, vm=0x7ffff79aa940 <vlib_global_main>)
at /home/demo/src/vpp/build-data/../src/vlib/main.c:1240
#10 dispatch_process (vm=0x7ffff79aa940 <vlib_global_main>, p=0x7fffb59f9000, last_time_stamp=6216603291757294, f=0x0)
at /home/demo/src/vpp/build-data/../src/vlib/main.c:1283
#11 0x0000000300000002 in ?? ()
#12 0x6361727400000005 in ?? ()
#13 0x0000000300000065 in ?? ()
#14 0x000000250000000c in ?? ()
#15 0x736c706d2d367069 in ?? ()
#16 0x692d6c6562616c2d in ?? ()
#17 0x6f697469736f706d in ?? ()
#18 0x6e72757465722d6e in ?? ()
#19 0x000000006425203a in ?? ()
#20 0x0000000c00000000 in ?? ()
#21 0x000000230000000c in ?? ()
#22 0x736c706d2d367069 in ?? ()
#23 0x692d6c6562616c2d in ?? ()
#24 0x6f697469736f706d in ?? ()
#25 0x203a6c6c61632d6e in ?? ()
#26 0x0000000000006425 in ?? ()
#27 0x0000000c00000000 in ?? ()
#28 0x0000001900000008 in ?? ()
#29 0x736c706d2d347069 in ?? ()
#30 0x692d6c6562616c2d in ?? ()
#31 0x6f697469736f706d in ?? ()
#32 0x000000080000006e in ?? ()
#33 0x0000001900000008 in ?? ()
#34 0x736c706d2d367069 in ?? ()
#35 0x692d6c6562616c2d in ?? ()
#36 0x6f697469736f706d in ?? ()
--Type <return> to continue, or q <return> to quit--
#37 0x000000080000006e in ?? ()
#38 0xf746494000000046 in ?? ()
#39 0xb5a0215000007fff in ?? ()
#40 0x0000032e00007fff in ?? ()
#41 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fff722a8700 (LWP 30867))]
#0 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1 0x00000000005bc4a6 in ip46_fib_stats_delay (sec=sec@entry=10, nsec=nsec@entry=0, sm=0xbdada0 <stats_main>)
at /home/demo/src/vpp/build-data/../src/vpp/stats/stats.c:251
#2 0x00000000005c34a8 in stats_thread_fn (arg=<optimized out>) at /home/demo/src/vpp/build-data/../src/vpp/stats/stats.c:626
#3 0x00007ffff68a1850 in clib_calljmp () at /home/demo/src/vpp/build-data/../src/vppinfra/longjmp.S:110
#4 0x00007fff722a7da0 in ?? ()
#5 0x00007ffff63596fa in start_thread (arg=0xf200000004) at pthread_create.c:333
#6 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 0x7fff72aa9700 (LWP 30866))]
#0 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00000000004fbf54 in eal_intr_handle_interrupts (totalfds=<optimized out>, pfd=13)
at /home/demo/src/vpp/build-root/build-vpp-native/dpdk/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_interrupts.c:768
#2 eal_intr_thread_main (arg=<optimized out>)
at /home/demo/src/vpp/build-root/build-vpp-native/dpdk/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_interrupts.c:852
#3 0x00007ffff63596fa in start_thread (arg=0x7fff72aa9700) at pthread_create.c:333
#4 0x00007ffff5e8bb5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109