Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-597

VPP goes into a hung state while ACLs are being programmed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Medium Medium
    • None
    • None
    • None
    • None

      This does not happen right away but occasionally during the process of programming ACLs by the networking-vpp agent.

      vpp# show version verbose
      Version: v17.04-rc0~81-g655fcc4
      Compiled by: root
      Compile host: sjo-smf-ubuntu-server-2
      Compile date: Tue Jan 10 17:41:27 PST 2017
      Compile location: /home/demo/src/vpp
      Compiler: GCC 5.4.0 20160609
      Current PID: 5845
      DPDK Version: DPDK 16.11.0
      DPDK EAL init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp -b 0000:08:00.0 -b 0000:09:00.0 --master-lcore 0 --socket-mem 256,256

      Here's the console output and backtrace of all the threads:
      VPP console output when it is hung:

      vpp# 0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 0 ip6: 1
      0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 1 ip6: 0
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 2 ip6: 3
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 4, setting tables to the following: ip4: 3 ip6: 2

      vpp#
      vpp#
      vpp# 0: send_acl_details:1538: Sending acl details for ACL index 0
      0: send_acl_details:1538: Sending acl details for ACL index 1
      0: send_acl_details:1538: Sending acl details for ACL index 2
      0: send_acl_details:1538: Sending acl details for ACL index 3
      0: send_acl_details:1538: Sending acl details for ACL index 4
      0: send_acl_details:1538: Sending acl details for ACL index 5
      0: send_acl_details:1538: Sending acl details for ACL index 6
      0: send_acl_details:1538: Sending acl details for ACL index 7
      0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 8 ip6: 9
      0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 9 ip6: 8
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 10 ip6: 11
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 11 ip6: 10

      vpp#
      vpp# 0: svm_client_scan_this_region_nolock:1139: /vpe-api: cleanup ghost pid 30698
      0: svm_client_scan_this_region_nolock:1139: /vpe-api: cleanup ghost pid 30873
      0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 8 ip6: 9
      0: acl_hook_l2_input_classify:545: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 9 ip6: 8
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 10 ip6: 11
      0: acl_hook_l2_output_classify:603: ACL enabling on interface sw_if_index 6, setting tables to the following: ip4: 11 ip6: 10
      <VPP HUNG>

      GDB trace:

      (gdb) info threads
      Id Target Id Frame

      • 1 Thread 0x7ffff7fe38c0 (LWP 30865) "vpp_main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        2 Thread 0x7fff722a8700 (LWP 30867) "vpp_stats" 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
        3 Thread 0x7fff72aa9700 (LWP 30866) "eal-intr-thread" 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
        (gdb) bt
        #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        #1 0x00007ffff79be5bb in unix_shared_memory_queue_add (q=q@entry=0x305fec80, elem=elem@entry=0x7fffb5a01e68 "<7G0", nowait=nowait@entry=0)
        at /home/demo/src/vpp/build-data/../src/vlibmemory/unix_shared_memory_queue.c:216
        #2 0x00007ffff79b14dd in vl_msg_api_send_shmem (q=q@entry=0x305fec80, elem=elem@entry=0x7fffb5a01e68 "<7G0")
        at /home/demo/src/vpp/build-data/../src/vlibmemory/memory_shared.c:547
        #3 0x00000000005d2ba6 in vl_api_control_ping_t_handler (mp=0x3052573c) at /home/demo/src/vpp/build-data/../src/vpp/api/api.c:1277
        #4 0x00007ffff7bcc9c3 in vl_msg_api_handler_with_vm_node (am=am@entry=0x7ffff7dd62e0 <api_main>, the_msg=the_msg@entry=0x3052573c,
        vm=vm@entry=0x7ffff79aa940 <vlib_global_main>, node=node@entry=0x7fffb59f9000)
        at /home/demo/src/vpp/build-data/../src/vlibapi/api_shared.c:510
        #5 0x00007ffff79b86b4 in memclnt_process (vm=<optimized out>, node=0x7fffb59f9000, f=<optimized out>)
        at /home/demo/src/vpp/build-data/../src/vlibmemory/memory_vlib.c:487
        #6 0x00007ffff774e146 in vlib_process_bootstrap (_a=<optimized out>) at /home/demo/src/vpp/build-data/../src/vlib/main.c:1218
        #7 0x00007ffff68a1850 in clib_calljmp () at /home/demo/src/vpp/build-data/../src/vppinfra/longjmp.S:110
        #8 0x00007fffb5b41e20 in ?? ()
        #9 0x00007ffff774f0f9 in vlib_process_startup (f=0x0, p=0x7fffb59f9000, vm=0x7ffff79aa940 <vlib_global_main>)
        at /home/demo/src/vpp/build-data/../src/vlib/main.c:1240
        #10 dispatch_process (vm=0x7ffff79aa940 <vlib_global_main>, p=0x7fffb59f9000, last_time_stamp=6216603291757294, f=0x0)
        at /home/demo/src/vpp/build-data/../src/vlib/main.c:1283
        #11 0x0000000300000002 in ?? ()
        #12 0x6361727400000005 in ?? ()
        #13 0x0000000300000065 in ?? ()
        #14 0x000000250000000c in ?? ()
        #15 0x736c706d2d367069 in ?? ()
        #16 0x692d6c6562616c2d in ?? ()
        #17 0x6f697469736f706d in ?? ()
        #18 0x6e72757465722d6e in ?? ()
        #19 0x000000006425203a in ?? ()
        #20 0x0000000c00000000 in ?? ()
        #21 0x000000230000000c in ?? ()
        #22 0x736c706d2d367069 in ?? ()
        #23 0x692d6c6562616c2d in ?? ()
        #24 0x6f697469736f706d in ?? ()
        #25 0x203a6c6c61632d6e in ?? ()
        #26 0x0000000000006425 in ?? ()
        #27 0x0000000c00000000 in ?? ()
        #28 0x0000001900000008 in ?? ()
        #29 0x736c706d2d347069 in ?? ()
        #30 0x692d6c6562616c2d in ?? ()
        #31 0x6f697469736f706d in ?? ()
        #32 0x000000080000006e in ?? ()
        #33 0x0000001900000008 in ?? ()
        #34 0x736c706d2d367069 in ?? ()
        #35 0x692d6c6562616c2d in ?? ()
        #36 0x6f697469736f706d in ?? ()
        --Type <return> to continue, or q <return> to quit--
        #37 0x000000080000006e in ?? ()
        #38 0xf746494000000046 in ?? ()
        #39 0xb5a0215000007fff in ?? ()
        #40 0x0000032e00007fff in ?? ()
        #41 0x0000000000000000 in ?? ()

      (gdb) thread 2
      [Switching to thread 2 (Thread 0x7fff722a8700 (LWP 30867))]
      #0 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      84 ../sysdeps/unix/syscall-template.S: No such file or directory.
      (gdb) bt
      #0 0x00007ffff6362c5d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      #1 0x00000000005bc4a6 in ip46_fib_stats_delay (sec=sec@entry=10, nsec=nsec@entry=0, sm=0xbdada0 <stats_main>)
      at /home/demo/src/vpp/build-data/../src/vpp/stats/stats.c:251
      #2 0x00000000005c34a8 in stats_thread_fn (arg=<optimized out>) at /home/demo/src/vpp/build-data/../src/vpp/stats/stats.c:626
      #3 0x00007ffff68a1850 in clib_calljmp () at /home/demo/src/vpp/build-data/../src/vppinfra/longjmp.S:110
      #4 0x00007fff722a7da0 in ?? ()
      #5 0x00007ffff63596fa in start_thread (arg=0xf200000004) at pthread_create.c:333
      #6 0x0000000000000000 in ?? ()

      (gdb) thread 3
      [Switching to thread 3 (Thread 0x7fff72aa9700 (LWP 30866))]
      #0 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      84 ../sysdeps/unix/syscall-template.S: No such file or directory.
      (gdb) bt
      #0 0x00007ffff5e8c153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1 0x00000000004fbf54 in eal_intr_handle_interrupts (totalfds=<optimized out>, pfd=13)
      at /home/demo/src/vpp/build-root/build-vpp-native/dpdk/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_interrupts.c:768
      #2 eal_intr_thread_main (arg=<optimized out>)
      at /home/demo/src/vpp/build-root/build-vpp-native/dpdk/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_interrupts.c:852
      #3 0x00007ffff63596fa in start_thread (arg=0x7fff72aa9700) at pthread_create.c:333
      #4 0x00007ffff5e8bb5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

            Unassigned Unassigned
            naveenjoy Naveen Joy
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: