Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-1413

VPP main thread gets stuck in a deadlock on running CLI in loop

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Low Low
    • 19.08
    • 18.01
    • CLI
    • vpp# show version verbose
      Version: v18.01.1-21~gcbeb5c5-dirty
      Compiled by: root
      Compile date: Sat Apr 14 05:16:25 BST 2018
      Compile location: /root/vpp_dt
      Compiler: GCC 4.8.5 20150623 (Red Hat 4.8.5-16)

       

       

      I am running a script which fires a VPP CLI in loop and keeps collecting some data. The CLI hangs after sometimes and the main thread gets locked at "mheap_maybe_lock". 
       
      This is happening with other CLI commands as well and for me it is easily reproducible.
       
      Here in a snip from "info thr"
      {{ 2 Thread 0x7fb2337fc700 (LWP 5026) "vpp_stats" 0x00007fb4597bdf3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81}}
      * 1 Thread 0x7fb45be9c740 (LWP 5016) "vpp_main" 0x00007fb459f09a3b in mheap_maybe_lock (v=0x7fb398fc9000)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mheap.c:66}}
       
       
       
      Also, here is the Backtrace taken using GDB :
      (gdb) bt
      #0 0x00007fb459f09a3b in mheap_maybe_lock (v=0x7fb398fc9000)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mheap.c:66}}
      #1 mheap_get_aligned (v=0x7fb398fc9000, n_user_data_bytes=8, n_user_data_bytes@entry=5, align=<optimized out>, align@entry=4,
      {{ align_offset=0, align_offset@entry=4, offset_return=offset_return@entry=0x7fb399c16698)}}
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mheap.c:675}}
      #2 0x00007fb459f33730 in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=4, align=4, size=5)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mem.h:91}}
      #3 vec_resize_allocate_memory (v=<optimized out>, length_increment=length_increment@entry=1, data_bytes=5,
      {{ header_bytes=<optimized out>, header_bytes@entry=0, data_align=data_align@entry=4)}}
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/vec.c:59}}
      #4 0x00007fb45b83477b in _vec_resize (data_align=<optimized out>, header_bytes=<optimized out>, data_bytes=<optimized out>,
      {{ length_increment=<optimized out>, v=<optimized out>)}}
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/vec.h:142}}
      #5 unix_cli_add_pending_output (uf=0x7fb399ccbd4c, buffer=0x7fb45b84a8d6 "\r", buffer_bytes=1, cf=<optimized out>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/unix/cli.c:528}}
      #6 0x00007fb45b8374c0 in unix_cli_file_welcome (cf=0x7fb39a5d4a40, cm=<optimized out>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/unix/cli.c:1137}}
      #7 0x00007fb459f3e1ab in timer_interrupt (signum=<optimized out>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/timer.c:125}}
      #8 <signal handler called>
      #9 mheap_maybe_unlock (v=0x7fb398fc9000) at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mheap.c:85
      #10 mheap_get_aligned (v=0x7fb398fc9000, n_user_data_bytes=<optimized out>, n_user_data_bytes@entry=12, align=<optimized out>,
      {{ align@entry=4, align_offset=0, align_offset@entry=4, offset_return=offset_return@entry=0x7fb399c16ef8)}}
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mheap.c:717}}
      #11 0x00007fb459f33730 in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=4, align=4, size=12)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/mem.h:91}}
      #12 vec_resize_allocate_memory (v=v@entry=0x0, length_increment=1, data_bytes=12, header_bytes=<optimized out>,
      {{ header_bytes@entry=0, data_align=data_align@entry=4)}}
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/vec.c:59}}
      #13 0x00007fb45b83af23 in _vec_resize (data_align=0, header_bytes=0, data_bytes=<optimized out>,
      {{ length_increment=<optimized out>, v=<optimized out>)}}
      --Type <return> to continue, or q <return> to quit--
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/vec.h:142}}
      #14 vlib_process_get_events (data_vector=<synthetic pointer>, vm=0x7fb45ba572c0 <vlib_global_main>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/node_funcs.h:562}}
      #15 unix_cli_process (vm=0x7fb45ba572c0 <vlib_global_main>, rt=0x7fb399c06000, f=<optimized out>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/unix/cli.c:2414}}
      #16 0x00007fb45b8037a6 in vlib_process_bootstrap (_a=<optimized out>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/main.c:1231}}
      #17 0x00007fb459efe808 in clib_calljmp ()
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vppinfra/longjmp.S:110}}
      #18 0x00007fb39a5c8c20 in ?? ()
      #19 0x00007fb45b804ae9 in vlib_process_startup (f=0x0, p=0x7fb399c06000, vm=0x7fb45ba572c0 <vlib_global_main>)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/main.c:1253}}
      #20 dispatch_process (vm=0x7fb45ba572c0 <vlib_global_main>, p=0x7fb399c06000, last_time_stamp=0, f=0x0)
      {{ at /root/vpp_dt/build-root/rpmbuild/vpp-18.01.1.0/build-data/../src/vlib/main.c:1296}}

            chrisluke Chris Luke
            siddsr Siddarth Rai
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: