Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-1534

VPP crash on mpls tunnel delete.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Open
    • Icon: High High
    • None
    • None
    • MPLS
    • None

      Openstack programs  the VPP with sr mpls and lfib routes. The following sequence consistently reproducing the crash. 

      Create MPLS tunnel, Delete MPLS tunnel, Create MPLS tunnel, Delete MPLS tunnel and CRASH.  

       

      FIB state before the crash:

      vpp# show sr mpls policiesvpp# show sr mpls policiesSR MPLS policies:[0].- BSID: 101 TE disabled Type: Default Segment Lists:  [0].- < 22000 >-----------vpp# show mpls fibMPLS-VRF:0, fib_index 0ip4-explicit-null:neos/21 fib:0 index:10 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[14] locks:2 flags:exclusive, uPRF-list:10 len:0 itfs:[]      path:[14] pl-index:14 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dst-address,unicast lookup in interface's mpls table
       forwarding:   mpls-neos-chain  [@0]: dpo-load-balance: [proto:mpls index:13 buckets:1 uRPF:10 to:[0:0]]    [0] [@4]: dst-address,unicast lookup in interface's mpls tableip4-explicit-null:eos/21 fib:0 index:9 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[13] locks:2 flags:exclusive, uPRF-list:9 len:0 itfs:[]      path:[13] pl-index:13 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dst-address,unicast lookup in interface's ip4 table
       forwarding:   mpls-eos-chain  [@0]: dpo-load-balance: [proto:mpls index:12 buckets:1 uRPF:9 to:[0:0]]    [0] [@3]: dst-address,unicast lookup in interface's ip4 tablerouter-alert:neos/21 fib:0 index:7 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[11] locks:2 flags:exclusive, uPRF-list:7 len:0 itfs:[]      path:[11] pl-index:11 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dpo-punt
       forwarding:   mpls-neos-chain  [@0]: dpo-load-balance: [proto:mpls index:10 buckets:1 uRPF:7 to:[0:0]]    [0] [@2]: dpo-puntrouter-alert:eos/21 fib:0 index:8 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[12] locks:2 flags:exclusive, uPRF-list:8 len:0 itfs:[]      path:[12] pl-index:12 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dpo-punt
       forwarding:   mpls-eos-chain  [@0]: dpo-load-balance: [proto:mpls index:11 buckets:1 uRPF:8 to:[0:0]]    [0] [@2]: dpo-puntipv6-explicit-null:neos/21 fib:0 index:12 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[16] locks:2 flags:exclusive, uPRF-list:12 len:0 itfs:[]      path:[16] pl-index:16 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dst-address,unicast lookup in interface's mpls table
       forwarding:   mpls-neos-chain  [@0]: dpo-load-balance: [proto:mpls index:15 buckets:1 uRPF:12 to:[0:0]]    [0] [@4]: dst-address,unicast lookup in interface's mpls tableipv6-explicit-null:eos/21 fib:0 index:11 locks:2  src:special refs:1 entry-flags:exclusive, src-flags:added,contributing,active,    path-list:[15] locks:2 flags:exclusive, uPRF-list:11 len:0 itfs:[]      path:[15] pl-index:15 mpls weight=1 pref=0 exclusive:  oper-flags:resolved, cfg-flags:exclusive,        [@0]: dst-address,unicast lookup in interface's ip6 table
       forwarding:   mpls-eos-chain  [@0]: dpo-load-balance: [proto:mpls index:14 buckets:1 uRPF:11 to:[0:0]]    [0] [@5]: dst-address,unicast lookup in interface's ip6 table101:neos/21 fib:0 index:23 locks:4  src:SR refs:1 src-flags:added,contributing,active,    path-list:[27] locks:4 flags:shared, uPRF-list:27 len:2 itfs:[1, 2, ]      path:[27] pl-index:27 mpls weight=1 pref=0 recursive:  oper-flags:resolved,        via 22000 neos in fib:0 via-fib:25 via-dpo:[dpo-load-balance:28]    Extensions:     path:27  labels:[[implicit-null pipe ttl:0 exp:0]]  src:recursive-resolution refs:1 src-flags:added, cover:-1
       forwarding:   mpls-neos-chain  [@0]: dpo-load-balance: [proto:mpls index:26 buckets:1 uRPF:27 to:[0:0]]    [0] [@7]: dpo-load-balance: [proto:mpls index:28 buckets:2 uRPF:28 to:[0:0]]          [0] [@6]: mpls-label[@3]:[22000:64:0:neos]              [@1]: mpls via 50.50.50.1 TenGigabitEthernet6/0/1: mtu:9000 008a965a68583cfdfeb58d998847          [1] [@6]: mpls-label[@2]:[22000:64:0:neos]              [@1]: mpls via 60.60.60.1 TenGigabitEthernet81/0/1: mtu:9000 008a965a58583cfdfeb58a898847101:eos/21 fib:0 index:24 locks:2  src:SR refs:1 src-flags:added,contributing,active,    path-list:[27] locks:4 flags:shared, uPRF-list:27 len:2 itfs:[1, 2, ]      path:[27] pl-index:27 mpls weight=1 pref=0 recursive:  oper-flags:resolved,        via 22000 neos in fib:0 via-fib:25 via-dpo:[dpo-load-balance:28]    Extensions:     path:27  labels:[[implicit-null pipe ttl:0 exp:0]] forwarding:   mpls-eos-chain  [@0]: dpo-load-balance: [proto:mpls index:27 buckets:1 uRPF:27 to:[0:0]]    [0] [@7]: dpo-load-balance: [proto:mpls index:28 buckets:2 uRPF:28 to:[0:0]]          [0] [@6]: mpls-label[@3]:[22000:64:0:neos]              [@1]: mpls via 50.50.50.1 TenGigabitEthernet6/0/1: mtu:9000 008a965a68583cfdfeb58d998847          [1] [@6]: mpls-label[@2]:[22000:64:0:neos]              [@1]: mpls via 60.60.60.1 TenGigabitEthernet81/0/1: mtu:9000 008a965a58583cfdfeb58a8988475555:eos/21 fib:0 index:26 locks:2  src:API refs:1 entry-flags:attached, src-flags:added,contributing,active,    path-list:[29] locks:2 flags:shared, uPRF-list:30 len:0 itfs:[]      path:[31] pl-index:29 ethernet weight=1 pref=0 intf-rx:  oper-flags:resolved, cfg-flags:interface-rx,        [@0]: mpls-tunnel0-rx-dpo: ethernet
       forwarding:   mpls-eos-chain  [@0]: dpo-load-balance: [proto:mpls index:30 buckets:1 uRPF:30 to:[0:0]]    [0] [@8]: mpls-tunnel0-rx-dpo: ethernet22000:neos/21 fib:0 index:25 locks:4  src:API refs:1 src-flags:added,contributing,active,    path-list:[31] locks:2 flags:shared, uPRF-list:28 len:2 itfs:[1, 2, ]      path:[30] pl-index:31 ip4 weight=1 pref=0 attached-nexthop:  oper-flags:resolved,        50.50.50.1 TenGigabitEthernet6/0/1      [@0]: ipv4 via 50.50.50.1 TenGigabitEthernet6/0/1: mtu:9000 008a965a68583cfdfeb58d990800      path:[33] pl-index:31 ip4 weight=1 pref=0 attached-nexthop:  oper-flags:resolved,        60.60.60.1 TenGigabitEthernet81/0/1      [@0]: ipv4 via 60.60.60.1 TenGigabitEthernet81/0/1: mtu:9000 008a965a58583cfdfeb58a890800    Extensions:     path:30  labels:[[22000 pipe ttl:64 exp:0]]     path:33  labels:[[22000 pipe ttl:64 exp:0]]  src:recursive-resolution refs:1 src-flags:added, cover:-1
       forwarding:   mpls-neos-chain  [@0]: dpo-load-balance: [proto:mpls index:28 buckets:2 uRPF:28 to:[0:0]]    [0] [@6]: mpls-label[@3]:[22000:64:0:neos]        [@1]: mpls via 50.50.50.1 TenGigabitEthernet6/0/1: mtu:9000 008a965a68583cfdfeb58d998847    [1] [@6]: mpls-label[@2]:[22000:64:0:neos]        [@1]: mpls via 60.60.60.1 TenGigabitEthernet81/0/1: mtu:9000 008a965a58583cfdfeb58a898847vpp# show mpls tunnel[@0] mpls_tunnel0: sw_if_index:4 hw_if_index:4 flags:L2, via:    path-list:[29] locks:2 flags:shared, uPRF-list:30 len:0 itfs:[]      path:[31] pl-index:29 ethernet weight=1 pref=0 intf-rx:  oper-flags:resolved, cfg-flags:interface-rx,        [@0]: mpls-tunnel0-rx-dpo: ethernet    Extensions:     path:28 mpls-flags:[no-ip-tll-decr] labels:[[5555 pipe ttl:64 exp:0]] forwarding: ethernet [@1]: dpo-load-balance: [proto:ethernet index:31 buckets:1 uRPF:-1 to:[0:0]]    [0] [@2]: mpls-label[@1]:[5555:64:0:eos]        [@1]: dpo-load-balance: [proto:mpls index:26 buckets:1 uRPF:27 to:[0:0]]            [0] [@7]: dpo-load-balance: [proto:mpls index:28 buckets:2 uRPF:28 to:[0:0]]                  [0] [@6]: mpls-label[@3]:[22000:64:0:neos]                      [@1]: mpls via 50.50.50.1 TenGigabitEthernet6/0/1: mtu:9000 008a965a68583cfdfeb58d998847                  [1] [@6]: mpls-label[@2]:[22000:64:0:neos]                      [@1]: mpls via 60.60.60.1 TenGigabitEthernet81/0/1: mtu:9000 008a965a58583cfdfeb58a898847

      vpp# show version verbose
      Version: v18.10-14~gad5f2de~b61
      Compiled by: root
      Compile host: e3159adecd01
      Compile date: Sat Dec 1 00:42:40 UTC 2018
      Compile location: /w/workspace/vpp-merge-1810-centos7
      Compiler: GCC 7.3.1 20180303 (Red Hat 7.3.1-5)

      (gdb) bt
      #0 0x00007f772dd721f7 in raise () from /lib64/libc.so.6
      #1 0x00007f772dd738e8 in abort () from /lib64/libc.so.6
      #2 0x0000000000407cde in os_exit (code=code@entry=1) at /usr/src/debug/vpp-18.10/src/vpp/vnet/main.c:349
      #3 0x00007f772f42a48c in unix_signal_handler (signum=<optimized out>, si=<optimized out>, uc=<optimized out>)
      at /usr/src/debug/vpp-18.10/src/vlib/unix/main.c:157
      #4 <signal handler called>
      #5 fib_node_list_extract (elt=<optimized out>, elt=<optimized out>, head=<optimized out>) at /usr/src/debug/vpp-18.10/src/vnet/fib/fib_node_list.c:212
      #6 fib_node_list_remove (list=<optimized out>, sibling=sibling@entry=36) at /usr/src/debug/vpp-18.10/src/vnet/fib/fib_node_list.c:243
      #7 0x00007f772fec79ec in fib_node_child_remove (parent_type=<optimized out>, parent_index=<optimized out>, sibling_index=36)
      at /usr/src/debug/vpp-18.10/src/vnet/fib/fib_node.c:131
      #8 0x00007f772fed1dde in fib_path_list_child_remove (path_list_index=<optimized out>, si=<optimized out>)
      at /usr/src/debug/vpp-18.10/src/vnet/fib/fib_path_list.c:1265
      #9 0x00007f772fc8d178 in vnet_mpls_tunnel_path_remove (sw_if_index=sw_if_index@entry=4, rpaths=rpaths@entry=0x7f76ee0077cc)
      at /usr/src/debug/vpp-18.10/src/vnet/mpls/mpls_tunnel.c:732
      #10 0x00007f772fc8e891 in vl_api_mpls_tunnel_add_del_t_handler (mp=0x1300d0ae8) at /usr/src/debug/vpp-18.10/src/vnet/mpls/mpls_api.c:391
      #11 0x00007f773033adb7 in vl_msg_api_handler_with_vm_node (am=am@entry=0x7f7730546cc0 <api_main>, the_msg=0x1300d0ae8,
      vm=vm@entry=0x7f772f640200 <vlib_global_main>, node=node@entry=0x7f76eca2c000) at /usr/src/debug/vpp-18.10/src/vlibapi/api_shared.c:502
      #12 0x00007f773031cc25 in void_mem_api_handle_msg_i (am=<optimized out>, q=<optimized out>, node=0x7f76eca2c000, vm=0x7f772f640200 <vlib_global_main>)
      at /usr/src/debug/vpp-18.10/src/vlibmemory/memory_api.c:692
      #13 vl_mem_api_handle_msg_main (vm=vm@entry=0x7f772f640200 <vlib_global_main>, node=node@entry=0x7f76eca2c000)
      at /usr/src/debug/vpp-18.10/src/vlibmemory/memory_api.c:702
      #14 0x00007f773032daa4 in vl_api_clnt_process (vm=<optimized out>, node=0x7f76eca2c000, f=<optimized out>)
      --Type <return> to continue, or q <return> to quit--
      at /usr/src/debug/vpp-18.10/src/vlibmemory/vlib_api.c:349
      #15 0x00007f772f3fd116 in vlib_process_bootstrap (_a=<optimized out>) at /usr/src/debug/vpp-18.10/src/vlib/main.c:1232
      #16 0x00007f772eb05694 in clib_calljmp () from /lib64/libvppinfra.so.18.10
      #17 0x00007f76ecdffe10 in ?? ()
      #18 0x00007f772f3fe289 in dispatch_process (vm=0x7f772f640200 <vlib_global_main>, p=0x7f76eca2c000, last_time_stamp=776620797906, f=0x0)
      at /usr/src/debug/vpp-18.10/src/vlib/main.c:1254
      #19 0x01b50c000035c704 in ?? ()
      #20 0x576f0418000011e6 in ?? ()
      #21 0x00077401bd0c0000 in ?? ()
      #22 0x0c00003952042000 in ?? ()
      #23 0x04280000077401c3 in ?? ()
      #24 0x4b01c40c00004e08 in ?? ()
      #25 0x0062633130000009 in ?? ()
      #26 0x380000103701c70c in ?? ()
      #27 0x01c80c000024cd04 in ?? ()
      #28 0x420e006000000083 in ?? ()
      #29 0x24000011c2000000 in ?? ()
      #30 0x19080600000000d2 in ?? ()
      #31 0x0000000000000133 in ?? ()
      #32 0x2fd22bc000000000 in ?? ()
      #33 0xeca9024c00007f77 in ?? ()
      #34 0x00000a2400007f76 in ?? ()
      Current PID: 166

            nranns Neale Ranns
            vasumahalingam Vasu Mahalingam
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: