Uploaded image for project: 'vpp'
  1. vpp
  2. VPP-1700

GSO issue in Contiv-VPP delpoyment

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Medium Medium
    • 19.08
    • None
    • None
    • None

      In Contiv-VPP deployment with v19.04.1-release, we are hitting an issue after deploying istio pods when GSO is enabled on TAP interfaces going towards the istio pods. The same works if GSO is disabled.

      Istio pilot, policy and telemetry pods are crashing ans reporting connectivity issues:

      $ kubectl get pods -n istio-system -o wide
NAME                                     READY   STATUS              RESTARTS   AGE   IP          NODE          NOMINATED NODE
istio-citadel-55cdfdd57c-d7v2h           1/1     Running             0          21m   10.1.2.77   k8s-worker1   <none>
istio-cleanup-secrets-pllcg              0/1     Completed           0          21m   10.1.2.70   k8s-worker1   <none>
istio-egressgateway-7798845f5d-25xhp     1/1     Running             0          21m   10.1.2.72   k8s-worker1   <none>
istio-galley-76bbb946c8-mmbcr            0/1     ContainerCreating   0          21m   <none>      k8s-worker1   <none>
istio-ingressgateway-78c6d8b8d7-fqxv4    1/1     Running             0          21m   10.1.2.73   k8s-worker1   <none>
istio-pilot-5fcb895bff-hwtr2             1/2     CrashLoopBackOff    14         21m   10.1.2.76   k8s-worker1   <none>
istio-policy-7b6cc95d7b-bx5qz            1/2     CrashLoopBackOff    11         21m   10.1.2.74   k8s-worker1   <none>
istio-sidecar-injector-9c6698858-hhl7k   0/1     ContainerCreating   0          21m   <none>      k8s-worker1   <none>
istio-telemetry-bfc9ff784-lfqsh          1/2     CrashLoopBackOff    11         21m   10.1.2.75   k8s-worker1   <none>
prometheus-65d6f6b6c-nkmtv               1/1     Running             0          21m   10.1.2.78   k8s-worker1   <none>

      Error logs of the failed pods:

      2019-06-12T12:35:58.045297Z error Unexpected error when reading response body: &net.OpError{Op:"local error", Net:"", Source:net.Addr(nil), Addr:net.Addr(nil), Err:0x14}

      The only suspicious thing that I was able to figure out on VPP so far is vmxnet3-input errors counting (even though there is no vmxnet3 interface enabled on VPP):

      vpp# sh error
   Count                    Node                  Reason
    345636              vmxnet3-input             Rx packet error - no SOP
    379586              vmxnet3-input             Rx packet error
    364611        nat44-ed-out2in-slowpath        good out2in packets processed
     20113        nat44-ed-out2in-slowpath        TCP packets
       764        nat44-ed-out2in-slowpath        UDP packets
         8        nat44-ed-out2in-slowpath        ICMP packets
    362918             nat44-ed-out2in            good out2in packets processed
    354890             nat44-ed-out2in            TCP packets
      8028             nat44-ed-out2in            UDP packets
         2     nat44-ed-in2out-output-slowpath    unsupported ICMP type
    385783        nat44-ed-in2out-slowpath        good in2out packets processed
        61        nat44-ed-in2out-slowpath        TCP packets
        61        nat44-ed-in2out-slowpath        UDP packets
         4        nat44-ed-in2out-slowpath        ICMP packets
    343899             nat44-ed-in2out            good in2out packets processed
    335107             nat44-ed-in2out            TCP packets
    335107             nat44-ed-in2out            UDP packets
         3                arp-input               ARP replies sent
        22                arp-input               ARP request IP4 source address learned
        70                ip4-input               Multicast RPF check failed
    343717              vxlan4-input              good packets decapsulated
    383806              vxlan4-encap              good packets encapsulated
    383806                l2-output               L2 output packets
    727523                l2-input                L2 input packets
         4             ethernet-input             no error

      show interface:

      vpp# show interface 
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
GigabitEthernet0/8/0              1      up          9000/0/0/0     rx packets               1844603
                                                                    rx bytes              1460153091
                                                                    tx packets               2097491
                                                                    tx bytes               320458161
                                                                    drops                         69
                                                                    ip4                      1844511
local0                            0     down          0/0/0/0       
loop0                             2      up          9000/0/0/0     
loop1                             5      up          9000/0/0/0     rx packets               1844511
                                                                    rx bytes              1342098867
                                                                    tx packets               4192098
                                                                    tx bytes               431103066
                                                                    ip4                      1844511
tap0                              3      up          1450/0/0/0     rx packets                  4778
                                                                    rx bytes                  426686
                                                                    tx packets                  4581
                                                                    tx bytes                  444119
                                                                    drops                         41
                                                                    ip4                         4737
tap10                             15     up          1450/0/0/0     rx packets                     2
                                                                    rx bytes                     168
                                                                    drops                          2
                                                                    ip6                            2
tap1                              10     up          1450/0/0/0     rx packets                 21775
                                                                    rx bytes                 1705039
                                                                    tx packets                 21094
                                                                    tx bytes                 2662522
                                                                    drops                         15
                                                                    ip4                        21766
                                                                    ip6                            9
tap2                              7      up          1450/0/0/0     
tap4                              16     up          1450/0/0/0     rx packets                 21797
                                                                    rx bytes                 1706648
                                                                    tx packets                 21103
                                                                    tx bytes                 2663711
                                                                    drops                         10
                                                                    ip4                        21789
                                                                    ip6                            8
tap5                              18     up          1450/0/0/0     rx packets                 17026
                                                                    rx bytes                 1977051
                                                                    tx packets                 17009
                                                                    tx bytes                 7044630
                                                                    drops                          9
                                                                    ip4                        17018
                                                                    ip6                            8
tap6                              12     up          1450/0/0/0     rx packets                  8566
                                                                    rx bytes                 1200399
                                                                    tx packets                  8566
                                                                    tx bytes                 5504642
                                                                    drops                          8
                                                                    ip4                         8558
                                                                    ip6                            8
tap7                              11     up          1450/0/0/0     rx packets                  1114
                                                                    rx bytes                  178300
                                                                    tx packets                   999
                                                                    tx bytes                  276126
                                                                    drops                          8
                                                                    ip4                         1106
                                                                    ip6                            8
tap8                              9      up          1450/0/0/0     rx packets                321646
                                                                    rx bytes                35148494
                                                                    tx packets                301017
                                                                    tx bytes               248216559
                                                                    drops                          8
                                                                    ip4                       321638
                                                                    ip6                            8
tap9                              17     up          1450/0/0/0     rx packets               1709761
                                                                    rx bytes               174138989
                                                                    tx packets               1479153
                                                                    tx bytes              1101851717
                                                                    drops                          8
                                                                    ip4                      1709753
                                                                    ip6                            8
vxlan_tunnel0                     4      up           0/0/0/0       rx packets               1844511
                                                                    rx bytes              1367922021
                                                                    tx packets               2096049
                                                                    tx bytes               291009297

      show tap:

      pp# show tap       
Interface: tap0 (ifindex 3)
  name ""
  fd 20
  tap-fd 21
  gso-enabled 1
  Mac Address: 02:fe:2c:47:d0:bf
  Device instance: 0
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 7653, desc_next 384, desc_in_use 923
    avail.flags 0x1 avail.idx 8576 used.flags 0x1 used.idx 7653
    kickfd 23, callfd 22
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 7320, desc_next 153, desc_in_use 1
    avail.flags 0x1 avail.idx 7321 used.flags 0x0 used.idx 7321
    kickfd 25, callfd 24
Interface: tap2 (ifindex 7)
  name "tap-2930546482"
  fd 32
  tap-fd 33
  gso-enabled 1
  Mac Address: 02:fe:78:f7:21:3f
  Device instance: 2
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 8, desc_next 960, desc_in_use 952
    avail.flags 0x1 avail.idx 960 used.flags 0x1 used.idx 8
    kickfd 35, callfd 34
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 0, desc_next 0, desc_in_use 0
    avail.flags 0x1 avail.idx 0 used.flags 0x0 used.idx 0
    kickfd 37, callfd 36
Interface: tap8 (ifindex 9)
  name "tap-4120826507"
  fd 69
  tap-fd 70
  gso-enabled 1
  Mac Address: 02:fe:05:6d:2c:41
  Device instance: 4
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 60038, desc_next 576, desc_in_use 954
    avail.flags 0x1 avail.idx 60992 used.flags 0x1 used.idx 60038
    kickfd 72, callfd 71
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 39375, desc_next 468, desc_in_use 5
    avail.flags 0x1 avail.idx 39380 used.flags 0x0 used.idx 39380
    kickfd 74, callfd 73
Interface: tap1 (ifindex 10)
  name "tap-4213645063"
  fd 27
  tap-fd 28
  gso-enabled 1
  Mac Address: 02:fe:1d:3d:66:85
  Device instance: 5
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 21818, desc_next 192, desc_in_use 902
    avail.flags 0x1 avail.idx 22720 used.flags 0x1 used.idx 21818
    kickfd 30, callfd 29
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 21134, desc_next 655, desc_in_use 1
    avail.flags 0x1 avail.idx 21135 used.flags 0x0 used.idx 21135
    kickfd 38, callfd 31
Interface: tap7 (ifindex 11)
  name "tap-3100233382"
  fd 63
  tap-fd 64
  gso-enabled 1
  Mac Address: 02:fe:22:9c:1d:16
  Device instance: 6
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 1166, desc_next 64, desc_in_use 946
    avail.flags 0x1 avail.idx 2112 used.flags 0x1 used.idx 1166
    kickfd 66, callfd 65
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 1044, desc_next 23, desc_in_use 3
    avail.flags 0x1 avail.idx 1047 used.flags 0x0 used.idx 1047
    kickfd 68, callfd 67
Interface: tap6 (ifindex 12)
  name "tap-4022792266"
  fd 57
  tap-fd 58
  gso-enabled 1
  Mac Address: 02:fe:9f:1e:97:5a
  Device instance: 7
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 8566, desc_next 256, desc_in_use 906
    avail.flags 0x1 avail.idx 9472 used.flags 0x1 used.idx 8566
    kickfd 60, callfd 59
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 8565, desc_next 374, desc_in_use 1
    avail.flags 0x1 avail.idx 8566 used.flags 0x0 used.idx 8566
    kickfd 62, callfd 61
Interface: tap10 (ifindex 15)
  name "tap-1144762374"
  fd 81
  tap-fd 82
  gso-enabled 1
  Mac Address: 02:fe:d8:76:95:a1
  Device instance: 10
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 2, desc_next 960, desc_in_use 958
    avail.flags 0x1 avail.idx 960 used.flags 0x1 used.idx 2
    kickfd 84, callfd 83
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 0, desc_next 0, desc_in_use 0
    avail.flags 0x1 avail.idx 0 used.flags 0x0 used.idx 0
    kickfd 86, callfd 85
Interface: tap4 (ifindex 16)
  name "tap-1528684717"
  fd 45
  tap-fd 46
  gso-enabled 1
  Mac Address: 02:fe:e4:45:cb:05
  Device instance: 11
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 21838, desc_next 256, desc_in_use 946
    avail.flags 0x1 avail.idx 22784 used.flags 0x1 used.idx 21838
    kickfd 48, callfd 47
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 21143, desc_next 664, desc_in_use 1
    avail.flags 0x1 avail.idx 21144 used.flags 0x0 used.idx 21144
    kickfd 50, callfd 49
Interface: tap9 (ifindex 17)
  name "tap-1436846080"
  fd 75
  tap-fd 76
  gso-enabled 1
  Mac Address: 02:fe:b0:3b:8c:fa
  Device instance: 12
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 8545, desc_next 256, desc_in_use 927
    avail.flags 0x1 avail.idx 9472 used.flags 0x1 used.idx 8545
    kickfd 78, callfd 77
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 39687, desc_next 791, desc_in_use 16
    avail.flags 0x1 avail.idx 39703 used.flags 0x0 used.idx 39703
    kickfd 80, callfd 79
Interface: tap5 (ifindex 18)
  name "tap-294713375"
  fd 51
  tap-fd 52
  gso-enabled 1
  Mac Address: 02:fe:9b:59:f4:49
  Device instance: 13
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x13d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 17035, desc_next 576, desc_in_use 949
    avail.flags 0x1 avail.idx 17984 used.flags 0x1 used.idx 17035
    kickfd 54, callfd 53
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 17017, desc_next 634, desc_in_use 1
    avail.flags 0x1 avail.idx 17018 used.flags 0x0 used.idx 17018
    kickfd 56, callfd 55

        1. istio2.pcap
          70 kB
        2. packet-trace.zip
          7.56 MB
        3. packet-trace2.zip
          1.27 MB

            ayourtch Andrew Yourtchenko
            raszabo Rastislav Szabo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 days
                3d