Loading...

Type: Bug
Resolution: Duplicate
Priority: High
Fix Version/s: 18.07
Affects Version/s: None
Component/s: S-NAT
Labels:
None

Epic Link:
Contiv VPP (Kubernetes)

In the Contiv's single-NIC setup, DNS pod cannot access kubernetes service. The reason is that while kubernetes service static mapping is used to NAT the SYN packet, SYN-ACK gets NATed based on the identity mapping for node IP instead.

Configuration:

vpp# sh nat44 static mappings 
NAT44 static mappings:
tcp local 10.20.0.3:6443 external 10.96.0.1:443 vrf 0 self-twice-nat out2in-only   // <- Kubernetes service
udp local 10.20.0.3:4789 external 10.20.0.3:4789 vrf 0  
local 10.20.0.3 external 10.20.0.3 vrf 0   // <- identity mapping for node IP
tcp local 10.20.0.3:12379 external 10.20.0.3:32379 vrf 0 self-twice-nat out2in-only
tcp local 10.20.0.3:12379 external 10.98.208.57:12379 vrf 0 self-twice-nat out2in-only
tcp local 10.20.0.3:12379 external 10.20.0.4:32379 vrf 0 self-twice-nat out2in-only

Example of identity mapping in action (valid use) – shows that it has entry 0:

Packet 40:
00:03:54:767852: virtio-input
  virtio: hw_if_index 2 next-index 4 vring 0 len 66
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:03:54:767858: ethernet-input
  IP4: 00:00:00:00:00:02 -> 01:23:45:67:89:42
00:03:54:767862: ip4-input
  TCP: 10.20.0.3 -> 10.20.0.4
    tos 0x00, ttl 64, length 52, checksum 0x7b62
    fragment id 0xab33, flags DONT_FRAGMENT
  TCP: 6443 -> 42048
    seq. 0x761d8e33 ack 0x307439e1
    flags 0x10 ACK, tcp header: 32 bytes
    window 1452, checksum 0x0fdb
00:03:54:767867: nat44-in2out
  NAT44_IN2OUT_FAST_PATH: sw_if_index 2, next index 0, session 0
00:03:54:767870: ip4-lookup
  fib 0 dpo-idx 5 flow hash: 0x00000000
  TCP: 10.20.0.3 -> 10.20.0.4
    tos 0x00, ttl 64, length 52, checksum 0x7b62
    fragment id 0xab33, flags DONT_FRAGMENT
  TCP: 6443 -> 42048
    seq. 0x761d8e33 ack 0x307439e1
    flags 0x10 ACK, tcp header: 32 bytes
    window 1452, checksum 0x0fdb
00:03:54:767873: ip4-rewrite
  tx_sw_if_index 1 dpo-idx 5 : ipv4 via 10.20.0.4 GigabitEthernet0/8/0: 080027052543080027d3a1c90800 flow hash: 0x00000000
  00000000: 080027052543080027d3a1c9080045000034ab3340003f067c620a1400030a14
  00000020: 0004192ba440761d8e33307439e1801005ac0fdb00000101080a0000
00:03:54:767875: nat44-in2out-output
  NAT44_IN2OUT_FAST_PATH: sw_if_index 2, next index 0, session 0
00:03:54:767878: GigabitEthernet0/8/0-output
  GigabitEthernet0/8/0
  IP4: 08:00:27:d3:a1:c9 -> 08:00:27:05:25:43
  TCP: 10.20.0.3 -> 10.20.0.4
    tos 0x00, ttl 63, length 52, checksum 0x7c62
    fragment id 0xab33, flags DONT_FRAGMENT
 TCP: 6443 -> 42048
    seq. 0x761d8e33 ack 0x307439e1
    flags 0x10 ACK, tcp header: 32 bytes
    window 1452, checksum 0x0fdb
00:03:54:767879: GigabitEthernet0/8/0-tx
  GigabitEthernet0/8/0 tx queue 0
  buffer 0x14221: current data 0, length 66, free-list 0, clone-count 0, totlen-nifb 0, trace 0x3
                  nated l2-hdr-offset 0 l3-hdr-offset 14 
  PKT MBUF: port 65535, nb_segs 1, pkt_len 66
    buf_len 2176, data_len 66, ol_flags 0x0, data_off 128, phys_addr 0x659088c0
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
  IP4: 08:00:27:d3:a1:c9 -> 08:00:27:05:25:43
  TCP: 10.20.0.3 -> 10.20.0.4
    tos 0x00, ttl 63, length 52, checksum 0x7c62
    fragment id 0xab33, flags DONT_FRAGMENT
  TCP: 6443 -> 42048
    seq. 0x761d8e33 ack 0x307439e1
    flags 0x10 ACK, tcp header: 32 bytes
    window 1452, checksum 0x0fdb

And then the mismatch between SYN and SYN-ACK for connection targeting kubernetes service (10.96.0.1:443) - including the RST sent by the client as a consequence:

SYN:

00:06:04:234349: virtio-input
  virtio: hw_if_index 4 next-index 4 vring 0 len 74
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:06:04:234355: ethernet-input
  IP4: 00:00:00:00:00:02 -> 02:fe:fc:07:21:82
00:06:04:234359: ip4-input
  TCP: 10.1.1.2 -> 10.96.0.1
    tos 0x00, ttl 64, length 60, checksum 0x550f
    fragment id 0xd049, flags DONT_FRAGMENT
  TCP: 39394 -> 443
    seq. 0x4aceb6ec ack 0x00000000
    flags 0x02 SYN, tcp header: 40 bytes
    window 29200, checksum 0xadda
00:06:04:234362: nat44-out2in
  NAT44_OUT2IN: sw_if_index 4, next index 1, session index 86
00:06:04:234368: ip4-lookup
  fib 0 dpo-idx 6 flow hash: 0x00000000
  TCP: 10.1.1.2 -> 10.20.0.3
    tos 0x00, ttl 64, length 60, checksum 0x5559
    fragment id 0xd049, flags DONT_FRAGMENT
  TCP: 39394 -> 6443
    seq. 0x4aceb6ec ack 0x00000000
    flags 0x02 SYN, tcp header: 40 bytes
    window 29200, checksum 0x96b4
00:06:04:234370: ip4-local
    TCP: 10.1.1.2 -> 10.20.0.3
      tos 0x00, ttl 64, length 60, checksum 0x5559
      fragment id 0xd049, flags DONT_FRAGMENT
    TCP: 39394 -> 6443
      seq. 0x4aceb6ec ack 0x00000000
      flags 0x02 SYN, tcp header: 40 bytes
      window 29200, checksum 0x96b4
00:06:04:234373: ip4-punt
    TCP: 10.1.1.2 -> 10.20.0.3
      tos 0x00, ttl 64, length 60, checksum 0x5559
      fragment id 0xd049, flags DONT_FRAGMENT
    TCP: 39394 -> 6443
      seq. 0x4aceb6ec ack 0x00000000
      flags 0x02 SYN, tcp header: 40 bytes
      window 29200, checksum 0x96b4
00:06:04:234374: stn-ip4-punt
  dst_address: 10.20.0.3
  rule:
    rule_index: 0
    address: 10.20.0.3
    iface: tap0 (2)
    next_node: tap0-output (411)
00:06:04:234376: tap0-output
  tap0
  IP4: 00:00:00:00:00:01 -> 00:00:00:00:00:02
  TCP: 10.1.1.2 -> 10.20.0.3
    tos 0x00, ttl 64, length 60, checksum 0x5559
    fragment id 0xd049, flags DONT_FRAGMENT
  TCP: 39394 -> 6443
    seq. 0x4aceb6ec ack 0x00000000
    flags 0x02 SYN, tcp header: 40 bytes
    window 29200, checksum 0x96b4

SYN-ACK:

00:06:04:234421: virtio-input
  virtio: hw_if_index 2 next-index 4 vring 0 len 74
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:06:04:234423: ethernet-input
  IP4: 00:00:00:00:00:02 -> 01:23:45:67:89:42
00:06:04:234426: ip4-input
  TCP: 10.20.0.3 -> 10.1.1.2
    tos 0x00, ttl 64, length 60, checksum 0x25a3
    fragment id 0x0000, flags DONT_FRAGMENT
  TCP: 6443 -> 39394
    seq. 0xef37db1d ack 0x4aceb6ed
    flags 0x12 SYN ACK, tcp header: 40 bytes
    window 28960, checksum 0x57e4
00:06:04:234431: nat44-in2out
  NAT44_IN2OUT_FAST_PATH: sw_if_index 2, next index 0, session 0
00:06:04:234433: ip4-lookup
  fib 0 dpo-idx 4 flow hash: 0x00000000
  TCP: 10.20.0.3 -> 10.1.1.2
    tos 0x00, ttl 64, length 60, checksum 0x25a3
    fragment id 0x0000, flags DONT_FRAGMENT
  TCP: 6443 -> 39394
    seq. 0xef37db1d ack 0x4aceb6ed
    flags 0x12 SYN ACK, tcp header: 40 bytes
    window 28960, checksum 0x57e4
00:06:04:234436: ip4-rewrite
  tx_sw_if_index 4 dpo-idx 4 : ipv4 via 10.1.1.2 tap1: 00000000000202fefc0721820800 flow hash: 0x00000000
  00000000: 00000000000202fefc07218208004500003c000040003f0626a30a1400030a01
  00000020: 0102192b99e2ef37db1d4aceb6eda012712057e40000020405b40402
00:06:04:234438: tap1-output
  tap1
  IP4: 02:fe:fc:07:21:82 -> 00:00:00:00:00:02
  TCP: 10.20.0.3 -> 10.1.1.2
    tos 0x00, ttl 63, length 60, checksum 0x26a3
    fragment id 0x0000, flags DONT_FRAGMENT
  TCP: 6443 -> 39394
    seq. 0xef37db1d ack 0x4aceb6ed
    flags 0x12 SYN ACK, tcp header: 40 bytes
    window 28960, checksum 0x57e4

RST:

00:06:04:234544: virtio-input
  virtio: hw_if_index 4 next-index 4 vring 0 len 54
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:06:04:234545: ethernet-input
  IP4: 00:00:00:00:00:02 -> 02:fe:fc:07:21:82
00:06:04:234546: ip4-input
  TCP: 10.1.1.2 -> 10.20.0.3
    tos 0x00, ttl 64, length 40, checksum 0x2b85
    fragment id 0xfa31, flags DONT_FRAGMENT
  TCP: 39394 -> 6443
    seq. 0x4aceb6ed ack 0x00000000
    flags 0x04 RST, tcp header: 20 bytes
    window 0, checksum 0xe5fd
00:06:04:234548: nat44-out2in
  NAT44_OUT2IN: sw_if_index 4, next index 1, session index 0
00:06:04:234549: ip4-lookup
  fib 0 dpo-idx 6 flow hash: 0x00000000
  TCP: 10.1.1.2 -> 10.20.0.3
    tos 0x00, ttl 64, length 40, checksum 0x2b85
    fragment id 0xfa31, flags DONT_FRAGMENT
  TCP: 39394 -> 6443
    seq. 0x4aceb6ed ack 0x00000000
    flags 0x04 RST, tcp header: 20 bytes
    window 0, checksum 0xe5fd
00:06:04:234550: ip4-local
    TCP: 10.1.1.2 -> 10.20.0.3
      tos 0x00, ttl 64, length 40, checksum 0x2b85
      fragment id 0xfa31, flags DONT_FRAGMENT
    TCP: 39394 -> 6443
      seq. 0x4aceb6ed ack 0x00000000
      flags 0x04 RST, tcp header: 20 bytes
      window 0, checksum 0xe5fd
00:06:04:234552: ip4-punt
    TCP: 10.1.1.2 -> 10.20.0.3
      tos 0x00, ttl 64, length 40, checksum 0x2b85
      fragment id 0xfa31, flags DONT_FRAGMENT
    TCP: 39394 -> 6443
      seq. 0x4aceb6ed ack 0x00000000
      flags 0x04 RST, tcp header: 20 bytes
      window 0, checksum 0xe5fd
00:06:04:234553: stn-ip4-punt
  dst_address: 10.20.0.3
  rule:
    rule_index: 0
    address: 10.20.0.3
    iface: tap0 (2)
    next_node: tap0-output (411)
00:06:04:234553: tap0-output
  tap0
  IP4: 00:00:00:00:00:01 -> 00:00:00:00:00:02
  TCP: 10.1.1.2 -> 10.20.0.3
    tos 0x00, ttl 64, length 40, checksum 0x2b85
    fragment id 0xfa31, flags DONT_FRAGMENT
  TCP: 39394 -> 6443
    seq. 0x4aceb6ed ack 0x00000000
    flags 0x04 RST, tcp header: 20 bytes
    window 0, checksum 0xe5fd

is blocked by

VPP-1273 Implement endpoint dependent NAT filtering and mapping

Closed

relates to

VPP-1273 Implement endpoint dependent NAT filtering and mapping

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates