Here is the test bed topology:

 All captures provided here are done on NSE's VPP, this is where all SRv6 processing happens and the issue is observed.

client 1.0.0.1 ------------ NSM's VPP memif ------ memif 1.0.02 NSE's VPP — af-packet 2001:470:b16e:59::9 ------- 2001:470:b16e:59::5- xrv9k

we are implementing srv6 l3 service. Client's ipv4 packet gets encapsulated  into ipv6 packet by NSE's VPP and sent toward xrv9k attached to the core. The return from the remote PE vpn packet is supposed to hit NSE'sVPP, since it is SRv6 DT4 endpoint, VPP is supposed to strip outer header and make a lookup in a vrf table associated with the client. All forwarding entries in vrf table looks good.

What is observed is the return packet hit NSE's VPP but then sent back to xrv9k, instead of decapsulating it and doing a second lookup. This behaviour is confirmed by tcpdump packet capture on af-packet xrv9k facing interface.

12:39:36.036801 52:54:00:0e:9c:47 > d2:74:c1:54:ae:b6, ethertype IPv6 (0x86dd), length 138: (flowlabel 0x844c7, hlim 113, next-header IPIP (4) payload length: 84) 2001:5::4 > 2001:0:5:9:41::: (tos 0x0, ttl 254, id 1012, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.5.2 > 1.0.0.1: ICMP echo reply, id 38, seq 1, length 64
12:39:36.037676 d2:74:c1:54:ae:b6 > 52:54:00:0e:9c:47, ethertype IPv6 (0x86dd), length 138: (flowlabel 0x844c7, hlim 112, next-header IPIP (4) payload length: 84) 2001:5::4 > 2001:0:5:9:41::: (tos 0x0, ttl 254, id 1012, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.5.2 > 1.0.0.1: ICMP echo reply, id 38, seq 1, length 64
12:39:36.037919 52:54:00:0e:9c:47 > d2:74:c1:54:ae:b6, ethertype IPv6 (0x86dd), length 138: (flowlabel 0x844c7, hlim 111, next-header IPIP (4) payload length: 84) 2001:5::4 > 2001:0:5:9:41::: (tos 0x0, ttl 254, id 1012, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.5.2 > 1.0.0.1: ICMP echo reply, id 38, seq 1, length 64
12:39:36.038781 d2:74:c1:54:ae:b6 > 52:54:00:0e:9c:47, ethertype IPv6 (0x86dd), length 138: (flowlabel 0x844c7, hlim 110, next-header IPIP (4) payload length: 84) 2001:5::4 > 2001:0:5:9:41::: (tos 0x0, ttl 254, id 1012, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.5.2 > 1.0.0.1: ICMP echo reply, id 38, seq 1, length 64
12:39:36.038986 52:54:00:0e:9c:47 > d2:74:c1:54:ae:b6, ethertype IPv6 (0x86dd), length 138: (flowlabel 0x844c7, hlim 109, next-header IPIP (4) payload length: 84) 2001:5::4 > 2001:0:5:9:41::: (tos 0x0, ttl 254, id 1012, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.5.2 > 1.0.0.1: ICMP echo reply, id 38, seq 1, length 64

As you can see ipv6 source and destination do not change, but source and destination Mac addresses are alternating (ping-pong).

See more complete output in the attachment, but NSE's VPP carries in ip6 fib an entry for default:

::/0
unicast-ip6-chain
[@0]: dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:57 to:[727:90148]]
[0] [@5]: ipv6 via 2001:470:b16e:59::5 host-tor_vlan: mtu:9000 5254000e9c47d274c154aeb686dd

and for more specific entry:

2001:0:5:9:41::/128
unicast-ip6-chain
[@0]: dpo-load-balance: [proto:ip6 index:29 buckets:1 uRPF:29 to:[0:0]]
[0] [@19]: SR: localsid_index:[0]

IPv6 packets destined to 2001:0:5:9:41:: instead of taking 2001:0:5:9:41::/128 taking ::/0 going back to next hop router, and play ping-pong with it until ttl expires.

 Here is the output of local sid I suspect associated with the forwarding entry above:

vpp# sh sr localsids
SRv6 - My LocalSID Table:
=========================
Address: 2001:0:5:9:41::
Behavior: DT4 (Endpoint with decapsulation and specific IPv4 table lookup)
Table: 1
Good traffic: [0 packets : 0 bytes]
Bad traffic: [0 packets : 0 bytes]
--------------------

The issue is 100% reproducible, I can provide any additional outputs.

Please see attached traces.txt for complete output of ipv4 and ipv6 fib and packet trace from af-packet-input.