Summary after multiple comments:

Some ipsec test cases use small number of flows, so we expect RSS giving unbalanced number of flows per VPP worker thread. CSIT does not perform any flow / RSS configuration, so we rely on driver default behavior.
For DPDK driver, the hashing is deterministic, so even if workers are not loaded evenly, the performance is stable. This ticket is about AVF driver not resulting in deterministic RSS, so worker load (and performance) varies between runs, even if everything on CSIT and VPP side remains the same.
This difference between DPDK and AVF behavior is not new, but it was not visible in single core tests before CSIT increased RX queue to one per worker and interface.
In ipsec tests, CSIT generates random keys, but the behavior has been reproduced also in simple L2 tests (reduced number of flows, 3-node testbed).
Possibly, CSIT should do some flow / RSS configuration to get deterministic behavior in AVF tests. Or, VPP should offer deterministic RSS by default, and this is a symptom of a VPP bug.
VPP expert is needed to decide which option is the case, and point to documentation if a fix on CSIT side is needed.

Original ticket name: Three-band structure in 3n-skx AVF ipsec tests
Original description:

Visible in trending [0], from start of 2021-06, mainly in avf-ethip4ipsec4tnlsw-ip4base-int-aes256gcm, with three groups of MRR results. The most frequent around 3.5 Mpps, then around 2.8 Mpps and finally around 2.1 Mpps.
1518B case also shows similar pattern, although the distinction between bands is not as clear. Only AVF tests show this pattern, and only ipsec tests, even for 1000+ tunnels the pattern is not there (1000 tunnel tests has MRR performance around 3.0 Mpps, so better than two bands of 4 tunnel test).

This is also affecting PDR trending [1] and rls2106 results [2] (although the middle band is the most frequent in these two), causing big standard deviation is comparison table [3].

[0] https://docs.fd.io/csit/master/trending/trending/ipsec-3n-skx-xxv710.html#t1c
[1] https://docs.fd.io/csit/master/trending/ndrpdr_trending/ipsec-3n-skx-xxv710.html#t1c-pdr
[2] https://docs.fd.io/csit/rls2106/report/vpp_performance_tests/packet_throughput_graphs/ipsec-3n-skx-xxv710.html
[3] https://docs.fd.io/csit/rls2106/report/_static/vpp/performance-changes-3n-skx-2t1c-pdr.txt