This is an old issue, but in CSIT it usually manifests as a progression (both in MRR and PDR), so we were not reporting it.
We need to increase MTU for tests with jumbo frames, and over time we have refactored CSIT code to set MTU to 9200 bytes on physical interfaces regardless of frame size used in the test.
I believe the first progression (caused by setting MTU=9200 on tests which previously did not explicitly set MTU) was present already in report for rls1807. See [0], although it is not clear which parts are due to MTU and which parts are to genuine improvements in VPP code.

In rls2001, another CSIT refactor caused progressions in (at least) two tests with 64 byte frames. See [1], it is the cause number 9 (CSIT-1707) and 5 (CSIT-1705).

I was able to confirm that setting MTU to 2022 bytes has worse performance [2], and setting it to 2024 bytes has better performance [3] (run for 2023 is waiting for a free testbed, which can take a while as we started dry-runs for rls2005).
Performance of AVF tests is not affected by MTU value.

I believe this is a bug (maybe in DPDK). I would understand if higher MTU lead to worse performance (as structures able to handle jumbo packets are used), but I see no good reason for the performance to decrease when MTU is set to a smaller value.
Especially considering the default configuration (without explicit hw_interface_set_mtu call) leads to the worse performance.

[0] https://docs.fd.io/csit/rls1807/report/_static/vpp/performance-changes-1t1c-mrr.txt
[1] https://docs.fd.io/csit/rls2001/report/_static/vpp/performance-rca-2n-skx-2t1c-pdr.txt
[2] https://logs.fd.io/production/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-2n-skx/1270/archives/log.html.gz#s1-s1-s1-s1-s2-t1
[3] https://logs.fd.io/production/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-2n-skx/1267/archives/log.html.gz#s1-s1-s1-s1-s2-t1