The regression itself is not a big deal, but it can hint at an interesting behavior that may be affecting other processing paths.

Why not a big deal. L2patch is generally not recommended for any production use (use l2 cross connect instead). L2patch is interesting for testing, because it is the processing path that should show the best performance.
Also, the regression is visible only in specific circumstances. Out of different hardware platforms, the regression is visible mainly on Cascade Lake (2n-clx jobs). The regression is only visible with vfio-pci driver, maybe because AVF driver is faster so VPP is no longer the bottleneck.
The regression is around -3% both in MRR and in PDR. Standard deviation is quite high.
Skylake performance difference on skylake is 0.0%, other platforms usually show a regression between 1-2%.

The cause of the regression has been identified as this [0] change, more specifically it would be the cause if a fix [1] (to a breakage [2]) was merged before [0].

Why it is interesting. [0] should directly affect only code paths that use packet tracing (the performance tests doe not use it). [0] reduces code size, so if anything it should speed up performance. It does the opposite, so it hints something interesting is happening, perhaps related to instruction cache.

Comments from private e-mail thread started by my Gerrit comments on [0].

Benoit Ganne:
This is kind of surprising to me, as it should only impact performance when tracing is on (ie with 'trace add ...').
But maybe we are hitting one of those "code generation/placement has changed, impacting L1i-cache utilization" cases?...
Do you save any Linux perf outputs?

Dave Barach:
It's not plausible that shrinking unreachable code would cause a performance regression, EXCEPT insofar as it changes code placement within the cache hierarchy. An A-B comparison of i-cache evictions with/without the change in question would be quite interesting. Can you acquire those data?

[0] https://gerrit.fd.io/r/c/vpp/+/27467
[1] https://gerrit.fd.io/r/c/vpp/+/27807
[2] https://gerrit.fd.io/r/c/vpp/+/27413