Uploaded image for project: 'csit'
  1. csit
  2. CSIT-1729 Root Cause Analysis for 2005
  3. CSIT-1743

Investigate which high-stdev results are caused by testbeds


    • Icon: Sub-task Sub-task
    • Resolution: Unresolved
    • Icon: Low Low
    • None
    • None
    • None

      Some testbed types are known to have small differences between testbeds of the same type.
      The typical example is visible in trending [0] (NDR graph for avf-ethip4-ip4base), showing TB31 having consistently better results than TB32.

      This can affect also some PDR results, and if the performance gap is smaller (relative to stdev withing the same testbed), visual examination will not be so conclusive.
      This affects report comparison data. On one hand, if testbed distribution is balanced, high stdev can hide real performance differences (caused by CSIT or VPP code). On the other hand, if testbed distribution is not balanced, the comparison table can show relatively big performance with relatively small stdev, even if there was no real change in CSIT nor VPP code.

      In 2005 report we will probably not investigate tests this deeply, but we can do that later when trying to improve comparison methodology.

      [0] https://docs.fd.io/csit/master/trending/ndrpdr_trending/ip4-3n-skx-xxv710.html#b-ip4routing-base-avf-dpdk

            vrpolak Vratko Polak
            vrpolak Vratko Polak
            0 Vote for this issue
            1 Start watching this issue