Uploaded image for project: 'csit'
  1. csit
  2. CSIT-1900

temporary failures on 2n-icx (TB214)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Low Low
    • close old issues
    • None
    • None

      This is different from CSIT-1899 as the breakage did not happen around midnight, and the testbed did not get stuck in reserved state.
      But this is similar in how multiple runs of different jobs were affected.
      Low priority as the testbed looks fixed now, opening in case something similar happens again in future.

      Timeline:

      iter-27: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2302-2n-icx/27/console-timestamp.log.gz
      reserved: 2023-02-09 20:03:21
      unreserved: 2023-02-10 10:01:29
      notes:
      + last normal run before issues started
      + af_xdp tests on Intel-E810CQ forward zero packets
      ++ except very first 1c ip4base which passes

      iter-40: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2302-2n-icx/40/console-timestamp.log.gz
      reserved: 2023-02-10 10:03:57
      unreserved: 2023-02-11 10:15:23
      notes:
      + very first 1c ip4base test on Intel-E810CQ still passes
      + first deviation is af_xdp on Intel-E810CQ not dropping all packets
      ++ already for 2c ip4base at 2023-02-10 13:29:25
      + all subsequent tests on Intel-E810CQ fail with varying symptoms

      mrr-weekly: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-weekly-master-2n-icx/59/console-timestamp.log.gz
      reserved: 2023-02-11 10:17:29
      unreserved: 2023-02-11 10:33:54
      notes: top suite setup failed as Intel-E810CQ interfaces were up in Linux so VPP could not take them

      dpdk-weekly: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-dpdk-perf-mrr-weekly-master-2n-icx/57/console-timestamp.log.gz
      reserved: 2023-02-11 10:34:15
      unreserved: 2023-02-11 10:55:46
      notes: normal run as it does not interact with Intel-E810CQ

      ndrpdr-weekly: https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-2n-icx/60/console-timestamp.log.gz
      reserved: 2023-02-11 10:56:13
      unreserved: 2023-02-11 11:07:21
      notes: top suite setup failed: couldn't open log '/var/log/vpp/vpp.log'

      (not sure which run comes next, but the issue is currently fixed)

            Unassigned Unassigned
            vrpolak Vratko Polak
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: