Medium This issue may be "exotic" and hard to describe or reproduce, but I'll try my best.
I use vpp and vppctl are used in tests which I run in Jenkins CI. The tests are a set of scripts written in bash. The code for the test can be found in here: https://github.com/spdk/spdk/blob/master/test/iscsi_tgt/common.sh, look for "start_vpp" function.
For a reason unknown when these tests are run via CI (Jenkins with "Execute shell" step) vppctl commands execute properly, but give 141 (pipefail) return code. Example:
Command is: vppctl show version echo $? 11:43:53 # vppctl show version vpp v19.04.2-release built by sys_sgci on GrizzlyPass14 at Mon 09 Mar 2020 03:45:41 PM CET 11:43:53 # echo 141 141
This only happens when the test is executed by Jenkins CI and the system executing the test is Fedora 30. Jenkins & test system specs:
Jenkins ver. 2.176.3
Jenkins SSH Slaves Plugin 1.30.4
System: Fedora 30 5.5.6-100.fc30.x86_64
openssh-server.x86_64 8.0p1-5.fc30
Jenkins output when connecting the test node to the CI:
[03/12/20 07:29:36] [SSH] The remote user's environment is: BASH=/usr/bin/bash BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASHRCSOURCED=Y BASH_ALIASES=() BASH_ARGC=([0]="0") BASH_ARGV=() BASH_CMDS=() BASH_EXECUTION_STRING=set BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]="5" [1]="0" [2]="11" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu") BASH_VERSION='5.0.11(1)-release' DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus DIRSTACK=() EUID=1000 GROUPS=() HOME=/home/sys_sgci HOSTNAME=GP14 HOSTTYPE=x86_64 IFS=$' \t\n' LANG=en_US.UTF-8 LESSOPEN='||/usr/bin/lesspipe.sh %s' LOGNAME=sys_sgci MACHTYPE=x86_64-redhat-linux-gnu OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/home/sys_sgci/.local/bin:/home/sys_sgci/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin PIPESTATUS=([0]="0") PPID=2318 PS4='+ ' PWD=/home/sys_sgci QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include QTLIB=/usr/lib64/qt-3.3/lib SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 SSH_CLIENT='X.Y.Z.111 38434 22' SSH_CONNECTION='A.B.C.111 38434 A.B.C.94 22' TERM=dumb UID=1000 USER=sys_sgci XDG_RUNTIME_DIR=/run/user/1000 XDG_SESSION_CLASS=user XDG_SESSION_ID=5 XDG_SESSION_TYPE=tty _=PATH ftp_proxy=http://proxy-addr.com git_proxy=http://proxy-addr.com http_proxy=http://proxy-addr.com https_proxy=http://proxy-addr.com gawklibpath_append () { [ -z "$AWKLIBPATH" ] && AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`; export AWKLIBPATH="$AWKLIBPATH:$*" } gawklibpath_default () { unset AWKLIBPATH; export AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'` } gawklibpath_prepend () { [ -z "$AWKLIBPATH" ] && AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`; export AWKLIBPATH="$*:$AWKLIBPATH" } gawkpath_append () { [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`; export AWKPATH="$AWKPATH:$*" } gawkpath_default () { unset AWKPATH; export AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'` } gawkpath_prepend () { [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`; export AWKPATH="$*:$AWKPATH" } [03/12/20 07:29:36] [SSH] Checking java version of /var/jenkins/jdk/bin/java Couldn't figure out the Java version of /var/jenkins/jdk/bin/java bash: /var/jenkins/jdk/bin/java: No such file or directory[03/12/20 07:29:36] [SSH] Checking java version of java [03/12/20 07:29:37] [SSH] java -version returned 1.8.0_242. [03/12/20 07:29:37] [SSH] Starting sftp client. [03/12/20 07:29:37] [SSH] Copying latest remoting.jar... Source agent hash is D2D1A740134BD20D6F0855B356344342. Installed agent hash is D2D1A740134BD20D6F0855B356344342 Verified agent jar. No update is necessary. Expanded the channel window size to 4MB [03/12/20 07:29:37] [SSH] Starting agent process: cd "/var/jenkins" && java -jar remoting.jar -workDir /var/jenkins -jar-cache /var/jenkins/remoting/jarCache Mar 12, 2020 10:35:13 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/jenkins/remoting as a remoting work directory Both error and output logs will be printed to /var/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 3.29 This is a Unix agent Evacuated stdout Agent successfully connected and online
This issue does not reproduce if the same test is run manually on the test server (SSH/CLI).
It also does not reproduce if the test is run via CI and target OS is dirrerent, eg. Fedora 29.
