Catch all errors when ESMF build fails (various ways)

[theurich]

The current test script implementation does correctly catch and report libesmf build errors. However, while working on the trace preload library issues under Darwin, there was a temporary issue introduced on the ESMF side that resulted in failing of build target build_tracelibs. When that happened, the test scripts would abort, instead of reporting the ESMF build failure as they should. The backtrace from one of the failed runs was captured:

+ tee /Users/oehmke/ESMF_AutoTest/gfortranclang_12.2.0_14.0.0_mpich_O_develop/build.log

Traceback (most recent call last):

  File "/Users/oehmke/ESMF_AutoTest/./esmf-test-scripts/python_scripts/test_esmf.py", line 359, in <module>

    go(args)

  File "/Users/oehmke/ESMF_AutoTest/./esmf-test-scripts/python_scripts/test_esmf.py", line 302, in go

    test.start()

  File "/Users/oehmke/ESMF_AutoTest/./esmf-test-scripts/python_scripts/test_esmf.py", line 271, in start

    _submit_case(_c)  # may bock

  File "/Users/oehmke/ESMF_AutoTest/./esmf-test-scripts/python_scripts/test_esmf.py", line 199, in _submit_case

    _case.submit(no_artifacts=self.no_artifacts)

  File "/Users/oehmke/ESMF_AutoTest/esmf-test-scripts/python_scripts/case.py", line 109, in submit

    self.build_job_num = self.machine.scheduler.submit_job(script_file=self.build_script)

  File "/Users/oehmke/ESMF_AutoTest/esmf-test-scripts/python_scripts/noscheduler.py", line 18, in submit_job

    cmd.runcmd(f"{script_file}")

  File "/Users/oehmke/ESMF_AutoTest/esmf-test-scripts/python_scripts/cmd.py", line 20, in runcmd

    raise cpe

  File "/Users/oehmke/ESMF_AutoTest/esmf-test-scripts/python_scripts/cmd.py", line 15, in runcmd

    out = subprocess.check_output(cmd, shell=True, stderr=_stderr).strip().decode("utf-8")

  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output

    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run

    raise CalledProcessError(retcode, process.args,

subprocess.CalledProcessError: Command '/Users/oehmke/ESMF_AutoTest/gfortranclang_12.2.0_14.0.0_mpich_O_develop/build.bat' returned non-zero exit 

status 2.

Instead of this abort, all ESMF build errors are expected to be caught by the test system and pushed correctly to the artifacts repo.