ESMF error in dev-MC_25km_jra_iaf+wombatlite

[aekiss]

dev-MC_25km_jra_iaf+wombatlite crashes on model date 1963-01-15

lots of PET*.ESMF_LogFile have errors such as this:

/home/156/aek156/payu/MC_25km_jra_iaf+wombatlite/work/PET0007.ESMF_LogFile

20260327 091958.467 ERROR            PET0007 ERROR: 135 nans found in So_u
20260327 091958.468 ERROR            PET0007 ERROR: 135 nans found in So_v
20260327 091958.468 ERROR            PET0007 med_methods_mod.F90:2475 ABORTING JOB, see PET file for details
20260327 091958.468 ERROR            PET0007 med_phases_prep_ice_mod.F90:154 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 MED:src/addon/NUOPC/src/NUOPC_ModelBase.F90:2104 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:3622 Failure  - Phase 'med_phases_prep_ice' Run for modelComp 1 did not return ESMF_SUCCESS
20260327 091958.468 ERROR            PET0007 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:3860 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:3536 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:3622 Failure  - Phase 'RunPhase1' Run for modelComp 1 did not return ESMF_SUCCESS
20260327 091958.468 ERROR            PET0007 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:3860 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:3536 Failure  - Passing error in return code
20260327 091958.468 ERROR            PET0007 esmApp.F90:147 Failure  - Passing error in return code
20260327 091958.468 INFO             PET0007  Finalizing ESMF with endflag==ESMF_END_ABORT

This is repeatable, but interestingly it runs 2 days longer with @manodeep's performance tweak NONBLOCKING_UPDATES=True. [edited]

run details:
https://github.com/ACCESS-Community-Hub/access-om3-experiments/tree/MC_25km_jra_iaf%2Bwombatlite-test3-07f45749
/home/156/aek156/payu/MC_25km_jra_iaf+wombatlite
/scratch/tm70/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test3-07f45749
/scratch/tm70/aek156/access-om3/work/MC_25km_jra_iaf+wombatlite-test3-07f45749

[anton-seaice]

That's strange, there also lots of errors printed from the atm_ocn flux calculation in the mediator, presumably for the same reason

e.g.:

                     NaN                     NaN  1.000000000000000E-002
           5
 No iterations performed in flux_atmocn_mod

So_u and So_v should be ocean currents calculated in MOM6:

https://github.com/ACCESS-NRI/MOM6/blob/c664721ebd58c033964b502e7fcdcccd05f02947/config_src/drivers/nuopc_cap/mom_cap_methods.F90#L728-L743

[aekiss]

Nothing looks amiss in work/access-om3.mom6.2d.speed.1day.mean._1963.nc.*

[anton-seaice]

I guess whatever strange this happened would have to have been in the last timestep the model ran, so it might need timestep diagnostics

[aekiss]

Output of surface speed at every timestep (900s) reveals nothing obviously wrong in the last timestep before the crash.

[aekiss]

Final output is at 16:00, ie not at a junction between the 3-hourly JRA55-do forcing data.

[aekiss]

Currents get quite fast in the Bosporus (~2m/s) now that the channel is confined.

I tried turning this off (CHANNEL_CONFIG = "none") and it still crashed (but ran 12 model hours longer).

[aekiss]

It's odd that it runs longer with @manodeep's performance tweak,

mpi:
  flags:
    - --mca mpi_yield_when_idle true

Edit: I actually meant NONBLOCKING_UPDATES=True.
given that this tweak is supposed to produce bitwise identical results.

I have only done one run with this tweak, and the subsequent one has been crashing. I wonder whether it did something to the restarts that led to instability? Seems implausible, given it's supposed to be bitwise identical and didn't crash in the first day.

I tried to redo the last run segment without this tweak to double-check but got

FATAL from PE     0:  reset_face_lengths_file: Unable to open ./INPUT/ocean_geometry.nc

any tips on what that's about?

[dougiesquire]

The geometry file should be called

access-om3.mom6.geometry.nc

Not sure why its looking for the default filename.

[manodeep]

It's odd that it runs longer with @manodeep's performance tweak,

mpi:
  flags:
    - --mca mpi_yield_when_idle true

given that this tweak is supposed to produce bitwise identical results.

Given that that parameter is only used within the OpenMPI runtime library, I would be extremely surprised if that parameter caused any perceptible changes in the userscope. Only thing that comes to mind is some race condition in user-code that gets exposed (but that's stretching my imagination).

[anton-seaice]

Output of surface speed at every timestep (900s) reveals nothing obviously wrong in the last timestep before the crash.

Do you have time to try with mediator diagnostics ?

from nuopc.runconfig:

     history_n_atm_avg = -999
     history_n_atm_inst = 1
     history_n_glc_avg = -999
     history_n_glc_inst = -999
     history_n_ice_avg = -999
     history_n_ice_inst = 1
     history_n_lnd_avg = -999
     history_n_lnd_inst = -999
     history_n_med_inst = 1
     history_n_ocn_avg = -999
     history_n_ocn_inst = 1
     history_n_rof_avg = 1
     history_n_rof_inst = 1
     history_n_wav_avg = -999
     history_n_wav_inst = -999
     history_option_atm_avg = never
     history_option_atm_inst = nsteps
     history_option_glc_avg = never
     history_option_glc_inst = never
     history_option_ice_avg = never
     history_option_ice_inst = nsteps
     history_option_lnd_avg = never
     history_option_lnd_inst = never
     history_option_med_inst = nsteps
     history_option_ocn_avg = never
     history_option_ocn_inst = nsteps
     history_option_rof_avg = nmonths
     history_option_rof_inst = nsteps
     history_option_wav_avg = never
     history_option_wav_inst = never

[dougiesquire]

I tried to redo the last run segment without this tweak to double-check but got

FATAL from PE     0:  reset_face_lengths_file: Unable to open ./INPUT/ocean_geometry.nc

any tips on what that's about?

Have you set CHANNEL_CONFIG = "file" and not set CHANNEL_WIDTH_FILE. Strangely, the default CHANNEL_WIDTH_FILE is ocean_geometry.nc, which is also the default GEOM_FILE. This smells like a potential copy-paste error in MOM.

[aekiss]

Ah, PEBKAC: I needed CHANNEL_CONFIG = "list", not "file"

[aekiss]

Sorry @manodeep I actually meant to say it runs 2 days longer with NONBLOCKING_UPDATES=True. mpi_yield_when_idle has no effect on the crash, as you expected.

So it seems NONBLOCKING_UPDATES doesn't guarantee bitwise reproducibility?