Occasional incomplete grib files from FastHerbie

[williamhobbs]

I've confirmed that this has happened with IFS enfo. It might have happened with GEFS, but I'm not 100% sure about GEFS.

I'm occasionally getting grib files from FastHerbie that only include a subset of ensemble members for IFS enfo. It's only for some of the fxx values, so the resulting dataset looks ok, but ends up with nan at some time steps for some members.

I have a lot of details posted here williamhobbs/hefty#28, along with a sample file that I saved that should make this (partially) reproducible.

At a minimum, I'm trying to figure out a clean way to catch this, because I can then re-run FH.download with overwrite=True, but any other thoughts/tips/suggestions would be super helpful.

[williamhobbs]

I hacked together a solution: williamhobbs/hefty#28 (comment). The general idea is the count the number of non-nan values in each lat/lon plane in the dataset, i.e., across all steps (fxx values) and members. Then any returned zeros mean the member doesn't have data, and a error is raised.

Then, if there's an error, I can run FH.download a second time with overwrite=True to overwrite the messed up grib files.