National Water Model HDFReference Recipe

[norlandrhagen]

Pangeo-Forge HDFReference recipe of the kerchunk NWM example . Heavily adapted from notebook created by rsignell-usgs.

[andersy005]

/run NWM

[andersy005]

pre-commit.ci autofix

[andersy005]

/run NWM

[andersy005]

/run NWM

[andersy005]

/run NWM

[andersy005]

/run NWM

[TomAugspurger]

FYI, I was running a variant of this for some data in Azure and ran into some issues with the data pushed to blob storage by NODD. I'm not sure if the AWS bucket has data this old, but it seems that on 2022-06-28, the internal chunking of the NetCDF files changed

In Azure, the URLs are

1. "abfs://nwm/nwm.20220628/short_range/nwm.t11z.short_range.channel_rt.f001.conus.nc",
2. "abfs://nwm/nwm.20220628/short_range/nwm.t12z.short_range.channel_rt.f001.conus.nc",

Prior to that first URL, feature_id is chunked 925580. From the second URL (onwwards?), feature_id is chunked 2776738. This coincided with an update from NWM 2.1 to NWM 2.2

I don't believe kerchunk / Zarr can handle that kind of change.

I'll post an update once I have a working example, but I'm running into issues with kerchunk seemingly dropping a bunch of files from my expected output.

[TomAugspurger]

I blamed Kerchunk / Zarr too quickly for the dropped data (sorry Martin!). It was in pangeo-forge-recipes: pangeo-forge/pangeo-forge-recipes#515.