feat: add planetary-computer-multipart source, tests, and docs

[duncanmartyn]

Description

Adds a new source for multipart (multiple items and item assets) STAC collections on the open Microsoft Planetary Computer.

Design:

• Date handling in execute: matching date(s) passed to the method to asset URIs is necessary to mitigate datetime not found warnings.
• Source parent class rather than XarraySourceBase: requires multiple URIs and, possibly but unlikely, different storage options per asset. Also requires date matching to avoid warnings, which the current XarraySourceBase.execute does not support.
• query.datetime config key as a repetition of the dates section: facilitates access to the dataset's datetime range in __init__, resulting in fewer STAC API queries. Querying in execute may result in as many API requests as there are timestamps in the dates.start to dates.end range given iterative invocation of the method.
• Calendar handling: as with the rest of the project (from what I can tell), this source does not account for data not using the Gregorian calendar (e.g., CMIP6 models like UKESM1-0-LL using the 360-day calendar).
• Parameterised test: enables testing of a collection with multiple timesteps per ABFS URI Zarr store and another with one timestep per HTTPS URL NetCDF file using identical test code.
• Scoped to the open Planetary Computer: generalising to arbitrary STAC spec compliant catalogues would require parameterising the endpoint URL and authentication.
• Placed in the same file as the original planetary-computer source due to conceptual similarity and shared dependencies - happy to move to its own file if preferred.

Changes:

• coordinates.Coordinate.reduced and variable.Variable.sel - skip isel on scalar coordinates (e.g. met-office-global-deterministic-near-surface data time coordinate).

Two new dependencies are required by this change to handle remote NetCDF files with Xarray: h5netcdf and h5py.

What problem does this change solve?

The existing planetary-computer source pertains only to STAC collections for which there is a collection-level dataset asset under the zarr-abfs key corresponding to a single Zarr store containing all data. This source enables the use of collections in which data are in separate files or stores and referenced in distinct items and assets thereof.

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

---

📚 Documentation preview 📚: https://anemoi-datasets--610.org.readthedocs.build/en/610/

[floriankrb]

I understand you need something more to have the auto-discovery of the different parts of the data, so the current planetary-computer source is not enough. Thank you for sharing this code, this is likely to be helpfull for other and should be merged.

It would be nice to keep only one source though, could you please refactor to include the multi part code in the same source and do the branching with an option?

planetary-computer:
   mode: direct
  ...
  

and

planetary-computer:
   mode: multipart
  ...
  

Feel free to use the appropriate vocabulary, perhaps target and multiplart and mode are not the right terms to use.

[duncanmartyn]

Hi Florian, thanks for taking a look! My thinking behind not addint it to the original source was because the multipart doesn't (can't) use the XarraySourceBase. I wanted to avoid a scenario where the use / applicability of the parent class varied between wholly used with the singlepart zarr-abfs collections to not at all (beyond passing context) for the multipart collections that need a different execute method.

If I've got that wrong or you're happy with the above then I can definitely look into it. Thinking we could avoid a "mode" or similar config parameter by auto-detecting. That is, if the zarr-abfs key exists, use that, if not, assume multipart. Removes the burden of identifying the right mode from the user but being explicit might be preferable.

[floriankrb]

That is, if the zarr-abfs key exists, use that, if not, assume multipart.
I looks like a good idea, yes.

Sorry for insisting on this, I understand that the code may become more complex, but having a simpler interface for the user is more important imho.

[duncanmartyn]

Not a problem , makes sense. Pending CI checks I've consolidated the sources and updated docs and tests:

• Couldn't get flavour and rules working with the alternatively named level dimension for the met-office-global-deterministic-pressure test. In the data, the level dimension is named "pressure", and unless I passed a "pressure" parameter in the config (in place of "levels"), it wouldn't work. If you know the fix it'd be great for me to remedy that.
• Retained the original data_catalog_id arg rather than collection_id for backwards compat
• Changed filtering to explicit CQL2 in the config with support for string or dict formatting. Gives more flexibility where the previous auto-built a CQL2 query limited to AND and "=" filters.

EDIT: in any case, the test I mentioned looks to make CI significantly slower. Happy to remove as it may be surplus anyway.
EDIT2: removed the test. Prior to this it passed as expected in the 3.11 and 3.12 checks but stalled during 3.13 for some reason.

[floriankrb]

Approved.

This should be merged as soon as the branch is updated. @duncanmartyn , I will let you to update or rebase.

As a follow up, perhaps another PR, I wonder if this could be extended/share code with a stac source.

[duncanmartyn]

@floriankrb thanks, rebased. Looks like it needs an ATS label if you're able to add that, please.

On the shared / extended source, agreed. It's mainly a question of the differences between static and dynamic catalogues, of which Planetary Computer is the latter, but I'll look into it!