Improve configuration for automatically adding supplementary variables

[bouweandela]

To automatically add supplementary variables to a dataset, we need the following information:

1. Which facets are required for the data source(s) configured for the project?
2. Which facets should be inherited from the main variable?

These facets are then used in the following way:

ESMValCore/esmvalcore/_recipe/to_datasets.py

Lines 253 to 262 in 51b8c0d

	supplementary_facets: Facets = {
	facet: "*"
	for facet in FACETS.get(project, ["mip"])
	if facet not in _CMOR_KEYS + tuple(INHERITED_FACETS)
	}
	for key in ("frequency", "version"):
	# Do not inherit these facets as they tend to differ from the
	# main variable.
	if key in facets:
	supplementary_facets[key] = "*"

to figure out which facets and values to use to search for supplementary datasets. The _CMOR_FACETS in that code are the facets that are automatically added based on the variable definition in the CMOR table and

1. the facets used by the data source are currently defined in esmvalcore.io.esgf.facets.FACETS
2. the facets inherited from the main variable are currently defined in esmvalcore.dataset.INHERITED_FACETS

To make 1) configurable, we could add a facets attribute to all data sources, similar to the one already available on the IntakeESGFDatasource at

ESMValCore/esmvalcore/io/intake_esgf.py

Line 204 in 51b8c0d

facets: dict[str, str]

. For esmvalcore.io.local.LocalDataSource this could be automatically added from the file and directory name templates and for esmvalcore.io.esgf.ESGFDataSource it could be moved from esmvalcore.io.esgf.facets.FACETS to the configuration files. We could then use this, in combination with the list of inherited facets, to determine which facets from the main variable we need to replace with a * value.

To make 2) configurable, we could add an option called e.g. facets_inherited_by_supplementary_variables to the project configuration under projects, possibly with a default of ["project", "dataset"].

[bouweandela]

There is also a use of esmvalcore.io.esgf.facets.FACETS here:

ESMValCore/esmvalcore/preprocessor/_concatenate.py

Lines 218 to 222 in 51b8c0d

	# get the possible facet names in CMIP3, 5, 6 for exp
	# currently these are 'experiment', 'experiment_id'
	exp_facet_names = {
	project["exp"] for project in FACETS.values() if "exp" in project
	}

That may be more difficult to solve, unless we start writing the dirname and filename template for the local data source using the same mapping we use for the (intake-)esgf data source, but that may add complication for users.