What is a good way to tell a netCDF4 file from a "regular" HDF5 file?

[krisfed]

I was originally intending to use nc_open and check the status, but it seems to work fine on HDF5 files (that are NOT netCDF4 files), and return a reasonable-looking ncid.

Even nc_inq_format seems to think a "regular" HDF5 file is a netCDF4 file (NC_FORMAT_NETCDF4).

My next best idea is to check for the presence of _NCProperties global attribute with something like nc_inq_att. This seems to work (works fine on a netCDF4 file and returns a NC_ENOTATT error status on a "regular" HDF5 file), but I am wondering if there is a better way.

Is a _NCProperties global attribute guaranteed to be present for all valid netCDF4 files? Should I be using something else?

I am working with v4.9.2 if it matters.

Thanks!

[DennisHeimbigner]

Is a _NCProperties global attribute guaranteed to be present for all valid netCDF4 files?

This mostly a good way to tell the difference. There is a couple of issues, though.
First, the _NCProperties attribute was added in netcdf-c version 4.4.1-RC2 - May 13, 2016.
So files created before than (and there are probably a lot of them) will not have _NCProperties defined.
Second, I have recollection that there is python package wrapping HDF5 that also attempts to
add _NCProperties to its files. It can be distinguished by the (key,value) pairs of that attribute, but I am not
off hand sure what pairs it defines.

[krisfed]

Thank you, Dennis! Would you recommend using something else, or is _NCProperties pretty much the best option (even if it is not 100% accurate)?

[krisfed]

Is there something else I can check for that would be more accurate than checking for _NCProperties?

Has something like isNC method been considered to be added as part of the netCDF library to be able to determine if a given file is a netCDF file?

I also found it a bit counter-intuitive that nc_open works on "normal" HDF5 files...

Would greatly appreciate any ideas!

Thanks,
Kris

[DennisHeimbigner]

There is one other possibility. There is an attribute called '_IsNetcdf4' that is backed by a program called "NC4_isnetcdf4" in "libhdf5/nc4hdf.c". The program looks for various elements of the netcdf-4/hdf5 file to decide if it was created by libnetcdf.
Currently it tests for existence of a variety of attributes, dimension names,etc in the file. It occurs to me that if the program/attribute returns false, and the file is otherwise readable by libnetcdf, then is should be a pure HDF5 file.

[krisfed]

This is great, @DennisHeimbigner ! This is exactly what I was looking for! I am able to get this attribute on different HDF5 and netCDF files (as you said, it is computed on the fly rather than stored), and it seems fairy reliable.

I found a good description for the _IsNetcdf4 attribute here:

This attribute is ephemeral in that it is computed by walking the metadata of the file looking for attributes specific to netCDF-4 files.

The _IsNetcdf4 attribute has the form: _IsNetcdf4 = 0|1 where 1 means the file has various tags indicating it was produced thru the netcdf-4 API. The type of this attribute is NC_INT.

This attribute is computed by using the HDF5 API to walk the file to look for attributes specific to netcdf-4. False negatives are possible for a small subset of netcdf-4 files, especially those not containing dimensions. False positives are only possible by deliberate modifications to an existing HDF5 file thru the HDF5 API. For files with the _NCProperties attribute, this attribute is redundant. For files created prior to the introduction of the _NCProperties attribute, this may be a useful indicator of the provenance of the file.

Thank you!

[krisfed]

I do see that for a small netCDF4 file that has _NCProperties but no dimensions, the _IsNetcdf4 attribute is 0/false. So for most robust results it might be best to check both _NCProperties and _IsNetcdf4.

[DennisHeimbigner]

I could simplify that by having _IsNetcdf4 include a check for _NCProperties attribute along with its current checks.
BTW refresh my memory. There is a Python package that sets _NCProperties, but specifies its own library as part
of the _NCProperties (key,value) pairs. Its name escapes me.But if you want to exclude files constructed using that
package, you would need to parse _NCProperties and look at specific keys. I think there is a function in netcdf_aux.h
that does that parsing for you.

[krisfed]

Including a check for _NCProperties in _IsNetcdf4 check would be great! Then we can simplify our own code logic when we upgrade to new netCDF version with this change.

Re Python package that uses _NCProperties attribute - I have not come across that yet. I think at some level it might be ok to count those types of files as netCDF-4 given their use of this netCDF-specific attribute.. But good to know about potential useful functionality in netcdf_aux.h.

[Dave-Allured]

In recent PR #3125, the check for the _NCProperties attribute was added to _IsNetcdf4. However, as already noted above, all netcdf-4 files created with library versions earlier than 4.4.1 (2016) do not have the _NCProperties attribute. Therefore, if I understand that PR correctly, all of these older netcdf-4 files will now read _IsNetcdf4 = "not netcdf-4".

Do you have guidelines for how users should handle this apparent contradiction? Are you sure that adding the _NCProperties check is the right thing to do for the general community? Please excuse my fussing, but I think this could be trouble in some cases.

[DennisHeimbigner]

Did I screw up that PR? I intended that any single test passing in IsNetCDF4 would return true.

[Dave-Allured]

Oh, "any single test passing". That is what I missed. I did not look that closely, sorry. I will take a second look and let you know if I see any remaining problem.

[Dave-Allured]

@DennisHeimbigner Okay, I confirm that "any single test passing" is implemented as you intended. Thanks for clarifying that for me. Problem solved.

[krisfed]

I think I came across some false positives (HDF5 files identified as netCDF files) for _IsNetcdf4 attribute check:

1. ATLAS/ICESat-2 L3A Sea Ice Height https://nsidc.org/data/atl07/versions/6
2. SMAP Surface and root zone soil moisture https://nsidc.org/data/spl4smlm/versions/8

The file format of these files is listed as HDF5 in the description and they have .h5 extension. Trying to look through the files' HDF5 structure, I see that they do use DIMENSION_LIST attribute similar to netCDF4 files, but I don't see any attributes like _Netcdf4Dimid or _Netcdf4Coordinates or _NCProperties.

The _IsNetcdf4 attribute check returns 1 for these files using netCDF v4.9.3.

Not 100% sure if it is correct or incorrect to consider these files to be valid netCDF4 files, but wanted to bring them to attention.