Is it possible to download the .idx files locally?

[tsilas3]

I know I can use .download() with FastHerbie to download grib files themselves, but when creating a Herbie object for a given date/time/forecast hour, it still uses AWS to get the associated .idx files even if the grib files are available locally. Is there a way to download the .idx files in the same way? If I'm understanding correctly, this would mean I would be able to run Herbie code without an internet connection.

[blaylockbk]

Hi @tsilas3

This is a great idea and one that would be beneficial to add to Herbie. I'll create a new issue (#215) to add this feature. It should be easy to make a flag download_inventory=True to always download the inventory files (thus enable an "offline" mode for Herbie). I will also have to add logic to tell Herbie to look for the inventory file locally before looking on the remote. (You'll see I thought about this some time ago, it just slipped through the cracks...

Herbie/herbie/core.py

Lines 444 to 445 in f48ba8c

	# NOTE: We will still get the idx files from a remote
	# because they aren't stored locally, or are they? # TODO: If the idx file is local, then use that

---

Just some notes to myself...

It's not obvious how, but you can download your GRIB2 files locally, then use _WGRIB2() to make inventory files for every GRIB2 file in a path.

Herbie/herbie/wgrib2.py

Lines 46 to 94 in f48ba8c

	class _WGRIB2:
	"""Wrapper for wgrib2 program."""
	# PATH to wgrib2 executable
	wgrib2 = which("wgrib2")
	def inventory(self, FILE):
	"""Return wgrib2-style inventory of GRIB2 file."""
	cmd = f"{self.wgrib2} -s {Path(FILE).expand()}"
	return run_command(cmd)
	def create_inventory_file(self, path, suffix=".grib2"):
	"""Create and save wgrib2 inventory files for GRIB2 file/files.
	Note that this will overwrite any existing inventory file.
	Parameters
	----------
	path : pathlib.path
	If path is a file, then make inventory file for that file.
	If path is a directory, then make inventory files for all
	files with the indicated suffix.
	suffix : {".grib2", ".grib", ".grb", etc.}
	If path specified is a directory, then this is the suffix to
	look for GRIB2 files.
	"""
	path = Path(path).expand()
	if path.is_dir():
	# List all GRIB2 files in the directory
	files = list(path.rglob(f"*{suffix}"))
	elif path.is_file():
	# The path is a single file
	files = [path]
	if not files:
	raise ValueError(f"No grib2 files were found in {path}")
	idx_files = []
	for f in files:
	f_idx = Path(str(f) + ".idx")
	idx_files.append(f_idx)
	index_data = self.inventory(Path(f))
	with open(f_idx, "w+") as out_idx:
	out_idx.write(index_data)
	if len(idx_files) == 1:
	return idx_files[0]
	else:
	return idx_files

Caveat: This requires the wgrib2 software be installed (not easily installed on Windows).

To use that you would do something like

# Download a file locally
from herbie import Herbie
H = Herbie("2023-7-17", model="hrrr", fxx=3)
H.download()

# Then create inventory files for all locally downloaded models
from herbie.wgrib2 import _WGRIB2
_WGRIB2().create_inventory_file("/home/blaylock/data/hrrr/20230717", suffix=".grib2")

This could be benefiicial for cases when there isn't an index file to download. Now, the problem is that Herbie doesn't currently know that it should look for the index files locally.