`stats`: use cached crate in addition to existing stats cache, with the blake3 hash of the file as the main key

[jqnatividad]

Currently, the stats cache is stored in the same location as the input data.

One file is the cache results (ending in ".stats.csv"), and when the --stats-jsonl option is chosen, a ".jsonl" file - both with the same file stem as the input file. The .jsonl file contains the stats options and some cache metadata used to generate the ".stats.csv" file.

The current caching mechanism only checks for the filename and filetime metadata - which is serviceable and works IRL - changing the input file necessarily makes it newer than the cache files, automatically invalidating the cache. However, it introduces some "noise" to the file system.

Before, I saw it as a "feature", not a bug, as these files are tiny are can be used independently.

However, It had some limitations:

• two more tiny files
• the cache may be invalid and still be used if the input file was changed somehow, and its still older than the stats cache
• if there was an identical file with a different name and/or in another location, separate cache files were maintained for them
• there was no easy way to share cache files for reference data with other users in the network

By using the cached crate, we can:

• use a configurable Disk Cache or alternately, a Redis Cache
• as the filehash is the main key, if an identical file, even with a different name is used, will generate a cache hit as the main key is the sha256 hash of the file
• with a Redis Cache, we can share the Cache with other users.

Computing the sha256 hash doesn't take that long with our CPU-accelerated version - 250ms for the 520mb NYC 311 file (contrast that with shasum -a 256 on the same machine, which is at least 4x slower).

[jqnatividad]

Do this once the sled has been replaced in the cached crate

jaemk/cached#237

[kulnor]

Hmm, I do see the 'noise' argument as being valid... and the filesystem could also be read-only (in some secure or archive environments).

So to save the stats file, the user would have to use the --output parameter explicitly. This aligns the command behaviour on frequency as reported in #3070.

Relying on the filename and the last update is certainly risky. One thing to consider is using the SHA256 value as the cached base filename (e.g. <computed-sha256-value>.stats.csv).

The benefit of this is that if the file is moved, renamed, or multiple copies are floating around, the cached file remains the same. The problem is garbage collection. You could also have a metadata file alongside the cache file (.meta.json`, pointing to the known location(s) of the data file.

I would also investigate using a .parquet format (or zip) to reduce disk space usage. Suspect the read/write overhead is small (if any).

Actually, an additional benefit of requiring the user to specify --output is to prevent the stats file from being accidentally overwritten. This is usually fine, as we want to make sure the stats are up to date. However, I can also see a use case for leveraging statistics to keep track of changes over time.

[kulnor]

Actually, the filename in the cache does not matter, right? Just need the key.

[jqnatividad]

Yep! File hashing doesn't need the filename, as the SHA256 hash itself is the key.

But thinking about it more, the only reason why I'm using SHA256 is that its the recommended hashing algorithm per NIST (DOI-DO/dcat-us#225).

The thing with SHA256 though is that its not parallelizable.

We can use another algorithm like Blake3 which is.
https://crates.io/crates/blake3

[jqnatividad]

Added a blake3 helper - #3072

Will implement it first in the describegpt command.