[Chore]: Rewrite filesystem handling and URL handling

[ascopes]

Brief description

I would like to investigate potentially moving off of java.nio onto commons-vfs2. Failing this, I'd like to see if we can be able to wrap file system handling and URL handling in a provider type rather than relying on the JDK ServiceLoaders.

If not possible, this issue will be scoped into research into ways to make FileSystem and URL handling less of a nightmare.

Aims and goals

Improve file system handling and URL handling to be less of a pain in the backside to develop and maintain.

Current gripes

• No nice way of providing the ability to read proto files from TAR archives.
  • No NIO-compatible TAR implementation exists.
  • Apache Ant's TAR handling is analogous to ZipInputStream in the JDK - too low level without a bunch of logic to wrap it.
  • commons-vfs2 is not compatible with NIO, and the vfs2nio bridge is not actively maintained.
  • commons-compress which underpins commons-vfs2 is too low level for our use case.
  • Other TAR libraries are not maintained.
• Adding new FileSystem bridges is far from trivial.
  • Without an existing NIO implementation, half a dozen complex interleaved classes have to be implemented.
• URL support is separate from FileSystem support
  • When we want to fetch a single resource like a binary plugin, we just use a URL to access it, rather than NIO. This makes sense to some extent but means any new integrations need both a URLStreamHandlerProvider AND an entire NIO file system backend to work.
• URL handling is broken due to the way java.net.URL tries to use the system classloader.
  • We currently have to use URI to avoid early binding the URL and not using the correct classloader to detect SPI handler implementations.
  • Maven uses ClassWorlds to separate classloaders so we cannot run on the system classloader.
• Half the unit tests interacting with files use JUnit's @TempDir, the other half use the in-memory file system, so there is inconsistency.
  • JUnit's @TempDir is simpler.
  • The in-memory file system can emulate Windows file system handling AND Linux file system handling. This is useful for testing various cross-OS concerns.
  • Neither of the current mechanisms for FS interaction in tests will work with Apache Commons VFS, for example.

[ascopes]

VFS appears to be a no-go due to numerous bugs with handling nested URLs. I'm now seeing if we can just handle URLs for now in this by wrapping commons-compress with custom URL providers until a better solution comes along.

[ascopes]

Related #795 (comment)

[ascopes]

Covered by #802