Support parsing commits from manifests

[G-Rath]

#94 adds support for using the osv.dev API which supports commits as well as versions - so we should make sure the manifest parsers are able to extract this information from their respective files as much as possible.

(This is mostly a place for me to dump info I find during research)

• cargo.lock
• package-lock.json
• yarn.lock
• pnpm-lock.yaml
• composer.lock
• Gemfile.lock
• go.mod
• pom.xml
• requirements.txt

[G-Rath]

cargo.lock

It looks like for crates.io, we want to extract this from the source property:

https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html#cargotoml-vs-cargolock

cargo.toml:

[package]
name = "hello_world"
version = "0.1.0"

[dependencies]
regex = { git = "https://github.com/rust-lang/regex.git" }

cargo.lock:

[[package]]
name = "hello_world"
version = "0.1.0"
dependencies = [
 "regex 1.5.0 (git+https://github.com/rust-lang/regex.git#9f9f693768c584971a4d53bc3c586c33ed3a6831)",
]

[[package]]
name = "regex"
version = "1.5.0"
source = "git+https://github.com/rust-lang/regex.git#9f9f693768c584971a4d53bc3c586c33ed3a6831"

[G-Rath]

package-lock.json

Some interesting learnings for this one - for npm v7+ (aka lockfile v2/3), a package.json like this:

{
  "name": "dep-app",
  "version": "1.0.0",
  "dependencies": {
    "mine": "G-Rath/npm-git-repo-2"
  }
}

Gives you a lockfile something like:

{
  "name": "dep-app",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "dep-app",
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "mine": "G-Rath/npm-git-repo-2"
      }
    },
    "node_modules/mine": {
      "name": "npm-git-repo-1",
      "version": "0.0.1",
      "resolved": "git+ssh://git@github.com/G-Rath/npm-git-repo-2.git#0a2d2506c1fe299691fc5db53a2097db3bd615bc",
      "license": "ISC",
      "dependencies": {
        "mine": "G-Rath/npm-git-repo-1"
      }
    },
    "node_modules/mine/node_modules/mine": {
      "name": "npm-git-repo-1",
      "version": "1.2.3",
      "resolved": "git+ssh://git@github.com/G-Rath/npm-git-repo-1.git#094e581aaf927d010e4b61d706ba584551dac502",
      "license": "ISC"
    }
  },
  "dependencies": {
    "mine": {
      "version": "git+ssh://git@github.com/G-Rath/npm-git-repo-2.git#0a2d2506c1fe299691fc5db53a2097db3bd615bc",
      "from": "mine@G-Rath/npm-git-repo-2",
      "requires": {
        "mine": "G-Rath/npm-git-repo-1"
      },
      "dependencies": {
        "mine": {
          "version": "git+ssh://git@github.com/G-Rath/npm-git-repo-1.git#094e581aaf927d010e4b61d706ba584551dac502",
          "from": "mine@G-Rath/npm-git-repo-1"
        }
      }
    }
  }
}

(G-Rath/npm-git-repo-2 has a dependency on G-Rath/npm-git-repo-1 with the same "mine" name).

It seems that the name property is taken from whatever the value of name in the repos package.json is, e.g. in the above both git dependencies have the same "name" because that's what's in their respective package.json files. Ultimately while this looks like a possible issue for the detector, I'm pretty sure it's irrelevant as there's no stable way to sort the two that doesn't have other issues (e.g. if we used the name in the key, alised packages wouldn't get checked correctly) - though I think it indicates another reason for using our own parser, as if we relied on npm we wouldn't have access to this info (it's output just uses the alised name). (actually we are parsing the name out of the key - I wonder if we should be trying to use name, as that strikes me as the smarter thing to do 🤔)

The version property in the lockfile is taken from whatever the version property is in the dependencies package.json - so we'll need to check resolved and try to extract the ref from that.

I've confirmed that this is the structure regardless of if you provide a branch or ref (including short), and using the short Github syntax vs the standard git+ssh syntax - it always gets resolved to a full ref in this format.

For npm v6 it's the same except that the string is stored as the value of version.

[G-Rath]

yarn.lock

Yarn looks to behave almost the same as npm, except on how it handles GitHub dependencies:

{
  "name": "dep-app",
  "version": "1.0.0",
  "dependencies": {
    "mine1": "git+ssh://git@github.com:G-Rath/npm-git-repo-2#0a2d2506c1",
    "mine2": "G-Rath/npm-git-repo-2#0a2d2506c1"
  }
}

gives

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


"mine1@git+ssh://git@github.com:G-Rath/npm-git-repo-2#0a2d2506c1":
  version "0.0.1"
  resolved "git+ssh://git@github.com:G-Rath/npm-git-repo-2#0a2d2506c1fe299691fc5db53a2097db3bd615bc"
  dependencies:
    mine G-Rath/npm-git-repo-1

mine2@G-Rath/npm-git-repo-2#0a2d2506c1:
  version "0.0.1"
  resolved "https://codeload.github.com/G-Rath/npm-git-repo-2/tar.gz/0a2d2506c1fe299691fc5db53a2097db3bd615bc"
  dependencies:
    mine G-Rath/npm-git-repo-1

mine@G-Rath/npm-git-repo-1:
  version "1.2.3"
  resolved "https://codeload.github.com/G-Rath/npm-git-repo-1/tar.gz/094e581aaf927d010e4b61d706ba584551dac502"

I don't expect that URL structure will change but there could be other url resources - for now we'll just stick with checking for codeload.github.com I think and deal with improvements later if they come up.

When using a branch target, the commit hash is still used:

"mine1@git+ssh://git@github.com:G-Rath/npm-git-repo-2#another-branch":
  version "0.0.2"
  resolved "git+ssh://git@github.com:G-Rath/npm-git-repo-2#aa3bdfcb1d845c79f14abb66f60d35b8a3ee5998"

mine2@G-Rath/npm-git-repo-2#another-branch:
  version "0.0.2"
  resolved "https://codeload.github.com/G-Rath/npm-git-repo-2/tar.gz/aa3bdfcb1d845c79f14abb66f60d35b8a3ee5998"

Interestingly, yarn berry doesn't like having the commit hash in short form - the lockfile ends up like this:

+"mine1@git+ssh://git@github.com:G-Rath/npm-git-repo-2#0a2d2506c1fe299691fc5db53a2097db3bd615bc":
+  version: 0.0.1
+  resolution: "mine1@git+ssh://git@github.com:G-Rath/npm-git-repo-2#commit=0a2d2506c1fe299691fc5db53a2097db3bd615bc"
+  dependencies:
+    mine: G-Rath/npm-git-repo-1
+  checksum: a9ff7e157df4838502712d6a6cde5a64d927c804614e8819987e2bc8097baeca17d20baf8a4d355ea54b102a141549cc70ccc3be426f19a0c779b39986aef6da
+  languageName: node
+  linkType: hard
+
+"mine2@G-Rath/npm-git-repo-2#0a2d2506c1fe299691fc5db53a2097db3bd615bc":
+  version: 0.0.1
+  resolution: "mine2@https://github.com/G-Rath/npm-git-repo-2.git#commit=0a2d2506c1fe299691fc5db53a2097db3bd615bc"
+  dependencies:
+    mine: G-Rath/npm-git-repo-1
+  checksum: 2e7a9f3f9804e925195dcb5fe44416d140d92a786ec966eaf06f03e29f163e448a83969645dbb3900c3e39cb6489b73cd89810890076cbb0ef982f7554def2e5
+  languageName: node
+  linkType: hard
+
+mine@G-Rath/npm-git-repo-1:
+  version: 1.2.3
+  resolution: "mine@https://github.com/G-Rath/npm-git-repo-1.git#commit=094e581aaf927d010e4b61d706ba584551dac502"
+  checksum: ab97cf811a9520410120ceb30d43feba7f93ff254b4775990e9ecc242980154b9918e327cb07b6984c026d7bbcbe552f3e8c4e23123e7cfd97b5cfbeebb492d0
+  languageName: node
+  linkType: hard

(ignoring diff markers - I only have one project setup with yarn berry)

Likewise this doesn't change if using e.g. branches.

So I think for v2 we'll want to check resolution for commit=<value>.

[G-Rath]

For go.mod things seem pretty well spec'd - we're wanting to parse "pseudo-versions": https://go.dev/ref/mod#glos-pseudo-version

A version that encodes a revision identifier (such as a Git commit hash) and a timestamp from a version control system. For example, v0.0.0-20191109021931-daa7c04131f5. Used for compatibility with non-module repositories and in other situations when a tagged version is not available.

and https://go.dev/ref/mod#vcs-pseudo:

The last 12 characters of the pseudo-version (daa7c04131f5 in the example above) indicate a revision in the repository to check out. The meaning of this depends on the version control system. For Git and Mercurial, this is a prefix of a commit hash. For Subversion, this is a zero-padded revision number.

Also: https://go.dev/ref/mod#pseudo-versions

So we should just check if a version string ends with - & 12 chars. The interesting question is should we remove this from the version string? 🤔

I've got an implementation for this, but am holding off landing it because it actually makes the detector weaker because e.g. for golang.org/x/crypto/ssh the version v0.0.0-20200220183623-1d94cc7ab1c6 can be matched correctly against some vulnerabilities both locally and by the API, but v0.0.0 cannot and the api doesn't support commits well enough yet to resolve the commit (1d94cc7ab1c6) to the correct vulnerabilities either.