fix: address review findings for hermetic prefetch scripts

- Fix CACHI2_OUT_DIR not used by hermeto-fetch-gomod.sh and download-rpms.sh,
  which would write deps to wrong directory under namespaced layout
- Fix npm download failures silently swallowed: add return 1 on wget failure
  and check xargs exit code
- Fix should_keep_for_arch false positives: parse wheel platform tag instead
  of naive substring matching (e.g. "any" in "manylinux" was always true)
- Fix --arch silently skipped when yq missing in hermeto-fetch-rpm.sh: now
  fails fast with clear error
- Normalize COMPONENT_DIR (strip trailing slash) before hashing to match
  Makefile's patsubst behavior
- Use symlinks instead of cp -r in hermeto-fetch-rpm.sh arch filtering
- Remove dead counter variables in download-npm.sh
- Update docs to reflect hashed cachi2/output/<hash>/ layout
- Fix stale README claim about download-pip-packages.py not being called
- Fix missing newline at end of create-requirements-lockfile.sh

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Author: jiridanek

docs/learnings/hermetic-build-architecture_.md

@@ -21,11 +21,11 @@ Orchestrates five lockfile generators in sequence:
 
 | Step | Generator | Input | Output |
 |------|-----------|-------|--------|
-| 1 | `create-artifact-lockfile.py` | `artifacts.in.yaml` | `cachi2/output/deps/generic/` (GPG keys, nfpm, node headers, oc client, VS Code extensions) |
-| 2 | `create-requirements-lockfile.sh` | `pyproject.toml` | `cachi2/output/deps/pip/` (Python wheels) |
-| 3 | `download-npm.sh` | `package-lock.json` files | `cachi2/output/deps/npm/` (npm tarballs) |
-| 4 | `hermeto-fetch-rpm.sh` | `rpms.lock.yaml` | `cachi2/output/deps/rpm/{arch}/` (RPMs + repo metadata) |
-| 5 | `create-go-lockfile.sh` | `go.mod` (via git submodule) | `cachi2/output/deps/gomod/` (Go modules) |
+| 1 | `create-artifact-lockfile.py` | `artifacts.in.yaml` | `cachi2/output/<hash>/deps/generic/` (GPG keys, nfpm, node headers, oc client, VS Code extensions) |
+| 2 | `create-requirements-lockfile.sh` | `pyproject.toml` | `cachi2/output/<hash>/deps/pip/` (Python wheels) |
+| 3 | `download-npm.sh` | `package-lock.json` files | `cachi2/output/<hash>/deps/npm/` (npm tarballs) |
+| 4 | `hermeto-fetch-rpm.sh` | `rpms.lock.yaml` | `cachi2/output/<hash>/deps/rpm/{arch}/` (RPMs + repo metadata) |
+| 5 | `create-go-lockfile.sh` | `go.mod` (via git submodule) | `cachi2/output/<hash>/deps/gomod/` (Go modules) |
 
 Variants are selected via `--rhds` flag:
 - Default (`odh`): uses CentOS Stream + UBI repos (no subscription needed)
@@ -34,15 +34,20 @@ Variants are selected via `--rhds` flag:
 ### Makefile auto-detection
 
 ```makefile
-$(eval CACHI2_VOLUME := $(if $(and $(wildcard cachi2/output),$(wildcard $(BUILD_DIR)prefetch-input)),\
-    --volume $(ROOT_DIR)cachi2/output:/cachi2/output:Z \
-    --volume $(ROOT_DIR)cachi2/output/deps/rpm/$(RPM_ARCH)/repos.d/:/etc/yum.repos.d/:Z,))
+$(eval COMPONENT_DIR_STR := $(patsubst %/,%,$(BUILD_DIR)))
+$(eval CACHI2_HASH := $(shell python3 -c "import hashlib; print(hashlib.md5('$(COMPONENT_DIR_STR)'.encode()).hexdigest())"))
+$(eval CACHI2_DIR := cachi2/output/$(CACHI2_HASH))
+$(eval CACHI2_VOLUME := $(if $(and $(wildcard $(CACHI2_DIR)),$(wildcard $(BUILD_DIR)prefetch-input)),\
+    --volume $(ROOT_DIR)$(CACHI2_DIR):/cachi2/output:Z \
+    --volume $(ROOT_DIR)$(CACHI2_DIR)/deps/rpm/$(RPM_ARCH)/repos.d/:/etc/yum.repos.d/:Z,))
 ```
 
-When both `cachi2/output/` and `<target>/prefetch-input/` exist, the Makefile
-automatically mounts the prefetched dependencies into the build. The second
-mount overlays `/etc/yum.repos.d/` with hermeto-generated repos, making local
-builds behave like Konflux (repos are already in place when the Dockerfile runs).
+When both `cachi2/output/<hash>/` and `<target>/prefetch-input/` exist, the Makefile
+automatically mounts the per-component prefetched dependencies into the build.
+The `<hash>` is the MD5 of the component directory name, allowing concurrent
+builds of different components without collisions. The second mount overlays
+`/etc/yum.repos.d/` with hermeto-generated repos, making local builds behave
+like Konflux (repos are already in place when the Dockerfile runs).
 
 ### sandbox.py
 
@@ -57,25 +62,33 @@ build context.
 
 ## cachi2/output Directory Structure
 
-After prefetching, the directory looks like:
+After prefetching, each component gets its own namespaced directory under
+`cachi2/output/<hash>/` (where `<hash>` is the MD5 of the component directory
+name, e.g. `cachi2/output/a1b2c3.../`). This prevents collisions when building
+multiple components in parallel.
 
 ```text
 cachi2/output/
-├── deps/
-│   ├── rpm/
-│   │   ├── x86_64/
-│   │   │   ├── <repo-name>/       # RPM files + repodata/
-│   │   │   └── repos.d/           # Generated .repo files with file:// URLs
-│   │   ├── aarch64/
-│   │   ├── ppc64le/
-│   │   └── s390x/
-│   ├── npm/                       # npm tarballs
-│   ├── pip/                       # Python wheels
-│   └── generic/                   # GPG keys, tarballs, etc.
-├── bom.json
-└── .build-config.json
+└── <hash>/                        # per-component namespace (MD5 of component dir)
+    ├── deps/
+    │   ├── rpm/
+    │   │   ├── x86_64/
+    │   │   │   ├── <repo-name>/   # RPM files + repodata/
+    │   │   │   └── repos.d/       # Generated .repo files with file:// URLs
+    │   │   ├── aarch64/
+    │   │   ├── ppc64le/
+    │   │   └── s390x/
+    │   ├── npm/                   # npm tarballs
+    │   ├── pip/                   # Python wheels
+    │   └── generic/               # GPG keys, tarballs, etc.
+    ├── bom.json
+    └── .build-config.json
 ```
 
+The Makefile mounts `cachi2/output/<hash>/` at `/cachi2/output/` inside the
+container, so the Dockerfile always sees `/cachi2/output/deps/...` regardless
+of which component is being built.
+
 Key detail: when `rpms.in.yaml` declares `moduleEnable: [nodejs:22]`, hermeto
 downloads module metadata (`modules.yaml`) alongside the RPMs and includes it
 in the generated repodata. This allows `dnf module enable nodejs:22` to work
@@ -91,7 +104,7 @@ scripts/lockfile-generators/prefetch-all.sh --component-dir codeserver/ubi9-pyth
 make codeserver-ubi9-python-3.12
 ```
 
-Makefile detects `cachi2/output/` and auto-injects the volume mount.
+Makefile detects `cachi2/output/<hash>/` and auto-injects the volume mount.
 
 ### GitHub Actions
 
@@ -108,11 +121,13 @@ The TEMPLATE workflow (`build-notebooks-TEMPLATE.yaml`) handles it transparently
 
 1. PipelineRun YAML declares `prefetch-input` entries pointing to lockfiles
 2. cachi2's `prefetch-dependencies` task downloads everything using hermeto
-3. Build task mounts `/cachi2/output/` automatically
+3. Build task mounts deps at `/cachi2/output/` automatically
 4. Network isolation enforced at the pipeline level
 
-All three environments produce the same `/cachi2/output/deps/` structure because
-they all use hermeto under the hood for RPM prefetching.
+All three environments produce the same `/cachi2/output/deps/` layout inside
+the container because they all use hermeto under the hood for RPM prefetching.
+On the host, local/GHA builds use `cachi2/output/<hash>/deps/` while Konflux
+uses its own staging directory.
 
 ## Variant Directories (ODH vs RHDS)
 

scripts/lockfile-generators/README.md

@@ -153,7 +153,7 @@ internally. Option 6 (Git submodule) is a manual setup.
 | Helper | Used by | Purpose |
 |--------|---------|---------|
 | `helpers/pylock-to-requirements.py` | pip | Convert `pylock.<flavor>.toml` (PEP 751) to pip-compatible `requirements.<flavor>.txt` with `--hash` lines. |
-| `helpers/download-pip-packages.py` | pip | Standalone pip downloader: downloads wheels/sdists from a `requirements.txt` (with `--hash` lines) into `cachi2/output/deps/pip/`. Not called by `create-requirements-lockfile.sh` (which has its own inline download from pylock.toml). |
+| `helpers/download-pip-packages.py` | pip | Pip downloader: downloads wheels/sdists from a `requirements.txt` (with `--hash` lines) into `cachi2/output/<hash>/deps/pip/`. Called by `create-requirements-lockfile.sh --download`. Supports `--arch` filtering and parallel downloads. |
 | `helpers/download-rpms.sh` | RPM | Download RPMs from `rpms.lock.yaml` via `wget` into `cachi2/output/deps/rpm/` and create DNF repo metadata. Standalone alternative to `hermeto-fetch-rpm.sh`. |
 | `helpers/hermeto-fetch-rpm.sh` | RPM | Download RPMs from `rpms.lock.yaml` using [Hermeto](https://github.com/hermetoproject/hermeto) in a container. Handles RHEL entitlement cert extraction for `cdn.redhat.com` auth. Called by `create-rpm-lockfile.sh --download`. |
 | `helpers/hermeto-fetch-npm.sh` | npm | Alternative npm fetcher using [Hermeto](https://github.com/hermetoproject/hermeto) in a container. |

scripts/lockfile-generators/create-requirements-lockfile.sh

@@ -171,4 +171,4 @@ echo "  pylock.toml      : ${PYLOCK_FILE}"
 echo "  requirements     : ${REQUIREMENTS_FILE}"
 if [[ "$DO_DOWNLOAD" == true ]]; then
   echo "  wheels           : ${OUT_DIR}/"
-fi
+fi

scripts/lockfile-generators/download-npm.sh

@@ -245,10 +245,6 @@ fi
 mkdir -p "$DEST_DIR"
 
 total=$(echo "$refs" | wc -l | tr -d ' ')
-count=0
-downloaded=0
-skipped=0
-failed=0
 
 echo ""
 echo "Found $total unique packages to download."
@@ -266,12 +262,16 @@ download_file() {
         else
             echo "FAIL  Failed: $download_url" >&2
             rm -f "$DEST_DIR/$filename"
+            return 1
         fi
     fi
 }
 export -f download_file
 
-echo "$refs" | xargs -n 2 -P 10 bash -c 'download_file "$1" "$2"' _
+if ! echo "$refs" | xargs -n 2 -P 10 bash -c 'download_file "$1" "$2"' _; then
+    echo "Some npm downloads failed" >&2
+    exit 1
+fi
 
 echo ""
 echo "Finished! Location: $DEST_DIR"

scripts/lockfile-generators/helpers/download-pip-packages.py

@@ -180,33 +180,38 @@ def download_and_verify(item):
 
 
 def should_keep_for_arch(filename: str, arch: str) -> bool:
-    if "any" in filename or "noarch" in filename:
+    # sdists and non-wheel files have no platform tag — always keep
+    if not filename.endswith(".whl"):
         return True
-        
-    arch_tags = []
-    if arch in ["amd64", "x86_64"]:
-        arch_tags = ["x86_64", "amd64"]
-    elif arch in ["arm64", "aarch64"]:
-        arch_tags = ["aarch64", "arm64"]
-    elif arch == "ppc64le":
-        arch_tags = ["ppc64le"]
-    elif arch == "s390x":
-        arch_tags = ["s390x"]
-        
-    all_arches = ["x86_64", "amd64", "aarch64", "arm64", "ppc64le", "s390x"]
-    has_arch = False
-    for a in all_arches:
-        if a in filename:
-            has_arch = True
-            break
-            
-    if not has_arch:
+
+    # Wheel format: {name}-{ver}(-{build})?-{python}-{abi}-{platform}.whl
+    # The platform tag is the last segment before .whl
+    stem = filename[:-4]  # strip .whl
+    parts = stem.split("-")
+    if len(parts) < 3:
         return True
-        
-    for a in arch_tags:
-        if a in filename:
-            return True
-    return False
+    platform_tag = parts[-1]
+
+    # Pure-python wheels use "any" or "none" platform tags
+    if platform_tag in ("any", "none") or "any" in platform_tag.split("_"):
+        return True
+
+    ARCH_ALIASES = {
+        "amd64": ["x86_64", "amd64"],
+        "x86_64": ["x86_64", "amd64"],
+        "arm64": ["aarch64", "arm64"],
+        "aarch64": ["aarch64", "arm64"],
+        "ppc64le": ["ppc64le"],
+        "s390x": ["s390x"],
+    }
+    ALL_ARCHES = {"x86_64", "amd64", "aarch64", "arm64", "ppc64le", "s390x"}
+
+    # Check if the platform tag mentions any known architecture
+    if not any(a in platform_tag for a in ALL_ARCHES):
+        return True
+
+    # Keep only if it matches the target architecture
+    return any(a in platform_tag for a in ARCH_ALIASES.get(arch, []))
 
 
 def main():

scripts/lockfile-generators/helpers/download-rpms.sh

@@ -15,7 +15,7 @@ set -euo pipefail
 
 # --- Configuration & Defaults ---
 SCRIPTS_PATH="scripts/lockfile-generators"
-DEST_DIR="./cachi2/output/deps/rpm"
+DEST_DIR="${CACHI2_OUT_DIR:-cachi2/output}/deps/rpm"
 LOCKFILE=""
 
 # --- Functions ---

scripts/lockfile-generators/helpers/hermeto-fetch-gomod.sh

@@ -12,7 +12,7 @@ set -euo pipefail
 # directory. No separate lockfile is needed — go.sum pins dependencies.
 
 HERMETO_IMAGE="ghcr.io/hermetoproject/hermeto:0.46.2"
-HERMETO_OUTPUT="./cachi2/output"
+HERMETO_OUTPUT="${CACHI2_OUT_DIR:-cachi2/output}"
 
 PREFETCH_DIR=""
 

scripts/lockfile-generators/helpers/hermeto-fetch-rpm.sh

@@ -214,7 +214,8 @@ HERMETO_STAGING=$(mktemp -d)
 trap 'rm -rf "$HERMETO_STAGING" ${CDN_CERT_DIR:+"$CDN_CERT_DIR"} ${CLEANUP_SOURCE:+"$HERMETO_SOURCE"}' EXIT
 
 HERMETO_SOURCE="$(pwd)/$PREFETCH_DIR"
-if [[ -n "$ARCH" ]] && command -v yq &>/dev/null; then
+if [[ -n "$ARCH" ]]; then
+  command -v yq &>/dev/null || error_exit "--arch requires yq to filter rpms.lock.yaml"
   # Translate generic arch to rpm arch format
   rpm_arch="$ARCH"
   [[ "$ARCH" == "amd64" ]] && rpm_arch="x86_64"
@@ -223,8 +224,10 @@ if [[ -n "$ARCH" ]] && command -v yq &>/dev/null; then
   echo "--- Filtering rpms.lock.yaml for architecture: $rpm_arch ---"
   HERMETO_SOURCE=$(mktemp -d)
   CLEANUP_SOURCE=1
-  cp -r "$PREFETCH_DIR"/* "$HERMETO_SOURCE/"
-  yq eval "del(.arches[] | select(.arch != \"$rpm_arch\" and .arch != \"noarch\"))" -i "$HERMETO_SOURCE/rpms.lock.yaml"
+  # Symlink everything, then replace rpms.lock.yaml with a filtered copy
+  ln -s "$(pwd)/$PREFETCH_DIR"/* "$HERMETO_SOURCE/" 2>/dev/null || true
+  rm -f "$HERMETO_SOURCE/rpms.lock.yaml"
+  yq eval "del(.arches[] | select(.arch != \"$rpm_arch\" and .arch != \"noarch\"))" "$(pwd)/$PREFETCH_DIR/rpms.lock.yaml" > "$HERMETO_SOURCE/rpms.lock.yaml"
 fi
 
 echo "--- Downloading RPMs via hermeto ---"

scripts/lockfile-generators/prefetch-all.sh

@@ -141,9 +141,11 @@ while [[ $# -gt 0 ]]; do
 done
 
 [[ -z "$COMPONENT_DIR" ]] && error_exit "--component-dir is required."
+COMPONENT_DIR="${COMPONENT_DIR%/}"
 [[ -d "$COMPONENT_DIR" ]] || error_exit "Component directory not found: $COMPONENT_DIR"
 
-export CACHI2_OUT_DIR="cachi2/output/$(python3 -c "import hashlib; print(hashlib.md5('$COMPONENT_DIR'.encode()).hexdigest())")"
+CACHI2_HASH="$(python3 -c "import hashlib, sys; print(hashlib.md5(sys.argv[1].encode()).hexdigest())" "$COMPONENT_DIR")"
+export CACHI2_OUT_DIR="cachi2/output/${CACHI2_HASH}"
 export ARCH
 
 # CLI args take priority; fall back to env vars so GHA can pass secrets