feat: implement parallel repository management workflow#9224
feat: implement parallel repository management workflow#9224igorpecovnik merged 1 commit intomainfrom
Conversation
This commit implements a complete parallel repository management system
that allows building and publishing Debian repositories in parallel,
significantly reducing build time for multiple distributions.
- `update-main`: Builds common/main component once for all releases
- `update -R <release>`: Builds release-specific components in isolated DBs
- `merge`: Combines common + release-specific components into final repos
- Isolated databases (aptly-isolated-<release>) avoid locking during parallel builds
- Common component built once, not duplicated per release
- Release-specific components (utils, desktop) built independently
- Final merge combines all components with proper GPG signing
- Fixed GPG signing to target top-level Release files (dists/{release}/Release)
- Pool cleanup before publishing avoids "file already exists" errors
- Smart package import skips duplicates during merge
- Proper handling of empty repositories and missing components
- Improved error handling and logging throughout
1. update-main: Build common component (once)
2. update -R <release>: Parallel workers build release-specific components
3. merge: Combine all components and publish with GPG signatures
This enables GitHub Actions to run multiple release builders in parallel,
reducing total repository build time from hours to minutes.
Signed-off-by: Igor Pecovnik <[email protected]>
github-actions
bot
added
size/large
📝 WalkthroughWalkthroughThe repository script refactors its isolated-mode workflow to defer common component building to a dedicated merge step. Workers now skip common repository operations during parallel builds, focusing only on release-specific components. The merge process is significantly extended to integrate common components with isolated-built releases, handling snapshot importing, re-publishing, and final signing. Changes
Sequence Diagram(s)sequenceDiagram
participant Workers as Worker Nodes<br/>(Isolated)
participant IDB as Isolated DBs
participant MDB as Main DB
participant Pub as Publishing
participant Sign as Signing
participant Repo as Repository<br/>Control
Workers->>IDB: Build & snapshot<br/>release components
Workers->>Pub: Skip publishing<br/>(isolated mode)
Note over Workers: Process exits early
Note over MDB: Merge Step Initiated
rect rgb(200, 220, 255)
Pub->>MDB: Create/check main DB config
Pub->>MDB: Validate common snapshot exists
Pub->>MDB: Import snapshots from isolated DBs
IDB->>MDB: Transfer release-specific packages
MDB->>MDB: Deduplicate against main DB
Pub->>Pub: Re-publish with common components
Pub->>Sign: Final signing (non-isolated)
Sign->>Repo: Update control file
Repo->>Repo: Show merged repository content
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
bot
added
Needs review
igorpecovnik
added
Ready to merge
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @tools/repository/repo.sh:
- Around line 736-744: The conditional that checks membership using [[ "
${releases[@]} " =~ " ${release} " ]] is unsafe (ShellCheck SC2199); replace it
with a safe membership test such as piping printf '%s\n' "${releases[@]}" into
grep -qx "$release" (or implement an associative array/set for O(1) lookup) and
only append to the releases array when that test fails; update the block around
the releases and release variables accordingly so the code uses the grep/assoc
lookup instead of the regex-on-concatenated-array.
- Around line 808-828: The import loops use the pipeline form find | while read
... which executes the while-body in a subshell so run_aptly failures (which
call exit) only terminate the subshell; change both loops to use process
substitution so the while runs in the main shell (e.g., replace find
"$isolated_pool" -name "*.deb" -type f | while read -r deb_file; do ... done
with while IFS= read -r deb_file; do ... done < <(find "$isolated_pool" -name
"*.deb" -type f) ), ensure you preserve the same variable names (deb_file,
deb_info, deb_key, packages, main_db_packages) and the run_aptly
-config="$main_db_config" repo add calls, and apply the same change to the
desktop import loop to make exit propagate to the parent script.
🧹 Nitpick comments (1)
tools/repository/repo.sh (1)
706-709: Consider trap-based cleanup for temporary files.Temporary config files (
main_db_config,isolated_config) could leak if the script exits unexpectedly. A trap would ensure cleanup on exit.💡 Suggested improvement
merge_repos() { local input_folder="$1" local output_folder="$2" # Track temp files for cleanup local temp_files=() trap 'rm -f "${temp_files[@]}"' EXIT local main_db_config main_db_config="$(mktemp)" temp_files+=("$main_db_config") # ... rest of functionThis is optional since the current manual cleanup covers the main code paths.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
tools/repository/repo.sh
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: igorpecovnik
Repo: armbian/build PR: 9087
File: .github/workflows/pr-check-pictures.yml:138-146
Timestamp: 2025-12-16T13:40:07.649Z
Learning: In the Armbian build repository, when introducing new requirements or checks (like the board assets verification workflow), the project prefers an initial educational-only period where violations post helpful PR comments and warnings but don't block merges. This allows contributors to become familiar with new requirements before enforcement is enabled (typically after ~6 months).
📚 Learning: 2025-12-18T23:40:41.627Z
Learnt from: rpardini
Repo: armbian/build PR: 9101
File: lib/functions/image/compress-checksum.sh:44-48
Timestamp: 2025-12-18T23:40:41.627Z
Learning: In the Armbian build framework, scripts run with set -e, so a non-zero exit status will abort the script. During reviews, assume failures will stop execution unless explicitly handled. Verify that commands that must fail are checked, errors are propagated, and any critical steps have proper error handling (e.g., using pipefail where appropriate, checking exit codes, and not masking failures).
Applied to files:
tools/repository/repo.sh
🧬 Code graph analysis (1)
tools/repository/repo.sh (1)
tools/repository/extract-repo.sh (1)
log(21-23)
🪛 GitHub Actions: Shellcheck - PR #9224 ("feat: implement parallel repository management workflow")
tools/repository/repo.sh
[error] 739-739: ShellCheck SC2199 (error): Arrays implicitly concatenate in [[ ]]. Use a loop (or explicit * instead of @).
🔇 Additional comments (11)
tools/repository/repo.sh (11)
312-317: LGTM!Clear documentation of the workflow change where workers skip common repo building in isolated mode, deferring to the merge command.
374-382: LGTM!Proper conditional initialization of component arrays - isolated mode correctly excludes the common component which will be merged later.
438-444: LGTM!Correct early return in isolated mode - workers prepare repos and snapshots but defer publishing to the merge step.
462-466: LGTM!Good defensive programming - guards against publishing with no components, which would likely cause aptly to fail.
669-686: LGTM!The signing logic correctly targets only top-level Release files (
dists/{release}/Release) using both-maxdepth 2and slash counting validation. This prevents signing component-level Release files that don't need it.
750-763: LGTM!Proper validation logic - gracefully handles empty releases (no-op) and correctly errors when common snapshot is missing with helpful guidance.
908-916: LGTM!Good defensive practice using
${published_pool:?}to prevent accidentalrm -rf /*if the variable is unset. Pool cleanup before publishing is the correct approach to avoid "file already exists" conflicts.
918-996: LGTM!Solid publishing loop implementation with proper package checks, snapshot creation, and error handling with diagnostic output on failure.
998-1019: LGTM!Proper cleanup of temporary config and complete finalization including signing, GPG key distribution, control file update, and content display.
1228-1242: LGTM!Documentation correctly reflects the new workflow where workers focus only on release-specific components while common is built once and merged later.
593-597: LGTM!Improved logging guidance for single-release mode with clear instructions about the
update-maincommand.
| # Skip common distribution | ||
| [[ "$release" == "common" ]] && continue | ||
| # Add if not already in list | ||
| if [[ ! " ${releases[@]} " =~ " ${release} " ]]; then | ||
| releases+=("$release") | ||
| fi | ||
| fi | ||
| done | ||
| fi |
There was a problem hiding this comment.
Fix ShellCheck SC2199: Array concatenation in [[ ]] context.
The pipeline failure indicates that ${releases[@]} in the regex match implicitly concatenates array elements, which can lead to incorrect matching behavior.
🔧 Proposed fix
- # Add if not already in list
- if [[ ! " ${releases[@]} " =~ " ${release} " ]]; then
- releases+=("$release")
- fi
+ # Add if not already in list
+ local already_exists=false
+ for existing in "${releases[@]}"; do
+ if [[ "$existing" == "$release" ]]; then
+ already_exists=true
+ break
+ fi
+ done
+ if [[ "$already_exists" == false ]]; then
+ releases+=("$release")
+ fiAlternatively, a more concise approach using a helper function or associative array for O(1) lookup:
# Using printf and grep (concise but spawns subshell)
if ! printf '%s\n' "${releases[@]}" | grep -qx "$release"; then
releases+=("$release")
fi🧰 Tools
🪛 GitHub Actions: Shellcheck - PR #9224 ("feat: implement parallel repository management workflow")
[error] 739-739: ShellCheck SC2199 (error): Arrays implicitly concatenate in [[ ]]. Use a loop (or explicit * instead of @).
🤖 Prompt for AI Agents
In @tools/repository/repo.sh around lines 736 - 744, The conditional that checks
membership using [[ " ${releases[@]} " =~ " ${release} " ]] is unsafe
(ShellCheck SC2199); replace it with a safe membership test such as piping
printf '%s\n' "${releases[@]}" into grep -qx "$release" (or implement an
associative array/set for O(1) lookup) and only append to the releases array
when that test fails; update the block around the releases and release variables
accordingly so the code uses the grep/assoc lookup instead of the
regex-on-concatenated-array.
| find "$isolated_pool" -name "*.deb" -type f | while read -r deb_file; do | ||
| # Get package info to check if it belongs to this repo | ||
| local deb_name deb_version deb_arch | ||
| deb_info=$(dpkg-deb -f "$deb_file" Package Version Architecture 2>/dev/null) | ||
| deb_name=$(echo "$deb_info" | sed -n '1s/Package: //p') | ||
| deb_version=$(echo "$deb_info" | sed -n '2s/Version: //p') | ||
| deb_arch=$(echo "$deb_info" | sed -n '3s/Architecture: //p') | ||
| local deb_key="${deb_name}_${deb_version}_${deb_arch}" | ||
|
|
||
| # Check if this package is in the utils repo (isolated) | ||
| # aptly output has leading spaces, so grep without anchors | ||
| if echo "$packages" | grep -qw "${deb_key}"; then | ||
| # Check if package already exists in main DB repo to avoid conflicts | ||
| if echo "$main_db_packages" | grep -qw "${deb_key}"; then | ||
| # Package already in main DB, skip it | ||
| continue | ||
| fi | ||
| run_aptly -config="$main_db_config" repo add -force-replace "${release}-utils" "$deb_file" | ||
| fi | ||
| done | ||
| fi |
There was a problem hiding this comment.
Critical: run_aptly failures inside while read subshell won't terminate the script.
The find ... | while read pattern runs the loop in a subshell. When run_aptly calls exit 1 on failure, it only exits the subshell, not the parent script. The import continues silently even if package additions fail.
🔧 Proposed fix using process substitution
- find "$isolated_pool" -name "*.deb" -type f | while read -r deb_file; do
+ while read -r deb_file; do
# Get package info to check if it belongs to this repo
local deb_name deb_version deb_arch
deb_info=$(dpkg-deb -f "$deb_file" Package Version Architecture 2>/dev/null)
deb_name=$(echo "$deb_info" | sed -n '1s/Package: //p')
deb_version=$(echo "$deb_info" | sed -n '2s/Version: //p')
deb_arch=$(echo "$deb_info" | sed -n '3s/Architecture: //p')
local deb_key="${deb_name}_${deb_version}_${deb_arch}"
# Check if this package is in the utils repo (isolated)
if echo "$packages" | grep -qw "${deb_key}"; then
if echo "$main_db_packages" | grep -qw "${deb_key}"; then
continue
fi
run_aptly -config="$main_db_config" repo add -force-replace "${release}-utils" "$deb_file"
fi
- done
+ done < <(find "$isolated_pool" -name "*.deb" -type f)The same issue exists in the desktop import loop (lines 854-873). Apply the same fix there.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| find "$isolated_pool" -name "*.deb" -type f | while read -r deb_file; do | |
| # Get package info to check if it belongs to this repo | |
| local deb_name deb_version deb_arch | |
| deb_info=$(dpkg-deb -f "$deb_file" Package Version Architecture 2>/dev/null) | |
| deb_name=$(echo "$deb_info" | sed -n '1s/Package: //p') | |
| deb_version=$(echo "$deb_info" | sed -n '2s/Version: //p') | |
| deb_arch=$(echo "$deb_info" | sed -n '3s/Architecture: //p') | |
| local deb_key="${deb_name}_${deb_version}_${deb_arch}" | |
| # Check if this package is in the utils repo (isolated) | |
| # aptly output has leading spaces, so grep without anchors | |
| if echo "$packages" | grep -qw "${deb_key}"; then | |
| # Check if package already exists in main DB repo to avoid conflicts | |
| if echo "$main_db_packages" | grep -qw "${deb_key}"; then | |
| # Package already in main DB, skip it | |
| continue | |
| fi | |
| run_aptly -config="$main_db_config" repo add -force-replace "${release}-utils" "$deb_file" | |
| fi | |
| done | |
| fi | |
| while read -r deb_file; do | |
| # Get package info to check if it belongs to this repo | |
| local deb_name deb_version deb_arch | |
| deb_info=$(dpkg-deb -f "$deb_file" Package Version Architecture 2>/dev/null) | |
| deb_name=$(echo "$deb_info" | sed -n '1s/Package: //p') | |
| deb_version=$(echo "$deb_info" | sed -n '2s/Version: //p') | |
| deb_arch=$(echo "$deb_info" | sed -n '3s/Architecture: //p') | |
| local deb_key="${deb_name}_${deb_version}_${deb_arch}" | |
| # Check if this package is in the utils repo (isolated) | |
| # aptly output has leading spaces, so grep without anchors | |
| if echo "$packages" | grep -qw "${deb_key}"; then | |
| # Check if package already exists in main DB repo to avoid conflicts | |
| if echo "$main_db_packages" | grep -qw "${deb_key}"; then | |
| # Package already in main DB, skip it | |
| continue | |
| fi | |
| run_aptly -config="$main_db_config" repo add -force-replace "${release}-utils" "$deb_file" | |
| fi | |
| done < <(find "$isolated_pool" -name "*.deb" -type f) | |
| fi |
🤖 Prompt for AI Agents
In @tools/repository/repo.sh around lines 808 - 828, The import loops use the
pipeline form find | while read ... which executes the while-body in a subshell
so run_aptly failures (which call exit) only terminate the subshell; change both
loops to use process substitution so the while runs in the main shell (e.g.,
replace find "$isolated_pool" -name "*.deb" -type f | while read -r deb_file; do
... done with while IFS= read -r deb_file; do ... done < <(find "$isolated_pool"
-name "*.deb" -type f) ), ensure you preserve the same variable names (deb_file,
deb_info, deb_key, packages, main_db_packages) and the run_aptly
-config="$main_db_config" repo add calls, and apply the same change to the
desktop import loop to make exit propagate to the parent script.
1 participant
Summary
This PR implements a complete parallel repository management system that allows building and publishing Debian repositories in parallel, significantly reducing build time for multiple distributions.
Key Changes
New Commands
update-main: Builds common/main component once for all releasesupdate -R <release>: Builds release-specific components in isolated DBsmerge: Combines common + release-specific components into final reposArchitecture
aptly-isolated-<release>) avoid locking during parallel buildsFixes & Improvements
dists/{release}/Release)Workflow
Benefits
This enables GitHub Actions to run multiple release builders in parallel, reducing total repository build time from hours to minutes.
Testing
Documentation
Documentation references found:
Summary by CodeRabbit
Release Notes
✏️ Tip: You can customize this high-level summary in your review settings.