[JENKINS-66621] Staggered copy artifact and high load

[jenkins-infra-bot]

After running Jenkins for a couple of weeks, we start seeing high load when multiple jobs are copying artifacts at the same time. 

 

During investigation I found that instead of copying artifacts all in one go, it's copyied in chucks of 5 seconds with a 15 second delay between chucks. This not only means that copying artifcats take 4 times as long, but it also seems to increase loading times on the web interface which in turns triggers our monitoring system for Jenkins healthcheck 

 

The Jenkins Dashboard also shows a number of jobs in progress on the master, but also not using the single executor on the master. I assume this means it's running on the master, but it's not a job component ? Maybe a clearer way of show what the jobs is doing on the master would help with troubleshooting as well ? I've attached a screenshot as well. 

 

I've attached a screenshot of nload while experiencing the issue as well as a jstack

---

Originally reported by hentis, imported from: Staggered copy artifact and high load

• status: Open
• priority: Major
• component(s): copyartifact-plugin
• resolution: Unresolved
• votes: 0
• watchers: 1
• imported: 20260107-133739

Raw content of original issue

After running Jenkins for a couple of weeks, we start seeing high load when multiple jobs are copying artifacts at the same time. 

 

During investigation I found that instead of copying artifacts all in one go, it's copyied in chucks of 5 seconds with a 15 second delay between chucks. This not only means that copying artifcats take 4 times as long, but it also seems to increase loading times on the web interface which in turns triggers our monitoring system for Jenkins healthcheck 

 

The Jenkins Dashboard also shows a number of jobs in progress on the master, but also not using the single executor on the master. I assume this means it's running on the master, but it's not a job component ? Maybe a clearer way of show what the jobs is doing on the master would help with troubleshooting as well ? I've attached a screenshot as well. 

 

I've attached a screenshot of nload while experiencing the issue as well as a jstack

environment

Jenkins: 2.289.2<br/>
OS: Linux - 3.16.0-11-amd64<br/>
openjdk version &quot;1.8.0_171&quot;<br/>
PrioritySorter:4.0.0<br/>
ace-editor:1.1<br/>
ant:1.11<br/>
antisamy-markup-formatter:2.1<br/>
apache-httpcomponents-client-4-api:4.5.13-1.0<br/>
async-http-client:1.9.40.0<br/>
authentication-tokens:1.4<br/>
bootstrap4-api:4.6.0-3<br/>
bootstrap5-api:5.0.2-1<br/>
bouncycastle-api:2.21<br/>
branch-api:2.6.5<br/>
build-alias-setter:0.4<br/>
build-failure-analyzer:2.0.0<br/>
build-timeout:1.20<br/>
caffeine-api:2.9.1-23.v51c4e2c879c8<br/>
checks-api:1.7.2<br/>
cloudbees-folder:6.16<br/>
command-launcher:1.6<br/>
conditional-buildstep:1.4.1<br/>
copyartifact:1.46.1<br/>
credentials:2.5<br/>
credentials-binding:1.27<br/>
display-url-api:2.3.5<br/>
docker-commons:1.17<br/>
docker-workflow:1.26<br/>
durable-task:1.38<br/>
echarts-api:5.1.2-3<br/>
email-ext:2.83<br/>
extended-read-permission:3.2<br/>
external-monitor-job:1.7<br/>
font-awesome-api:5.15.3-4<br/>
ghprb:1.42.2<br/>
git:4.8.0<br/>
git-client:3.8.0<br/>
git-server:1.10<br/>
github:1.33.1<br/>
github-api:1.123<br/>
github-branch-source:2.11.2<br/>
github-oauth:0.33<br/>
github-organization-folder:1.6<br/>
gradle:1.37.1<br/>
groovy:2.4<br/>
handlebars:3.0.8<br/>
icon-shim:3.0.0<br/>
jackson2-api:2.12.4<br/>
javadoc:1.6<br/>
jdk-tool:1.5<br/>
jjwt-api:0.11.2-9.c8b45b8bb173<br/>
job-dsl:1.77<br/>
jobConfigHistory:2.28.1<br/>
jquery:1.12.4-1<br/>
jquery-detached:1.2.1<br/>
jquery-ui:1.0.2<br/>
jquery3-api:3.6.0-2<br/>
jsch:0.1.55.2<br/>
junit:1.51<br/>
ldap:2.7<br/>
lockable-resources:2.11<br/>
logfilesizechecker:1.5<br/>
mailer:1.34<br/>
mapdb-api:1.0.9.0<br/>
matrix-auth:2.6.8<br/>
matrix-project:1.19<br/>
maven-plugin:3.12<br/>
metrics:4.0.2.8<br/>
momentjs:1.1.1<br/>
msbuild:1.30<br/>
nodelabelparameter:1.8.1<br/>
okhttp-api:3.14.9<br/>
pam-auth:1.6<br/>
parameterized-trigger:2.41<br/>
pipeline-build-step:2.14<br/>
pipeline-github-lib:1.0<br/>
pipeline-graph-analysis:1.11<br/>
pipeline-input-step:2.12<br/>
pipeline-milestone-step:1.3.2<br/>
pipeline-model-api:1.8.5<br/>
pipeline-model-declarative-agent:1.1.1<br/>
pipeline-model-definition:1.8.5<br/>
pipeline-model-extensions:1.8.5<br/>
pipeline-rest-api:2.19<br/>
pipeline-stage-step:2.5<br/>
pipeline-stage-tags-metadata:1.8.5<br/>
pipeline-stage-view:2.19<br/>
pipeline-utility-steps:2.8.0<br/>
plain-credentials:1.7<br/>
plugin-util-api:2.4.0<br/>
popper-api:1.16.1-2<br/>
popper2-api:2.5.4-3<br/>
promoted-builds:3.10<br/>
rebuild:1.32<br/>
resource-disposer:0.16<br/>
run-condition:1.5<br/>
scm-api:2.6.4<br/>
script-security:1.77<br/>
snakeyaml-api:1.29.1<br/>
ssh-agent:1.23<br/>
ssh-credentials:1.19<br/>
ssh-slaves:1.32.0<br/>
sshd:3.1.0<br/>
structs:1.23<br/>
subversion:2.14.4<br/>
throttle-concurrents:2.3<br/>
timestamper:1.13<br/>
token-macro:2.15<br/>
trilead-api:1.0.13<br/>
variant:1.4<br/>
windows-slaves:1.8<br/>
workflow-aggregator:2.6<br/>
workflow-api:2.46<br/>
workflow-basic-steps:2.23<br/>
workflow-cps:2.93<br/>
workflow-cps-global-lib:2.21<br/>
workflow-durable-task-step:2.39<br/>
workflow-job:2.41<br/>
workflow-multibranch:2.26<br/>
workflow-scm-step:2.13<br/>
workflow-step-api:2.24<br/>
workflow-support:3.8<br/>
ws-cleanup:0.39<br/>

3 attachments

• 2021-09-13 10_33_58-Dashboard [Jenkins]-1.png

  

• 2021-09-13 10_34_54-nload.png

  

• jstack.log