Feature Request: Add retry mechanism

[YuviGold]

Expected Behaviour

arkade is being used in CI pipelines and from time to time might fail on network hiccups.
It would be great if internally it could retry its http requests for common failures.

Current Behaviour

In the following examples you can see a network failures that caused to exit with an error without retrying

[2022-10-05T12:40:07.291Z] Downloading: terragrunt
[2022-10-05T12:40:07.291Z] 2022/10/05 12:40:07 Looking up version for terragrunt
[2022-10-05T12:40:07.291Z] 2022/10/05 12:40:07 Found: v0.39.0
[2022-10-05T12:40:07.292Z] Downloading: https://github.com/gruntwork-io/terragrunt/releases/download/v0.39.0/terragrunt_linux_amd64
[2022-10-05T12:40:13.910Z] Error: incorrect status for downloading tool: 503

[2022-10-06T00:02:02.842Z] Downloading: terragrunt
[2022-10-06T00:02:02.842Z] 2022/10/06 00:02:00 Looking up version for terragrunt
[2022-10-06T00:02:11.003Z] Error: Head "https://github.com/gruntwork-io/terragrunt/releases/latest": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Are you a GitHub Sponsor yet (Yes/No?)

• Yes
• No

Possible Solution

Using https://github.com/hashicorp/go-retryablehttp that wouldn't require any refactoring. it just wraps the familiar HTTP client interface with automatic retries and exponential backoff.

Context

Your Environment

• What arkade version is this?

arkade version

            _             _      
  __ _ _ __| | ____ _  __| | ___ 
 / _` | '__| |/ / _` |/ _` |/ _ \
| (_| | |  |   < (_| | (_| |  __/
 \__,_|_|  |_|\_\__,_|\__,_|\___|

Open Source Marketplace For Developer Tools

Version: 0.8.44
Git Commit: b543e60a68285ce8a147d9b0a6493f573747b8d1

 🐳 arkade needs your support: https://github.com/sponsors/alexellis

[alexellis]

Hi @YuviGold

Thanks for your interest in arkade.

This is the first time hearing of issues with GitHub's releases pages which use S3 underneath.

How often do you get a 503 from GitHub / S3, and do you know why this affects you specifically?

I think that we'd have to hear pain from a bunch more users to consider changing the code.

What company is this for? Perhaps if they wanted to become a GitHub Sponsor it may make it worth us considering.

Alex

[Jasstkn]

FYI: There are some examples in the e2e-tests

[alexellis]

@Jasstkn thanks

Can you have the tests print out headers on error, to see if GitHub is trying to tell us something?

[Jasstkn]

@Jasstkn thanks

Can you have the tests print out headers on error, to see if GitHub is trying to tell us something?

usually it is the server error - 503/500 and so on. Can be the overload on the Github's side or just a network glitch. I think that we may want to introduce this as an option to the arkade get command --retry=2. This can cover some cases for CI/CD pipelines as well.

[matt-cavanagh-bloom]

This is a regular occurence for us, where arkade just fails because k8s.io fails to respond properly and we have to keep retrying...

[rgee0]

This may be something to implement on a per version strategy basis since the version strategy approach split out GH / k8s / GL. The timeouts already differ, which might have mitigated the original issues with GH hosted items

arkade/pkg/get/get.go

Lines 73 to 89 in 97b4b07

	var releaseLocations = map[string]ReleaseLocation{
	GitHubVersionStrategy: {
	Url: "https://github.com/%s/%s/releases/latest",
	Timeout: time.Second * 10,
	Method: http.MethodHead,
	},
	GitLabVersionStrategy: {
	Url: "https://gitlab.com/%s/%s/-/releases/permalink/latest",
	Timeout: time.Second * 5,
	Method: http.MethodHead,
	},
	k8sVersionStrategy: {
	Url: "https://cdn.dl.k8s.io/release/stable.txt",
	Timeout: time.Second * 5,
	Method: http.MethodGet,
	},
	}

[nicgrayson]

@matt-cavanagh-bloom Have you found a solution to this? I'm seeing this exact error frequently

[rgee0]

Does #1208 mitigate the effects at all.

I don't experience the issue as much as some of the folks here, so it would be good to get some input on the PR if it improves matters.

[matt-cavanagh-bloom]

@matt-cavanagh-bloom Have you found a solution to this? I'm seeing this exact error frequently

@nicgrayson sadly no, it was so frequent for us we moved to https://github.com/Azure/setup-kubectl/.