Reduce API call limit error

[kAIto47802]

Motivation

Currently, the load_module function often gets stuck due to the rate limit for the GitHub REST API.
The rate limit for unauthenticated requests is 60 requests per hour, which is very strict.

The error message looks like this:

Request GET /repos/optuna/optunahub-registry failed with 403: rate limit exceeded
INFO:github.GithubRetry:Request GET /repos/optuna/optunahub-registry failed with 403: rate limit exceeded
Setting next backoff to 3570.799172s
INFO:github.GithubRetry:Setting next backoff to 3570.799172s

Description of the changes

• Use git command when available to avoid the rate limit.

Benchmarking results

I benchmarked:

• how many times samplers/auto_sampler can be loaded within one minute; and
• the time it takes to load once.

Setups

The benchmark code I used here is as follows:

!pip install optuna
# or
# !pip install git+https://github.com/kAIto47802/optunahub@reduce-api-call-limit

!pip install -q cmaes


import optuna
import optunahub
import time

start = time.time()
for i in range(500):
    mod = optunahub.load_module("samplers/auto_sampler", force_reload=True)
    print(i)
    end = time.time()
    print("end - start:", end - start)
    if end - start > 60:
        break

print("end - start:", end - start)
print("Number of iterations:", i)

The benchmarks were run five times for both the original and the new implementation.

I used a different Google Colaboratory notebook for each benchmarking run to ensure a different IP address was used every time. (Also, I confirmed the attached VM's IP address is certainly different by using the command curl ifconfig.me.)

Results

The average number of times load_module was successfully executed without hitting the API rate limit is as follows:

Original	This PR
5.2	-

With the new implementation, running continuous downloads for one minute does not hit a rate limit across all five runs. (Therefore, it is indicated with a dash (–) in the table.)

The average time for downloading once is as follows:

Original	This PR
4.50 ± 0.45 sec	0.49 ± 0.12 sec

$m \pm \sigma$ indicates the average and the standard error over the five runs, respectively.

[codecov]

Codecov Report

Attention: Patch coverage is 84.44444% with 7 lines in your changes missing coverage. Please review.

Project coverage is 83.16%. Comparing base (6728c63) to head (855e4c9).
Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
optunahub/hub.py	84.44%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #93      +/-   ##
==========================================
- Coverage   89.08%   83.16%   -5.92%     
==========================================
  Files           9        9              
  Lines         174      196      +22     
==========================================
+ Hits          155      163       +8     
- Misses         19       33      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[c-bata]

@not522 Could you review this PR?

[not522]

Thank you for your PR. Could you check my comments?

[HideakiImamura]

@gen740 Could you review this PR?

[nabenabe0928]

I checked the PR except for _download_via_git:)

[nabenabe0928]

Suggested change

	If ``auth`` is :obj:`None` and the ``git`` command is available, this should be the base URI for the remote repository.
	In this case, specifying ``git@github.com`` allows access to private/internal repositories via SSH.
	Otherwise, this should be the base URL for the GitHub API.
	If ``auth`` is :obj:`None` and the ``git`` command is available, this should be the
	base URI, e.g., ``github.enterprise.com`` and ``gitlab.com``, for the remote
	repository.
	In this case, specifying ``git@github.com`` allows access to private/internal repositories via SSH.
	Otherwise, this should be the base URL for the GitHub API.

[kAIto47802]

Thank you for your suggestion! However, the examples you suggested are actually the opposite, and the URL is incorrect---https://api.github.com, https://github.enterprise.com/api/v3, and https://gitlab.com/api/v4 are the examples for the GitHub API. Also, the endpoints for GitHub Enterprise, GitLab, and other services are not limited to this, since they support custom domains.¹²
So let me update the docstring with modifications.

Footnotes

1. https://docs.gitlab.com/user/ssh/ ↩

2. https://docs.gitlab.com/api/rest/ ↩

[nabenabe0928]

I confirmed that the change worked nicely!

$ pip install git+https://github.com/kAIto47802/optunahub.git@reduce-api-call-limit
$ python -c "import optunahub; optunahub.load_module("samplers/ctpe")"
$ ls .cache/optunahub/github.com/optuna/optunahub-registry/main/package/samplers/
ctpe

[nabenabe0928]

LGTM!

[nabenabe0928]

Unit tests also look good to me!

[not522]

Thank you for update! It's almost LGTM. Please check my comment for _download_via_github_api.

[nabenabe0928]

I merged the main branch and verified that the merged version also works nicely:)

[nabenabe0928]

Thank you for your suggestion! However, the examples you suggested are actually the opposite, and the URL is incorrect---https://api.github.com, https://github.enterprise.com/api/v3, and https://gitlab.com/api/v4 are the examples for the GitHub API. Also, the endpoints for GitHub Enterprise, GitLab, and other services are not limited to this, since they support custom domains.
So let me update the docstring with modifications.

@kAIto47802
Btw, could you enhance the doc-string according to what you mentioned?
The reason why I would like to add some more examples such as github.enterprise.com is to avoid unintended changes in the future by some developers who could not know the intended usages.
More specifically, the documentation string does not really specify any URLs other than api.github.com and github.com, potentially making some people wonder why some processes in the function are that complicated.

[kAIto47802]

@nabenabe0928
Thank you for your comment. I've updated the docstring.
Since GitHub Enterprise allows custom domains, it is a little bit difficult to show a concrete example formally. Therefore, I used GitLab for the docstring, which includes custom domain examples using example.com in its official document ¹².

Footnotes

1. https://docs.gitlab.com/user/ssh/ ↩

2. https://docs.gitlab.com/api/rest/ ↩

[not522]

LGTM!
@nabenabe0928 Please merge this PR after your concern is resolved.