Performance regression in S3 URL signing

[akx]

Describe the bug

While profiling our app, I noticed quite a bit of time being spent signing URLs with Boto3/Botocore, so I decided to dig a little deeper, and turns out that there's been pretty significant performance regressions (that aren't helped by awscrt) – namely, current versions of botocore are around 3x slower when calling S3.Client.get_presigned_url.

The below results were generated with this benchmark code on my macOS machine.

boto3	botocore	awscrt	count	mean (ms)	median (ms)	std (ms)	min (ms)	max (ms)	Visual Chart
1.20.54	1.23.54	-	10	680.1	676.1	20.3	658.0	720.9	███████░░░░░░░░░░░░░
1.20.54	1.23.54	0.27.4	10	600.8	592.3	19.3	586.5	648.4	██████░░░░░░░░░░░░░░
1.21.46	1.24.46	-	10	671.7	666.3	21.6	654.9	729.1	███████░░░░░░░░░░░░░
1.21.46	1.24.46	0.27.4	10	590.9	589.0	8.3	582.1	611.0	██████░░░░░░░░░░░░░░
1.22.13	1.25.13	-	10	657.2	656.1	10.0	646.1	678.2	███████░░░░░░░░░░░░░
1.22.13	1.25.13	0.27.4	10	583.9	582.3	8.8	569.8	600.7	██████░░░░░░░░░░░░░░
1.23.10	1.26.10	-	10	649.6	650.0	4.0	640.8	654.0	███████░░░░░░░░░░░░░
1.23.10	1.26.10	0.27.4	10	588.5	587.0	11.5	571.8	614.7	██████░░░░░░░░░░░░░░
1.24.96	1.27.96	-	10	658.7	651.8	27.3	643.8	735.5	███████░░░░░░░░░░░░░
1.24.96	1.27.96	0.27.4	10	593.6	591.6	12.2	582.3	621.1	██████░░░░░░░░░░░░░░
1.25.5	1.28.5	-	10	867.0	868.2	5.7	855.3	876.4	█████████░░░░░░░░░░░
1.25.5	1.28.5	0.27.4	10	791.7	790.6	4.3	786.1	798.0	█████████░░░░░░░░░░░
1.26.165	1.29.165	-	10	866.3	866.8	5.7	858.3	876.4	█████████░░░░░░░░░░░
1.26.165	1.29.165	0.27.4	10	799.9	798.7	10.6	788.5	822.9	█████████░░░░░░░░░░░
1.27.1	1.30.1	-	10	868.2	866.5	10.2	853.4	892.5	█████████░░░░░░░░░░░
1.27.1	1.30.1	0.27.4	10	813.1	804.6	26.9	796.0	885.0	█████████░░░░░░░░░░░
1.28.85	1.31.85	-	10	862.5	860.0	8.9	851.1	876.2	█████████░░░░░░░░░░░
1.28.85	1.31.85	0.27.4	10	803.9	803.9	10.3	789.3	822.7	█████████░░░░░░░░░░░
1.29.7	1.32.7	-	10	1567.2	1562.6	31.6	1534.6	1636.7	█████████████████░░░
1.29.7	1.32.7	0.27.4	10	1747.7	1696.8	173.2	1578.1	2121.9	████████████████████
1.33.13	1.33.13	-	10	1615.0	1606.5	32.2	1589.0	1697.3	██████████████████░░
1.33.13	1.33.13	0.27.4	10	1553.3	1547.6	24.4	1521.6	1607.0	█████████████████░░░
1.34.162	1.34.162	-	10	1622.3	1620.6	14.7	1594.0	1650.8	██████████████████░░
1.34.162	1.34.162	0.27.4	10	1562.7	1564.1	21.5	1534.2	1596.0	█████████████████░░░
1.35.99	1.35.99	-	10	1632.7	1632.2	11.5	1615.9	1652.2	██████████████████░░
1.35.99	1.35.99	0.27.4	10	1588.0	1577.0	30.7	1564.6	1652.2	██████████████████░░
1.36.26	1.36.26	-	10	1648.2	1645.8	26.3	1617.4	1706.7	██████████████████░░
1.36.26	1.36.26	0.27.4	10	1586.0	1578.0	19.0	1569.6	1624.0	██████████████████░░
1.37.38	1.37.38	-	10	1692.2	1688.0	16.5	1676.7	1725.5	███████████████████░
1.37.38	1.37.38	0.27.4	10	1591.2	1589.9	12.8	1571.4	1611.4	██████████████████░░
1.38.46	1.38.46	-	10	1673.3	1667.4	17.4	1650.5	1703.1	███████████████████░
1.38.46	1.38.46	0.27.4	10	1646.0	1624.9	65.5	1602.4	1824.1	██████████████████░░

Of course it's not generally feasible to run old Botocore versions in production.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

URL signing to be as fast as it has been with older versions of Botocore.

For the time being, I wrote a small alternative implementation of S3V4 signing that runs circles around the Botocore code.

Current Behavior

Please see above.

Reproduction Steps

Via https://github.com/akx/botocore-sign-perf:

import boto3

s3_client = boto3.client(
    "s3",
    aws_access_key_id="AKIAI" * 10,
    aws_secret_access_key="nom" * 10,
    region_name="eu-west-1",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
    ),
)

t0 = time.time()
for x in range(10_000):
    s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": "bucket", "Key": str(x)},
        ExpiresIn=120,
    )
t1 = time.time()
sign_time_ms = (t1 - t0) * 1000

Possible Solution

Based on comparing the pyinstrument reports

uv run --with=pyinstrument --with=botocore==1.23.54 --with=boto3 pyinstrument --show='*/boto*' -o old.html main.py
uv run --with=pyinstrument --with=botocore==1.38.46 --with=boto3 pyinstrument --show='*/boto*' -o new.html main.py

it looks like the new versions spend more time in S3._resolve_endpoint_ruleset (that doesn't exist in 1.23.54) than in RequestSigner itself.

I suspect that has to do with evaluating the DSL in the 10k+ line JSON file https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json 😄

Maybe someone should take a critical look at whether EndpointRulesetResolver could cache results, or if it can be sidestepped altogether.

Additional Information/Context

No response

SDK version used

Multiple, see above.

Environment details (OS name and version, etc.)

macOS, but this is reproducible anywhere.

[akx]

Thanks to @hannesj for digging around the JS side of the fence – the AWS JS SDK has a cache for endpoint resolution

Botocore also attempts to cache these:

botocore/botocore/endpoint_provider.py

Lines 707 to 708 in 7ddc232

	@lru_cache_weakref(maxsize=CACHE_SIZE)
	def resolve_endpoint(self, **input_parameters):

However, the effectiveness of that cache is near-zero unless you're signing the exact same key over and over, as Key is one of the parameters being passed to it via

botocore/botocore/regions.py

Lines 502 to 509 in 7ddc232

	provider_params = self._get_provider_params(
	operation_model, call_args, request_context
	)
	LOG.debug(
	f'Calling endpoint provider with parameters: {provider_params}'
	)
	try:
	provider_result = self._provider.resolve_endpoint(

(as it is defined in

botocore/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json

Lines 61 to 65 in 7ddc232

	"Key": {
	"required": false,
	"documentation": "The S3 Key used to send the request. This is an optional parameter that will be set automatically for operations that are scoped to an S3 Key.",
	"type": "String"
	},

).

However, no rules in endpoint-rule-set-1.json actually refer to Key, so it probably shouldn't be defined as a endpoint ruleset parameter. It looks like it would be worth it to figure out why it's ended up there (and/or if there are other extraneous keys) – the bot commit 5d97d5b introducing it doesn't help much 😉

I can confirm that just removing the Key param from that JSON file has a delightful effect already:

$ uv run main.py
{"sign_time_ms": 1649.1360664367676, "python_version": [3, 12, 11, "final", 0], "boto3_version": "1.39.10", "botocore_version": "1.39.10", "awscrt_version": null, "hyperfine_iteration": null}
$ uv pip install -e ../botocore
$ uv run main.py
{"sign_time_ms": 914.9761199951172, "python_version": [3, 12, 11, "final", 0], "boto3_version": "1.39.10", "botocore_version": "1.39.10-devel", "awscrt_version": null, "hyperfine_iteration": null}
$

[adev-code]

Hello @akx, thanks for reaching out. The team is aware and looking at this issue. For any updates, we will reply here but for now this issue will remain open for tracking.

[akx]

@adev-code Thanks. Please see the PR linked above.

[akx]

@adev-code Any movement on this?

Newer versions of Botocore don't seem to be helping.

boto3	botocore	awscrt	count	mean (ms)	median (ms)	std (ms)	min (ms)	max (ms)	Visual Chart
1.39.10	1.39.10	0.28.4	30	1625.1	1622.7	35.1	1570.0	1718.0	██████████████████░░
1.39.10	1.40.71	-	10	1691.8	1671.7	59.3	1614.7	1769.7	███████████████████░
1.39.10	1.40.71	0.28.4	10	1571.8	1558.2	42.5	1535.8	1662.5	█████████████████░░░