boto3.Session ignores AWS_PROFILE when there are other credentials set in the environment

[primerano]

Describe the bug

boto3.Session() looks at AWS_PROFILE but ignores it if there are credentials specified in the environment.

Using boto3.Session(profile_name=os.environ['AWS_PROFILE']) fixes the issue but this should not be necessary.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expect boto3.Session() to use the credentials provided by the SSO profile specified via AWS_PROFILE instead of the inline credentials.

Current Behavior

If the following variables are set the AWS_PROFILE is not used even though boto3.Session().profile_name will show what is set in AWS_PROFILE

AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN

Reproduction Steps

If I set AWS_PROFILE=FOO, startup python and run

import boto3
session = boto3.Session()
print(session.profile_name)

It will print FOO and I will see the details of my FOO session in session.client('sts').get_caller_identity() as expect.

Note: the FOO profile is an AWS SSO profile.

Now if I paste in credentials for a different user

export AWS_ACCESS_KEY_ID="the key"
export AWS_SECRET_ACCESS_KEY="the secret"
export AWS_SESSION_TOKEN="the session"

and re-run

AWS_PROFILE=FOO python

and in python run

import boto3
session = boto3.Session()
print(session.profile_name)

it will print "FOO" but this profile will not be used by boto3. it uses the credentials from the 3 AWS variables instead.

This is confirmed by running session.client('sts').get_caller_identity()

Now, if I instead create the session with the profile name

session = boto3.Session(profile_name=os.environ['AWS_PROFILE'])

it will now use the profile instead of the AWS_ACCESS_KEY_ID/SECRET/SESSION.

Considering that boto3.Session().profile_name correctly picks up the name from AWS_PROFILE it would seem that it should then use those credentials.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

aws-cli/2.30.2, boto3/1.42.6 botocore/1.42.6

Environment details (OS name and version, etc.)

al2023 and ubuntu 22

[primerano]

Here is a suggested fix..

In session.py init, the following line correctly picks up the AWS_PROFILE and sets self._session.profile=AWS_PROFILE

self._session = botocore.session.get_session()

But set_config_variable is only called for this profile if it was a passed in as a parameter.

        if profile_name is not None:
            self._session.set_config_variable('profile', profile_name)

I recommend adding this elif.

        if profile_name is not None:
            self._session.set_config_variable('profile', profile_name)
        elif  self._session.profile is not None:
            self._session.set_config_variable('profile', self._session.profile)

profile_name parameter is highest priority, If not set it honors AWS_PROFILE or AWS_DEFAULT_PROFILE setting.

[adev-code]

Hello @primerano, thanks for reaching out. The order in which Boto3 searches for credentials is specified in this specific order, where Environment variables (No. 3) take precedence over profile-based credential sources like Shared credential file (No. 7) and AWS config file (No. 9).

When environment variables are set, Boto3 uses those credentials and stops searching - even if AWS_PROFILE is also set. To explicitly use a specific profile, use boto3.Session(profile_name=os.environ['AWS_PROFILE']) as you have mentioned.

Please take note that AWS_PROFILE and Environment variable credentials are separate credential sources. AWS_PROFILE looks at the credentials and config file as mentioned here.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[primerano]

Hi @adev-code, sorry for the slow response here.

Thank you for the information you provided. Shouldn't the boto3 code behave the same way as the AWS CLI?

In the CLI if you specify the --profile parameter, it wins over any environment variables but if you set AWS_PROFILE it will win over the other KEY/SECRET/SESSION variables. Or at least that is what I was seeing when using AWS SSO to log into my profile. I'll double check this in a week when I am back in the office. Maybe it is specific to SSO sessions.

That said, this page also agrees with what you have been saying.

https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#id1

"If AWS_PROFILE environment variable is set and the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, then the credentials provided by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override the credentials located in the profile provided by AWS_PROFILE"

Thank you for your insights!

[primerano]

Sigh.. I was wrong. the AWS KEY/SECRET/SESSSION variables do win over AWS_PROFILE or AWS_DEFAULT_PROFILE.

• aws sts get-caller-identity returns the identity associated with the KEY/SECRET/SESSSION variables
• AWS_PROFILE=FOO aws sts get-caller-identity returns the identity associated with the KEY/SECRET/SESSSION variables if set, otherwise it will use the identity associated with the AWS_PROFILE
• aws sts get-caller-identity --profile FOO returns the identity associated the FOO profile and will not use the identity in the KEY/SECRET/SESSSION variables.