Pipelines gives an error - "An error occurred (AccessDeniedException) when calling the operation" despite providing authorization via OIDC or "aws configure set"

Summary

This document covers a scenario where AWS API calls made from bitbucket pipelines might fail while authenticating Pipelines with AWS via OIDC or using "aws configure set" in the script. The error you will most likely see is -

Pre-Requisite

You should configure AWS authentication in Pipelines via one of the two methods below -

• via OIDC. You can find steps for the configuration in our documentation

• via AWS CLI command - "aws configure set". To set it up this way, please refer to the AWS CLI documentation

Cause

AWS has a precedence for passing credentials which you can find in the AWS documentation. They relevant ones also listed below -

1. Command line options – Overrides settings in any other location

2. Environment variables – You can store values in your system's environment variables.

3. Assume role – Assume the permissions of an IAM role through configuration or the aws sts assume-role

4. command.

5. Assume role with web identity – Assume the permissions of an IAM role using web identity through configuration or the aws sts assume-role

6. command.

7. AWS IAM Identity Center – The IAM Identity Center configuration settings are stored in the config file. Credentials are authenticated when you run the aws configure sso command. The config file is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME\.aws\config on Windows.

8. Credentials file – The credentials and config file are updated when you run the command aws configure. The credentials file is located at ~/.aws/credentials on Linux or macOS, or at C:\Users\USERNAME\.aws\credentials on Windows.

9. Custom process – Get your credentials from an external source.

10. Configuration file – The credentials and config file are updated when you run the command aws configure. The config file is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME\.aws\config on Windows.

The "aws configure set" is 6th in the precedence order since the command sets the values in the credentials and config file. OIDC is 4th in the precedence under "Assume Role with Web Identity". Hence, OIDC will override aws configure set. Additionally, Environment Variables and Command Line options will override both options.

However, we cannot specify credentials such as AWS_ACCESS_KEY_ID via Command Line Options as mentioned here. Hence, the only way to override OIDC and credential file is via Environment Variables. In Bitbucket Pipelines, variables such as workspace variables, deployment variables, repository variables will be treated as environment variables. The presence of a value in the variables that AWS will treat as a key or a secret will overwrite OIDC and credentials file.

Solution

We need to check if any variables are overriding the configuration we have set via OIDC or Credentials file. Check workspace variables, repository variables or deployment variables for one of these values - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The presence of these variables will be treated as the access key or the secret access by AWS which overrides any other authorization we provide. Deleting this variable or changing it to your preferred user's credentials will resolve the problem.