Unable to load AWS credentials from any provider in the chain
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Unable to set up an environment with an outbound proxy to both:
Make a connection to the Atlassian Marketplace through the outbound proxy.
Use the "nonProxyHosts" JVM argument to connect to an AWS Elasticsearch service without proxying the request.
See here for additional information on using an outbound proxy with Bitbucket Data Center.
Environment
Bitbucket Server/Data Center
AWS Elasticsearch
Outgoing proxy
Using IAM roles for authentication between servers
Diagnosis
The following shows up in the logs when you are able to connect to the Marketplace but not Elasticsearch:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
2020-11-02 14:43:41,299 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService An error was encountered while checking or creating the mapping in Elasticsearch
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unable to check whether a valid mapping exists in Elasticsearch
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.lambda$isMappingPresent$9(IndexingSynchronizationService.java:268)
at io.atlassian.fugue.Either$Left.fold(Either.java:478)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:267)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:109)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:82)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:82)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:64)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consumeSingle(Observables.java:92)
... 18 common frames omitted
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at vc.inreach.aws.request.AWSSigner.getSignedHeaders(AWSSigner.java:91)
at vc.inreach.aws.request.AWSSigningRequestInterceptor.process(AWSSigningRequestInterceptor.java:29)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:124)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at com.atlassian.elasticsearch.client.apache.httpclient.InstrumentedNHttpClientBuilder$1.execute(InstrumentedNHttpClientBuilder.java:93)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:74)
at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor.execute(ApacheRequestExecutor.java:132)
at com.atlassian.elasticsearch.client.internal.InternalClient.execute(InternalClient.java:29)
at com.atlassian.elasticsearch.client.Client.execute(Client.java:38)
at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.lambda$execute$1(DefaultConfigurableElasticsearchClient.java:34)
at java.util.Optional.map(Optional.java:215)
at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.execute(DefaultConfigurableElasticsearchClient.java:34)
at com.atlassian.bitbucket.internal.search.indexing.administration.DefaultIndexAdministrationService.codeSearchMappingExists(DefaultIndexAdministrationService.java:55)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:266)
... 17 common frames omittedCause
There is a metadata IP address that Bitbucket needs to be able to hit in addition to the Elasticsearch URL, as per this AWS guide.
This error indicates that an outbound proxy or some other network-level restriction is preventing access to this endpoint.
Solution
Ensure that Bitbucket is able to send traffic to the IP address 169.254.169.254, and add this IP address to the -Dhttp.nonProxyHosts list if using an outbound proxy.
Was this helpful?