Diagnose and Troubleshoot TCP Resets from Bitbucket Cloud due to Anycast Shift
Platform Notice: Cloud Only - This article only applies to Atlassian products on the cloud platform.
Summary
This knowledge base article will provide a complete explanation of the issue users encounter while troubleshooting TCP resets from Bitbucket cloud due to Anycast Shift, causing network performance degradation as shown in the diagnoses examples.
Bitbucket.org's network IP address 104.192.141.0/24 is an anycast address available globally from 100+' edge locations that leverages an AWS feature known as Global Accelerator, which we launched in October 2020 and announced on our blog.
Sometimes a network configuration or Internet Service Provider's network may have multiple, equal-cost paths toward Bitbucket to choose from: a concept known in networking as Equal Cost Multi-Path (ECMP).
A network device such as a router will almost always forward all the packets within a single TCP Session across the same path and, therefore, to the exact Bitbucket edge location. This is achieved via an algorithm called 'Hash-based Load Balancing' (RFC2992) because the path choice is consistently made based on a hash function of the source and destination IP and ports - consistent during a TCP session.
Environment
Bitbucket Cloud
Diagnosis
Some errors we have observed from Git and HTTP clients due to receiving a TCP Reset mid-operation are shown below as a guide:
These errors are most likely to be caused by other networking issues. They are not definitive signs that anycast shift is occurring.
Example #1:
1
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to bitbucket.org:443Example #2:
1
2
git fetch returned status code 128:
stderr: fatal: unable to access 'https://bitbucket.org/': Network file descriptor is not connectedExample #3:
1
2
3
git fetch returned status code 128:
stdout:
stderr: fatal: unable to access 'https://bitbucket.org/': TCP connection reset by peerExample #4:
1
2
3
4
5
6
7
8
9
10
11
curl -v https://bitbucket.org
About to connect() to bitbucket.org port 443 (#0)
Trying 104.192.141.1...
Connected to bitbucket.org (104.192.141.1) port 443 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
NSS error -5961 (PR_CONNECT_RESET_ERROR)
TCP connection reset by peer
Closing connection 0
curl: (35) TCP connection reset by peerExample #5:
1
2
ssh git@bitbucket.org > /dev/null
packet_write_wait: Connection to 104.192.141.1 port 22: Broken pipeCause
A problem emerges if any of a network or Internet Service Provider'snetwork is equal-cost multi-pathing (ECMP)individual packetswithin a TCP session rather than consistently multi-pathing entire TCP sessions via a hashing function.
In this scenario, different Bitbucket edge locations can inadvertently receive different parts of an existing TCP session. Because these edge locations will not know the TCP session, they will return a TCP Reset - this is an expected behaviour and is consistent with the appropriate IETF RFCs.
Solution
The recommended approach is to identify, along with the Internet Service Provider and the team responsible for networking in your company, any router with more than one equal-cost path to Bitbucket.org's network IP address 104.192.141.0/24. Linux and Unix tools liketracerouteormtrcan help identify the device.
Once identified, a packet capture might be performed to discover if the device is improperly splitting packets within individual flows. With the evidence from the packet capture, it's possible to review product documentation or work with the device vendor to resolve the issue.
How to perform a packet capture
To perform packet capture, the command below can be used:
1sudo tcpdump -i any -nn net 104.192.141.1 -w bitbucket.pcapOnce the command is running, connect to Bitbucket on another window
When the connection attempt completes, you can end the terminal session
A bitbucket.pcap file should be generated at the path where the command was run.
Temporary Workaround
You can either directly modify or modify via automation systems (like Docker, Salt, Ansible, etc.) your operating system's hosts file to direct traffic to Bitbucket's legacy IP addresses - which are not anycast to the internet:
1
2
3
18.205.93.2
18.205.93.0
18.205.93.1Linux/Mac OS
This can be achieved by adding an entry to /etc/hosts on Linux and Unix-based systems as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
sudo vi /etc/hosts
# Add a line like so:
18.205.93.1 bitbucket.org
# :wq to exit vi
# or use any text editor of your choice
The final /etc/hosts will look like this:
$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
18.205.93.1 bitbucket.orgWindows
For the Windows system, the host files can be modified at:
1
c:\Windows\System32\Drivers\etc\hosts Atlassian has no plans to retire these IP addresses, but please be aware that the public DNS answer for Bitbucket.org is the authoritative IP address clients should use. Prolonged usage of a host override risks becoming outdated with the state of Bitbucket's infrastructure and will not be as performant as our anycast edge.
If the above instructions do not assist you with resolving this issue, please raise a support ticket or raise a community support ticket for further assistance with this.
Was this helpful?