Pushing and pulling private Docker image in Bitbucket Cloud Pipelines is timing out
Platform Notice: Cloud Only - This article only applies to Atlassian products on the cloud platform.
Summary
Occasionally, the Docker image push/pull process from a private registry may encounter timeout errors.
1
http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)This documentation aims to provide guidance on troubleshooting such errors.
Diagnosis
The "Client.Timeout exceeded while awaiting headers" is a symptom of several possible causes. In a simple case, it could be due to a private network firewall or a reverse proxy blocking the pipeline requests from accessing the private registry host machine. This error could also have been caused by updated DNS records for hosts of the Docker private registry.
Solution
Please check if the Docker registry host is reachable from pipelines. One can use the ping and telnet commands (to check if the port is open) to test this. If a ping/telnet fails, check your private network firewall or a reverse proxy to allowlist the pipeline IPs if necessary. You can use these Valid IP addresses for Bitbucket Pipelines to allowlist requests made from your build environments.
Eg:
1
2
3
4
5
6
7
8
9
image: praqma/network-multitool
pipelines:
branches:
master:
- step:
name: network stats
script:
- ping <private-docker-registry-host>
- telnet <private-docker-registry-host>Each time when you updateDNS(Domain Name System) records for your docker private registry host, in your domain's zone file, the rest of the Internet must catch up to the changes. This period of catching up is known as propagation. Usually, DNS changes will propagate within a few hours, but it can take up to 48 hours for everything to propagate across the Internet. Many things affect propagation time, including your TTL, your ISP, and your domain's registry.
If there is an error "Client.Timeout exceeded while awaiting headers" while pulling or pushing Docker images, please run the below sample pipeline to check:
Run the nslookup command to check if domain resolution is happening correctly.
Run mtr to analyze the network traffic hop-to-hop using ICMP packets. look for the average round-trip time. The higher the time, the more chances there are that DNS changes have not been fully propagated to the rest of the internet yet. In this case, please wait for some time and rerun the pipeline to check if DNS changes have been fully propagated and that the docker pull/push happens without any timeout errors.
1
2
3
4
5
6
7
8
9
image: praqma/network-multitool
pipelines:
branches:
master:
- step:
name: network stats
script:
- nslookup <private-docker-registry-host>
- mtr <private-docker-registry-host> --tcp -P 443 --report --report-cycles=20Note:One can also use the Google DNS server to test the DNS changes propagation. After configuring Google DNS, check if the MTR reports are normal and one is able to docker pull/push without any errors.
1
2
3
4
5
6
7
8
pipelines:
branches:
master:
- step:
script:
- echo "nameserver 8.8.8.8" > /etc/resolv.conf
- mtr <private-docker-registry-host> --tcp -P 443 --report --report-cycles=20
- <docker pull/push commands here>Was this helpful?