Troubleshooting Bitbucket Cloud MTU/MSS issues
To protect your data, and ensure you can access Bitbucket Cloud consistently, we've added a distributed denial-of-service (DDoS) protection solution.
The solution sends clean traffic to Bitbucket Cloud via a tunnel which adds an extra 24 bytes to all packets. Because this is the case, the maximum packet size supported by the Bitbucket Cloud platform is 1476 bytes.
Our solution will not forward individual IP packets larger than 1476 bytes to Bitbucket Cloud. In some instances, a device on your path to Bitbucket Cloud may incorrectly adjust the MSS we advertise to a higher value than what we support. When this occurs, you may experience issues connecting to the platform over HTTPS and/or SSH.
This packet size acceptance change should not be an issue because:
- Atlassian's cloud services are primarily served over TCP which supports a mechanism that allows Atlassian to tell clients what their maximum packet size should be when sending traffic to Atlassian (it's the MSS or maximum segment size option in the TCP header. Generally the relationship between MTU and MSS is that MTU = MSS + 40). Doing this is extremely common place and it covers the overwhelming majority of users and services. Atlassian adjusts the MSS it sends to clients to a value of 1436.
- If, for some reason, a client still sends Atlassian a packet that is larger than 1476 bytes, one of 2 things will happen:
- If the client has allowed the packet to be fragmented (df-bit not set), we will split up the packet in to 2 packets, and send it to Atlassian. This may cause a minor impact on performances but should generally work without noticeable impact to end users.
- If the client has not allowed the packet to be fragmented (this is often the case for SSH and HTTPS), we will send an ICMP message back to the client to inform them that the packet was dropped because it was too large. This is part of a process call Path MTU Discovery which allows the client to resend the information in smaller packets. This process is unfortunately not extremely reliable as it is quite common for firewalls to block those ICMP messages on the way back to the client.
If all those mechanisms fail for one reason or another, the clients will experience issues where their connection to Atlassian will get established, but will start stalling when they try to send large packets, and eventually time out and fail.
How to determine the MSS value received by your computer when communicating with Bitbucket
For Linux or OSX users
Start a packet capture by entering the command below in a terminal. It will display only the packets that come from Bitbucket Cloud and carry the MSS option we need to look at:
sudo tcpdump -i `netstat -nr |grep -E '(^[dD]efault|^0\.0\.0\.0)'|sed 's/.* //g'` -nn src net 126.96.36.199/24 and tcp=18
Attempt to connect to Bitbucket Cloud and reproduce the failure you're experiencing.
- Stop the capture by pressing 'ctrl-c'.
If the result shows the "mss" as 1436, you're not affected by an MTU issue. The result will look similar to the following example:
mybox:~ user1$ sudo tcpdump -i `netstat -nr |grep -E '(^[dD]efault|^0\.0\.0\.0)'|sed 's/.* //g'` -nn src net 188.8.131.52/24 and tcp=18 Password: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:14:55.460704 IP 184.108.40.206.443 > 172.22.57.225.56524: Flags [S.], seq 4152693889, ack 3904719392, win 28480, options [mss 1436,sackOK,TS val 301417943 ecr 797338616,nop,wscale 7], length 0 10:14:55.465935 IP 220.127.116.11.443 > 172.22.57.225.56519: Flags [S.], seq 2774522829, ack 4278176036, win 28480, options [mss 1436,sackOK,TS val 301417944 ecr 797338616,nop,wscale 7], length 0
If the result shows the 'mss' as a value higher than 1436, then a device on your path is adjusting the 'mss' incorrectly. The result will look similar to the following example:
mybox:~ user1$ sudo tcpdump -i `netstat -nr |grep -E '(^[dD]efault|^0\.0\.0\.0)'|sed 's/.* //g'` -nn src net 18.104.22.168/24 and tcp=18 Password: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes 07:30:07.446633 IP 22.214.171.124.22 > 192.168.0.103.40408: Flags [S.], seq 2581941579, ack 3766129149, win 28480, options [mss 1440,sackOK,TS val 202645932 ecr 3326411,nop,wscale 7], length 0
- Download and install Wireshark https://www.wireshark.org/#download (or use the portable app version if you prefer not to install it).
Apply the capture filter:
src net 126.96.36.199/24 and tcp=18
- Start the capture by double-clicking on the network connection used to connect to Bitbucket (most likely "Local Area Connection"):
- Try to connect to Bitbucket.
- Look for the MSS value in the captured packets:
- f the result shows the "mss" as 1436 or lower, you're not affected by an MTU issue. If the result shows the 'mss' as a value higher than 1436, then a device on your path is adjusting the 'mss' incorrectly.
Solutions and workarounds
It is very likely that the incorrect MSS adjustment is happening on the device connecting you to the internet, or by your ISP. To correct this talk to the team who manages your network, or your ISP. You'll need to tell them that a device on your internet connection is incorrectly adjusting the TCP MSS upwards to a value higher than 1436. To solve this they should either lower it to 1436 or switch to a different software/device that does not exhibit this behavior.
Anything that will drop the MTU of your connection to 1476 or below should solve this. The simplest way is to adjust the MTU of your workstation to 1476 or lower. The following workarounds require admin rights to your operating system.
Run the following command from a terminal:
sudo ifconfig `netstat -nr |grep -E '(^[dD]efault|^0\.0\.0\.0)'|sed 's/.* //g'` mtu 1476
This change won't be permanent - the procedure to make it permanent across a reboot will differ based on your OS and distribution.
- Click the Windows button on the task bar.
- Click All Programs.
- Click Accessories.
- Right-click on Command Prompt and click Run as administrator.
- If prompted click the Allow button.
- Type netsh interface ipv4 show subinterface
- Enter を押します。
- You will see a list of network interfaces.
- Type netsh interface ipv4 set subinterface “Local Area Connection” mtu=1476 store=persistent
You should replace Local Area Connection with the name that appeared in the “Interface” column from steps 1-3.
- Enter を押します。
- Restart you computer and then test again.
OS Agnostic workaround
Try using a VPN Client - VPN clients will typically drop the MTU of your connection since they need to encapsulate and encrypt your traffic. Enabling a VPN for the traffic going to Bitbucket Cloud may solve this issue.