要約

According to Install Bitbucket Data Center, for setting up a clustered version of Bitbucket Data Center, you need to establish a load balancer responsible for balancing requests between nodes using the same base URL.

The specificity of the load balancer is that it behaves exactly as a reverse proxy by replacing the headers of the incoming request with its own.

In the case of HTTPS, which operates on the 7th - Application layer of the OSI model, you can solve the issue with headers overwriting by using the option forwardfor or analog for other load balancers. It will make the load balancer send headers with both values, which Bitbucket can correctly write into access and audit logs. 

For SSH, the load balancing is performed on the 4th - Transport layer, which doesn't have any headers so that the application node of Bitbucket can get only the IP address from which it received the request. In our context, such a server will be precisely the load balancer. As a result, all the entries for incoming requests will be written into access and audit logs of Bitbucket with the IP address of the load balancer. It makes it difficult to comprehensively analyze the incoming traffic for SSH. 

The solution came with Proxy protocol, which adjusts the connection string from the load balancer to the application node in Bitbucket with additional information:

PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"

#The following is an example of the proxy protocol line for IPv4
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n

You can find more details on the PROXY protocol implementation in the following sources:

• Official PROXY protocol documentation
• Exploring the PROXY Protocol blog post by Benjamin Boudreau

環境

Bitbucket Data Center 7.20.x and newer
HAProxy 1.5-dev3 and newer

AWS Network Load balancer

ソリューション

Bitbucket part:

Bitbucket Data Center 7.20 and newer have PROXY protocol enabled by default.

However, you can always set it explicitly in $bitbucket_home/shared/bitbucket.properties:

plugin.ssh.haproxy.proxy-enabled=true

That's the only requirement from the Bitbucket side. No further configuration is required. If that doesn't work, the issue originates from the load balancer itself.

Load balancer part - HAProxy scenario

To configure the PROXY protocol with the HAProxy 1.5-dev3 and newer, you need to enable the protocol support (send-proxy) for each cluster node in haproxy.cfg section for SSH backend:

backend bitbucket_ssh_backend
    mode tcp
    balance roundrobin
    server bitbucket01 163.25.93.240:7999 check send-proxy port 7999
    server bitbucket02 192.128.0.220:7999 check send-proxy port 7999
    timeout server 15m

Restart HAProxy after the changes and try the clone once again.

After the changes are successfully applied, you will see double IP addresses in access logs:

CLIENT_IP,LOAD_BALANCER_IP | ssh | i@DX7BUKx631x44878x0 | username | 2022-05-25 10:31:38,622 | SSH - git-upload-pack '/demo/repo.git' | "SSH-2.0-OpenSSH_8.6" | - | - | - | - | - | 17i1kgv |
CLIENT_IP,LOAD_BALANCER_IP | ssh | o@DX7BUKx631x44878x0 | username | 2022-05-25 10:31:41,300 | SSH - git-upload-pack '/demo/repo.git' | "SSH-2.0-OpenSSH_8.6" | 0 | 727 | 22705297 | cache:hit, clone, protocol:2, ssh:user:id:2 | 2678 | 17i1kgv |

`

Load balancer part - AWS Network Load balancer scenario

In the case of AWS, only Network Load Balancer supports PROXY protocol version 2.

To enable proxy protocol v2 using the new console:

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. On the navigation pane, under LOAD BALANCING, choose Target Groups.
3. Choose the name of the target group to open its details page.
4. On the Attributes tab, select Edit.
5. On the Edit attributes page, select Proxy protocol v2
6. Choose Save changes.

To enable proxy protocol v2 using the old console:

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. On the navigation pane, under LOAD BALANCING, choose Target Groups
3. Select the target group
4. Choose Description, Edit attributes.
5. For Proxy protocol v2, choose Enable.
6. Choose Save.

For AWS-CLI please use modify-target-group-attributes withp proxy_protocol_v2.enabled

For more up-to-date information about proxy protocol support for AWS NLB please check https://docs.aws.amazon.com/elasticloadbalancing/latest/network/elb-ng.pdf

For Citrix Netscaler, please check the following guide https://support.citrix.com/article/CTX224265/how-to-configure-netscaler-to-send-proxy-protocol-to-backend-servers (Atlassian doesn't provide any assistance with Netscaler. Please reach official Netscaler support).