Use an Elasticsearch cluster with Bitbucket Data Center
Bitbucket Data Center requires a connection to a remote Elasticsearch installation to enable code search. Although code search is not critical for high availability, it is possible run a cluster of Elasticsearch nodes to achieve high availability for the Bitbucket's code search index. This page provides guidance on deploying a cluster of Elasticsearch nodes that run behind a load balancer connected to Bitbucket application nodes.
Bitbucket Data Center can have only one remote connection to Elasticsearch for your cluster. This may be a standalone Elasticsearch installation or a clustered installation behind a load balancer.
Bitbucket Data Center allows the use of clustered Elasticsearch installation, however Atlassian Support does not provide assistance for configuring them. Consequently, Atlassian cannot guarantee providing any support for them. If assistance with configuration is required, raise a question in the Atlassian Community.
Standard Bitbucket Data Center component diagram
Before thinking about deploying an Elasticsearch cluster to use with Bitbucket Data Center, it helps to understand what a standard Bitbucket Data Center installation looks like. The page Bitbucket Data Center requirements outlines the detailed requirements.
At minimum, a standard Bitbucket Data Center installation must have these components, each on a dedicated machine, and connected by a high speed LAN (click the links to view detailed requirements for each component):
- Bitbucket applications nodes, all running the same version of Bitbucket Data Center.
- Load balancer that supports session affinity (or, "sticky sessions").
- Shared database, able to take block-level snapshots.
- Shared filesystem, accessible via NFS as a single mount point.
- A remote Elasticsearch node, with a single connection to the Bitbucket application nodes.
Bitbucket Data Center component diagram with an Elasticsearch cluster
This component diagram provides an example configuration of a Bitbucket Data Center installation deployed with a cluster of Elasticsearch nodes behind a load balancer. In this example, we assume you'll run Elasticsearch in its own cluster.
Using Amazon's Elasticsearch service with Bitbucket Data Center
The easiest way to set up and deploy an Elasticsearch cluster for Bitbucket Data Center is to use the Amazon's Elasticsearch service. Atlassian cannot provide direct support for setting up your AWS Elasticsearch cluster, but a good place to start is with Amazon's documentation: Amazon Elasticsearch Service (documentation).
Be sure to use the correct version
Data Center will continue support for both Elasticsearch 2.3 and 5.5, only until the release of Bitbucket 6.0 to allow an upgrade to be planned. At this time, Elasticsearch 2.3 will no longer be supported for Data Center.
Setting up a standalone Elasticsearch cluster with Bitbucket Data Center
If you're not using AWS's Elasticsearch service, and instead are setting up a cluster of Elasticsearch nodes yourself, you generally want to set up a cluster of at least two Elasticsearch nodes. Two nodes allows each node to have a replica on it, meaning that if one node goes down for a short period, you'll still have all of your search results available to you while you repair/replace the other node. If you require more fault tolerance, increase the number of nodes in your cluster.
- Elastic's official documentation for setting up an Elasticsearch cluster: Elasticsearch - Installation (5.5).
Securing your Elasticsearch cluster
After setting up your Elasticsearch cluster, it's important you secure your cluster. You can use a variety of techniques to secure Elasticsearch in a cluster. To get started with basic authentication you'll need to use a plugin with Elasticsearch that will block attempts to access the cluster without a valid username and password. Atlassian created a small plugin named Buckler to do just that.
Secure Elasticsearch with Atlassian's Buckler plugin
For instructions on how to configure Buckler, see the page Install and configure a remote Elasticsearch instance - Step 3: Secure Elasticsearch. Buckler needs to be installed on every node in the cluster. In any case, if you're going to use Amazon's Elasticsearch Service, you must set the AWS region in the bitbucket.properties file to enable request signing.
Secure Elasticsearch with Elastic's Shield plugin
Bitbucket also supports authentication to Elasticsearch through other plugins that provide basic authentication, like Elastic's Shield plugin. This plugin isn't directly supported by Atlassian, but Bitbucket can still interoperate with it via utilising its basic authentication system.
Secure Elasticsearch with Amazon's request signing
Basic authentication and AWS request signing are supported by Bitbucket (AWS request signing Bitbucket 4.10+). AWS Request signing allows you to use the AWS Elasticsearch service to serve as your Elasticsearch cluster. This will allow you to secure your Elasticsearch cluster to only allow requests from the AIM user that the node Bitbucket is running on inside of AWS ec2 has.