Troubleshooting steps for Bitbucket Server Code Search

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

目的

This page contains the troubleshooting steps that can be performed to understand the reasons leading to the Bitbucket Server Code Search functionality not being available.

The steps below apply for the Embedded Elasticsearch. If you have "Bitbucket Data Center" it means that you do not have Embedded Elasticsearch but your own self-managed Elasticsearch instance and therefore some of the troubleshooting steps below might not apply to your environment.

Troubleshooting steps

 

Check that Elasticsearch is starting with the atlbitbucket user

Ideally you want Elasticsearch through the services that we deploy when you set your environment up by using the installer.

On Linux, you should run the following commands as root (the service will make sure to change into the atlbitbucket user):

# service atlbitbucket_search status
# service atlbitbucket_search start
# service atlbitbucket_search stop

Bitbucket Server 5.0

 Starting with Bitbucket Server 5.0+, there will no longer be a atlbitbucket_search service. Elasticsearch will be started as a result of starting the atlbitbucket service.

Check the username/password for Elasticsearch

Check if you're hitting this: Test button in Elasticsearch for Bitbucket server results in the access denied

Make sure that you did not update the Elasticsearch username and password in the Bitbucket Server user interface, if this is the case you'll need to reset them by following these steps:

  1. locate the $BITBUCKET_HOME/shared/search/buckler/buckler.yml file
  2. Retrieve the username (auth.basic.username attribute) and the password (auth.basic.password attribute) and update the ones available to Bitbucket Server in the user interface

If you did not update them, then there is no need to change anything.

Configure details of your Elasticsearch instance within Bitbucket Server

To configure the details of your Elasticsearch instance within the Bitbucket Server UI, go to the Administration settings page, then click Server Settings. At the bottom of the page is where you can configure the details of your Elasticsearch instance.

Important:

Another way to configure connection settings is via bitbucket.properties file (resides in $BITBUCKET_HOME/shared).

If a parameter is set in the properties file, it cannot be edited later from the admin UI. Any changes that need to be made to the Elasticsearch configuration must be made within the bitbucket.properties file (which requires an application restart to take effect).

  1. In the $BITBUCKET_HOME directory, create a new directory called shared.
  2. In the $BITBUCKET_HOME/shared directory, create a text file named bitbucket.properties.
  3. Add these Elasticsearch properties to the file.

    プロパティ
    Parameter name for properties file
    URL plugin.search.elasticsearch.baseurl
    ユーザ名 plugin.search.elasticsearch.username
    パスワード plugin.search.elasticsearch.password

 

Check the Elasticsearch is writing to its own logs

In a very specific situation, we saw Elasticsearch completely hang and not write in the logs or bind to its port as defined in BITBUCKET_HOME/shared/search/elasticsearch.yml

The symptoms are:

  • tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log
    • Nothing happens
  • ps -ef | grep elastic
    • reveals Elasticsearch is up and running
  • netstat -tulnp 
    • as atlbitbucket user does not show the port 7992 on listen. You can't see the PID of Elasticsearch listened
  • telnet localhost 7992
    • Connection is refused

If you run as atlbitbucket user:

jstack $ELASTIC_SEARCH_PID > elasticsearch_threads.`date +%s`.txt

You will get the following thread dump as a result:

"main" #1 prio=5 os_prio=0 tid=0x00007f4a6800a800 nid=0x6710 runnable [0x00007f4a6e25a000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
	at sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
	at sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
	at sun.nio.fs.UnixFileStore.devFor(UnixFileStore.java:55)
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:70)
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:48)
	at sun.nio.fs.LinuxFileSystem.getFileStore(LinuxFileSystem.java:112)
	at sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(UnixFileSystem.java:213)
	at sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(UnixFileSystem.java:224)
	- locked <0x00000000c0b3c088> (a sun.nio.fs.UnixFileSystem$FileStoreIterator)
	at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:515)
	at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:459)
	at org.apache.lucene.util.IOUtils.spins(IOUtils.java:448)
	at org.elasticsearch.env.ESFileStore.<init>(ESFileStore.java:57)
	at org.elasticsearch.env.Environment.<clinit>(Environment.java:90)
	at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:81)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
	at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

According to the discussion below:

A NFS mount that is unreacheable, for example, could cause such scenario. As the Elasticsearch dev explains, Elasticsearch checks all mount points on your machine during startup.

If you facing that it also means:

That a df on your system won't return the shell line; it will hang indefinitely.

ソリューション

You really need to check which one is the failing mount point. After that, restarting ES works.

Enable DEBUG level logs

If additional troubleshooting is required and the debug log level of the Bitbucket Server to Elasticsearch is required, the following command will enable it:

curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug

The additional log lines will be added to the atlassian-bitbucket.log file. Refer to the Bitbucket Server debug logging page for more information and alternative methods to enable debug logging.

 

Check the Elasticsearch and Bitbucket Server logs for error messages

You might have some obvious log messages that could help you address the issue.

These are logs you should check:

  • $BITBUCKET_HOME/log/atlassian-bitbucket.log
  • $BITBUCKET_HOME/log/search/bitbucket_search.log
Check the Elasticsearch service scripts are installed

Bitbucket Server 5.0

 Starting with Bitbucket Server 5.0+, there will no longer be a atlbitbucket_search service. Elasticsearch will be started as a result of starting the atlbitbucket service.

Verify if the atlbitbucket_search has been created:

ll /etc/init.d/atlbitbucket_search

If the file exists, but the process is not up and running, perform the following command as root:

# service atlbitbucket_search start

Note that the service to stop/start Elasticsearch is different to the service to stop/start Bitbucket server. More details are explained on Starting and stopping Bitbucket Server.

Bitbucket Server

# service atlbitbucket status
# service atlbitbucket stop
# service atlbitbucket start

Elasticsearch

# service atlbitbucket_search status
# service atlbitbucket_search start
# service atlbitbucket_search stop


Check that the Elasticsearch configuration files are present

 Check that the following files are present under BITBUCKET_HOME/shared/search:

├── buckler
│   └── buckler.yml
├── elasticsearch.yml
└── logging.yml

If these files are missing, do the following:

  1. Stop atlbitbucket_search
  2. Copy the contents of BITBUCKET_INSTALL/elasticsearch/config-template into BITBUCKET_HOME/shared/search
  3. Start the atlbitbucket_search service

    # service atlbitbucket_search status
    # service atlbitbucket_search start
    # service atlbitbucket_search stop

    Bitbucket Server 5.0

     Starting with Bitbucket Server 5.0+, there will no longer be a atlbitbucket_search service. Elasticsearch will be started as a result of starting the atlbitbucket service.

  4. Follow the steps in the "Check the username/password for Elasticsearch" to retrieve the password from the buckler.yml file and add it to the Search settings.

This unexpected behaviour is currently under investigation and tracked in our public facing issue tracker: BSERV-9056 - Getting issue details... STATUS

Check that you can connect to port 7992 locally

The user/password required by Elasticsearch can be obtained by following the steps on the expand box above "Check username/password for Elasticsearch".

The steps below should be run from the server where your Bitbucket Server/Elasticsearch instance are running. The reason why we reinforce it has to be a local connection is due to the configuration that we ship for Elasticsearch on $BITBUCKET_HOME/shared/search/elasticsearch.yml.

You won't be able to browse Elasticsearch from a remote machine based on the Elasticsearch configuration we ship. If you're installing your own remote Elasticsearch, you should configure the network.host: 0.0.0.0 on <Elasticsearch_Config_dir>/elasticsearch.yml.

Please refer to Elasticsearch documentation to understand why: Special values for network.host

Click to see the configuration we ship with the bundled Elasticsearch on elasticsearch.yml...
[root@mybbs search]# cat elasticsearch.yml
cluster.name: bitbucket_search
node:
  name: bitbucket_bundled
  local: true

network.host: _localhost_

discovery.zen.ping.multicast.enabled: false

action.auto_create_index: false
index.mapper.dynamic: false
http.port: 7992

 

Access the Elasticsearch user interface at the Elasticsearch bundled URL. From a browser window running locally on the server where Bitbucket is hosted, hit the following URL: http://localhost:7992

If the following login dialog is displayed, Elasticsearch is up and running:

 

 

If you don't have a local browser on the server that is running Bitbucket Server, you should also be able to run the following command successfully:

$ curl -u username:password http://localhost:7992
{
  "name" : "bitbucket_bundled",
  "cluster_name" : "bitbucket_search",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

または

$ telnet localhost 7992
Connected to localhost.
Escape character is '^]'.
 
 $ netstat -an | grep 7992

 

メモ:

  • As a result on your browser, you should expect to see something similar to that on your browser:
{
  "name" : "bitbucket_bundled",
  "cluster_name" : "bitbucket_search",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
Check the Elasticsearch process is up and running
pgrep -fl elasticsearch

or its equivalent

ps -ef | grep elasticsearch

This would reveal if your server is listening on port 7992 on Linux and which PID is doing it:

$ netstat -tulnp

最終更新日 2017 年 5 月 2 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.