要約

When attempting a code search in Bitbucket, the "Search is currently unavailable" error is displayed in the user interface.

Other errors may be seen in the logs as a response to the search failure. In some cases, this can lead to a high CPU for the bundled Search server Java process.

環境

• Bitbucket 5.x, 6.x, 7.x, 8.x  and 9.x Server/DC

診断

Errors in Log Files

There are many different messages to show that the Search server index is corrupt, somehow. Some of the errors seen so far are outlined below:

Errors in the atlassian-bitbucket.log:

404 errors

Unexpected response code from Elasticsearch: 404

com.atlassian.bitbucket.search.indexer.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=43) the HTTP Status code is: 404

A 404 status code indicates that the target resource could not be found.

503 errors

ERROR [http-nio-7990-exec-16] X182486 <session id> <username> <ip address> "POST /rest/search/latest/search HTTP/1.1" c.a.b.s.internal.rest.SearchResource Unexpected response code from Elasticsearch: 503

A 503 status code indicates that the target resource is unavailable.

400 errors

ERROR [http-nio-7990-exec-2746] user *1U2QK8x795x992313x2 c6fz54 10.10.10.5,10.10.10.13 "POST /rest/search/latest/search HTTP/1.1" c.a.b.i.s.search.rest.SearchResource Unexpected response code from Elasticsearch: 400

A 400 is a Bad Request and is typically associated with a misconfigured instance.

You are not permitted to access this resource

ERROR [https-jsse-nio-8443-exec-26] n0200293 <session id> <username> <ip address> "POST /rest/search/latest/search HTTP/1.1" c.a.b.i.s.search.rest.SearchResource Failed to process search request 'RestSearchRequest{entities={code=PagingInfo{start=0, limit=0}}, limits=Limits{primary=25, secondary=10}, query='<search keyword>'}' - error: You are not permitted to access this resource

Errors may also appear in the bitbucket_search.log:

IndexxNotFoundException - The index does not exist.

[bitbucket-search] IndexNotFoundException[no such index]

Unable to find a field mapper for fieldFoundException - Incorrect mapping

Caused by: ElasticsearchException[Unable to find a field mapper for field [quickSearchProjectName.length]. No 'missing' value defined.]

Failed to load metadata - Unable to read the index data. A sign of corruption.

[2020-12-02T10:03:52,001][ERROR][o.e.b.Bootstrap          ] [bitbucket_bundled]Exception
org.elasticsearch.ElasticsearchException: failed to load metadata
...
Failed to load metadata Caused by: java.io.IOException: failed to find metadata for existing index bitbucket-index-state [location: IAdbCfIHT4yFBHhlsSk5VQ, generation: 5]

IndexShardRecoveryException - Unable to find or read the Search server shard files. A sign of corruption.

[2016-11-23 16:10:27,080][WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][3]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
...
Caused by: [bitbucket-search-v1][[bitbucket-search-v1][3]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.engine.InternalEngine.< int>(InternalEngine.java:155)
...
Caused by: java.nio.file.NoSuchFileException: /var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
... 

IndexShardRecoveryException[failed to fetch index version after copying it over];  Unable to fetch indexes. A sign of corruption. The possible cause could be moving indexes(located in the <$BITBUCKET_HOME>/shared/search/data/bitbucket_search) from one server to another manually, without using Backup Client

From bitbucket_search.log,

[WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][4]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][4]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [_21.si, write.lock, _fr.fdt, _21.nvd, _fr.nvd, _fr_Lucene50_0.tim, _fr_Lucene54_0.dvd, _21.nvm, _21_Lucene50_0.doc, _21.fdt, _fr.si, _21_Lucene50_0.tim, _21.fnm, _fr.fdx, _fr_Lucene54_0.dvm, _21_Lucene54_0.dvm, _fr.fnm, _21.fdx, _fr_Lucene50_0.tip, _fr_Lucene50_0.pos, _21_Lucene50_0.pos, _21_Lucene54_0.dvd, _fr_Lucene50_0.doc, _21_Lucene50_0.tip, _fr.nvm]]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index),niofs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index))): files: [_21.fdt, _21.fdx, _21.fnm, _21.nvd, _21.nvm, _21.si, _21_Lucene50_0.doc, _21_Lucene50_0.pos, _21_Lucene50_0.tim, _21_Lucene50_0.tip, _21_Lucene54_0.dvd, _21_Lucene54_0.dvm, _fr.fdt, _fr.fdx, _fr.fnm, _fr.nvd, _fr.nvm, _fr.si, _fr_Lucene50_0.doc, _fr_Lucene50_0.pos, _fr_Lucene50_0.tim, _fr_Lucene50_0.tip, _fr_Lucene54_0.dvd, _fr_Lucene54_0.dvm, write.lock]];
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)

From atlassian-bitbucket.log,

ERROR [search-indexing:thread-1]  c.a.b.s.i.i.e.DefaultIndexEventWorker Indexing - Failed for project <project_name> (id: 489) with error: Elasticsearch returned 503 (service unavailable) for the request
com.atlassian.bitbucket.search.indexer.exceptions.ServiceUnavailableException: Elasticsearch returned 503 (service unavailable) for the request
	at com.atlassian.bitbucket.search.indexer.internal.util.ElasticsearchUtil.lambda$executeHandleUnavailable$0(ElasticsearchUtil.java:42) ~[bitbucket-search-index-2.1.4.jar:na]
	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54) [bitbucket-search-index-2.1.4.jar:na]

Check the content on the filesystem

List the search index files to see if the index has been corrupted or deleted, with one of the following two commands:

tree -L 8 <bitbucket-home>/shared/search/data

 The tree command, if installed, lists the contents of directories in a tree format. The -L argument displays the max depth of the tree.

If tree command is not available, you can use the following command:

ls -R <bitbucket-home>/shared/search/data

 The ls command lists directory content and the -R argument makes it recursive, which lists the subdirectory's contents.

The purpose of these commands is to list the whole structure of the data directory and its sub-directories and files. You can choose any of the commands above. These should return a large number of index files. Below is sample output after running the tree command:

...
_0_1.liv  _1.si   _4.si              _e_Lucene50_0.tim  _e.si   _g.si
_0.cfe    _2.cfe  _e.fdt             _e_Lucene50_0.tip  _f.cfe  segments_1
_0.cfs    _2.cfs  _e.fdx             _e_Lucene54_0.dvd  _f.cfs  write.lock
_0.si     _2.si   _e.fnm             _e_Lucene54_0.dvm  _f.si
_1.cfe    _4.cfe  _e_Lucene50_0.doc  _e.nvd             _g.cfe
_1.cfs    _4.cfs  _e_Lucene50_0.pos  _e.nvm             _g.cfs
...

The set of data retrieved by the commands above should be far larger than this. If this directory is empty or very few files exist, the index likely needs to be rebuilt.

原因

• The search index in <bitbucket-home>/shared/search/data has been deleted or corrupted and needs to be rebuilt.
• The repositories didn’t get fully indexed. Some possible reasons this might have happened are:

• • Search server issue
  • Temporary issue caused by high CPU utilization
  • Search server might have been down while the file was committed.

ソリューション

If there is an issue with the Indexing, you will see some of these errors in the atlassian-bitbucket.log or in the bitbucket_search.log. In case that happens, we recommend rebuilding the Search server index.

Please note that if an index is missing, all the indices must be removed before running the reindex API.

(Recommended) Resolution #1 - trigger a reindex

The Bitbucket Server REST API provides a way to restart indexing by making an HTTP request. This can be done when running a Bitbucket instance using the following curl command to make a POST to the /rest/indexing/latest/sync REST endpoint:

curl -u <admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' <bitbucket-url>/rest/indexing/latest/sync

To check the index status: 

The Search reindexing or indexing triggered is run on a single Bitbucket node. Hence, checking it directly using the Bitbucket base URL will not yield the correct results, as the request could land on any of the Bitbucket nodes randomly. So, it is advised to check the reindexing status directly on the Bitbucket node's IP using the HTTP port number to find the indexing status. If there are multiple nodes, this must be checked against each node to determine which one is running the indexing.

curl -u <admin-user> -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' <bitbucket-node-ip>:port_no/rest/indexing/latest/status

This will trigger a re-index of all repositories. Bitbucket will be available with all functionality other than the search of unindexed portions of your code.  The time it takes to index depends on how much indexable content you have. This will be the amount of code contained in files under 512 KB.

Resolution #2 - Delete the Index

This resolution requires downtime to stop the application since it tries to remove the content in the filesystem and fix the issue by attempting to rebuild the indexes after starting Bitbucket. If the "Resolution #1" option does not resolve the problem, perform the following steps:

Bitbucket Data Center with an external search server:

• Option 1: Using REST APIs. 
  

  • If you are using a Bitbucket Data Center version before 6.0 you can use the following curl command:

    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search-v1

  • If you are using Bitbucket Data Center 6.0 or newer run all of the following commands: 

    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search
    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-state
    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-version
    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-project
    curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-repository

    In case of an external instance, please replace "-u bitbucket" with the actual user your OpenSearch instance is configured with.
    More details in KB Bitbucket Data Center Code Search is unable to find results from some or all repositories

•  Option 2: Manual deletion

• • Stop Bitbucket Server
  • Stop the external Search server
  • Create a backup of <SEARCH-SERVER_HOME>/data/local/nodes directory in your Search server
  • Delete the contents of <SEARCH-SERVER_HOME>/data/local/nodes directory
  • Start the external search server
  • Start Bitbucket Server

Single node Bitbucket Data Center with a bundled Search server:

1. Stop Bitbucket Server
2. Create a backup of <Bitbucket-home>/shared/search/data/nodes directory
3. Delete the contents of <Bitbucket-home>/shared/search/data/nodes directory
4. Start Bitbucket Server