Elasticsearch has got corrupted index

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

この記事はアトラシアンのサーバー製品にのみ適用されます。クラウドとサーバー製品の違いについてはこちらをご確認ください。

問題

There are many different messages to show that the Elasticsearch index is corrupt, somehow. In case that happens, we recommend you to rebuild your Elasticsearch index. This procedure will list the required steps for that.

As an example of corruption, the following appears in the $BITBUCKET_HOME/log/search/bitbucket_search.log


[2016-11-23 16:10:27,080][WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][3]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
        at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
        at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: [bitbucket-search-v1][[bitbucket-search-v1][3]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.engine.InternalEngine.< int>(InternalEngine.java:155)
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
        at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
        at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
        ... 5 more
Caused by: java.nio.file.NoSuchFileException: /var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at org.elasticsearch.index.translog.Translog.openReader(Translog.java:374)
        at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:334)
        at org.elasticsearch.index.translog.Translog.< init>(Translog.java:179)
        at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
        at org.elasticsearch.index.engine.InternalEngine.< init>(InternalEngine.java:151)
        ... 11 more   

原因

There was a corruption in your Elasticsearch data.

ソリューション

We recommend you rebuild your Elasticsearch index. Here are the steps to be followed:

It is recommended that you have a working backup of your instance, although the procedure below deletes data purely related to Elasticsearch.

For Bitbucket Server version up to 4.x

 When running Bitbucket Server version up to 4.x, you can simply stop Elasticsearch (i.e. without restarting Bitbucket Server itself). Here are the steps:

  • Stop the Bitbucket Search process:

    # service atlbitbucket_search stop
  • Go into the $BITBUCKET_HOME/shared/search directory
     
  • List its contents. That is what you should see:

    # ls -l
    total 8
    drwxr-xr-x. 2 atlbitbucket atlbitbucket   24 Nov 21 02:50 buckler
    drwxrwxr-x. 3 atlbitbucket atlbitbucket   29 Nov 29 00:55 data
    -rw-r--r--. 1 atlbitbucket atlbitbucket  246 Nov 21 02:50 elasticsearch.yml
    -rw-r--r--. 1 atlbitbucket atlbitbucket 3009 Nov 21 02:50 logging.yml
    drwxrwxr-x. 2 atlbitbucket atlbitbucket    6 Nov 29 00:55 scripts
  • Delete the scripts and data directories:

    # pwd
    /var/atlassian/application-data/bitbucket/shared/search
    # rm -rf data scripts



  • Start Elasticsearch

    # service atlbitbucket_search start
  • Trigger a reindex:

    curl -u <admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' http://<YOUR_BTIBUCKET_ADDR>/rest/indexing/latest/sync
  • Wait for a little bit but at this stage your repositories will be fully reindexed and soon you will be able to search again. You can follow everything happening in the logs if you have a terminal open on $BITBUCKET_HOME/log/search and you are doing a tail -f bitbucket_search.log. If you want to check the progress of an index you can use the REST endpoint  /rest/indexing/latest/status
  • This provides a look into the current queue. If for some reason there's an event that tails to queue, or there's an error in indexing, it'll be placed onto a queue that delays by 5 minutes. When it sits at zero for a while, we can be confident that indexing is complete (for now). Of course that's a moving target as new events coming in will be added to the queue, but is a good guide for when indexing is done. Overall, as long as search is working, that delayed queue will eventually complete and the status will change to idle.


説明

There are many different messages to show that the Elasticsearch index is corrupt, somehow. In case that happens, we recommend you to rebuild your Elasticsearch index. This procedure will list the required steps for that.

製品 Bitbucket Server
プラットフォーム サーバー
最終更新日 2018 年 11 月 2 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.