Lexorank Rebalancing gets stuck with Error "Expected the row last migrated to be in the new bucket"
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
症状
When rebalancing is running, it does not progress. There is no error thrown in the Lexorank Management page.
診断
- Enable DEBUG for the rebalancing package
com.atlassian.greenhopper.service.lexorank.balance.
In the atlassian-jira.log, you'll see
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalancer] Locking lexorank balance with lock=com.atlassian.beehive.simple.SingleNodeLock@912d88d
2018-07-10 14:56:10,272 lexorank-executor-thread-0 INFO [service.lexorank.balance.LexoRankBalancer] Balancing 1 fields
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalancer] Balancing field with id[13000]
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG [greenhopper.manager.lexorank.LexoRankDaoImpl] LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "TYPE" = ? ORDER BY "RANK" ASC) WHERE ROWNUM <= 1 params[13000, 0]
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG [greenhopper.manager.lexorank.LexoRankDaoImpl] LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "TYPE" = ? ORDER BY "RANK" DESC) WHERE ROWNUM <= 1 params[13000, 2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalancer] Balancing next rank row for rank field [id=13000]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalanceOperation] Balancing next rank row to migrate from bucket[2] to bucket[0] for rank field[id=13000]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [greenhopper.manager.lexorank.LexoRankDaoImpl] LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "BUCKET" = ? ORDER BY "RANK" ASC) WHERE ROWNUM <= 1 params[13000, 2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [greenhopper.manager.lexorank.LexoRankDaoImpl] LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "BUCKET" = ? ORDER BY "RANK" DESC) WHERE ROWNUM <= 1 params[13000, 0]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalanceOperation] Fetched rows at balance boundary
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalanceOperation] Fetched rows at balance boundary
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalanceOperation] rowToMigrate : LexoRankRow{id=81332, fieldId=13000, issueId=97286, lockHash='null', lockTime=null, bucket=2, rank='2|hztity:', type=ISSUE_RANK_ROW}
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalanceOperation] rowLastMigrated : LexoRankRow{id=69625, fieldId=13000, issueId=87413, lockHash='null', lockTime=null, bucket=2, rank='2|i0366b:', type=ISSUE_RANK_ROW}
2018-07-10 14:56:10,288 lexorank-executor-thread-0 ERROR [service.lexorank.balance.LexoRankBalanceOperation] Expected the row last migrated to be in the new bucket [newBucket=0, rowBucket=2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG [service.lexorank.balance.LexoRankBalancer] Unlocking lexorank balance lock=com.atlassian.beehive.simple.SingleNodeLock@912d88d
2018-07-10 14:56:10,288 LexoRankReindexer:thread-1 ERROR [service.lexorank.balance.LexoRankBalancer] Indexing for LexoRank re-balancing was interrupted.
2018-07-10 14:56:10,288 lexorank-executor-thread-0 INFO [service.lexorank.balance.LexoRankBalancer] Balancing of 1 fields completed with 1 errors
2018-07-10 14:56:10,288 lexorank-executor-thread-0 WARN [service.lexorank.balance.LexoRankScheduledBalanceHandler] Balance for fieldId=13000 returned errors: Errors: {}
Error Messages: [Expected the row last migrated to be in the new bucket]
SQL queries to verify the mismatch "BUCKET column" vs "Bucket in RANK", they all should return 0:
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=0 and "RANK" not like '0|%'; select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=1 and "RANK" not like '1|%'; select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=2 and "RANK" not like '2|%';
In case of a non-zero result of the previous query, in the AO_60DB71_LEXORANK table, you'll see a record as shown below (in this case, rank values like '2|%' are marked under bucket 0, instead of bucket 2);
FIELD_ID, ID, ISSUE_ID, RANK, TYPE, BUCKET
13000, 69542, 87373, 2|hztjbb:f, 1, 0
13000, 69389, 87149, 2|hztjbb:p, 1, 0
13000, 69390, 87150, 2|hztjbb:s, 1, 0
原因
There is currently no known cause why the data gets corrupted with this integrity issue.
ソリューション
The resolution would be to fix the Bucket value in AO_60DB71_LEXORANK table.
- Generate JIRA backup
- Jira をシャットダウンします。
This is just an example, needs to be modified accordingly. Run the following SQL queries to update the bucket value to follow rank value (following the above example, we'll update the bucket value to 2 for those rank values like '2|%')
UPDATE "AO_60DB71_LEXORANK" SET "BUCKET"=2 WHERE "FIELD_ID"=<RANK_FIELD_ID> AND "BUCKET"<>2 AND "RANK" like '2|%' ;
Run the following SQL queries to verify that there is no corrupted entry in the AO_60DB71_LEXORANK table
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=0 and "RANK" not like '0|%'; select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=1 and "RANK" not like '1|%'; select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=2 and "RANK" not like '2|%';
- Trigger back the rebalancing, and wait for sometime and click refresh button. The rebalancing should be progressing.