Assets and memory requirements

As your instance grows, the in-memory index might need more Java Virtual Machine (JVM) memory over time. Make sure that your Jira is configured to handle the amount of data you plan to use in Assets. Check out the Assets memory recommendations

How indexing works in Assets

In Assets, indexing is managed through an in-memory cache system Google's Guava Cache. In this system, different types of Assets data use different indexes, each optimized for specific data categories. With this approach, the system remains responsive and effective even as data complexity increases. More about Google’s Guava Cache

Indexed flag

An indexed flag is a setting that determines whether specific attributes of an asset are included in the index for search and retrieval purposes. In Assets, all attributes are indexed by default, except for the text area.

You can choose to remove certain attributes from the index by disabling the indexed flag on those attributes. While this can help reduce memory consumption, it might slow down searches and imports, as attributes will need to be retrieved directly from the database. More about attribute configuration

Disk storage

Unlike Jira, Assets indexes aren't stored in Lucene (except for the free text index). This means the Assets indexes don't usually remain on a disk. Administrators can manually transfer the indexes to a disk. Alternatively, they can do that automatically on shutdown by enabling Store cache on shutdown in the assets configuration.

When the system is shut down and Store cache on shutdown is enabled or when the Persist Assets index to file is manually triggered, the following indexes are written to files in the JiraLocalHome/caches/insight_indexes directory:

• オブジェクト

• オブジェクト スキーマ

• オブジェクト タイプ

• Object type attributes

• Object ticket connections

This process involves writing the index to a temporary file, deleting any existing index file, and then moving the temporary file to the insight_indexes directory.

Disk storage on startup

On startup, if Restore Assets index from file is enabled, the index will be recreated from the index files on disk. If this setting isn't enabled, the index will be re-created from the database.

For a large number of objects, restoring indexes from a file is more efficient on startup compared to re-creating the index from the database. However, re-creating the index from the database could mitigate the risk of a potentially corrupted index file, which may cause data inconsistency. More about configuration settings

Free text index

The free text index for Assets is stored in the JiraLocalHome/caches/insight_indexes/freetext/directory, with one index maintained per schema. Every night at midnight, Assets automatically runs a scheduled job to reindex the free text index. Currently, there's no option to disable this job. Additionally, a free text reindex is triggered whenever a manual reindex is performed. This setup ensures that the indexing remains up-to-date, although it can limit flexibility in managing reindexing schedules.

Clustering and replication

In a clustered environment, each node maintains its own index, requiring changes to be replicated between nodes so that all nodes are aware of the current data state. This replication process is complex, as it must handle network disruptions, heavy loads (such as during imports), and general performance challenges. Replication occurs with every single object change.

Starting from Jira Service Management 5.8.0, invalidation notifications in Assets use Atlassian Cache node-to-nod invalidation instead of cluster messages for the object index. More about performance improvements in Assets

Replication flow

Replication includes the following object operations: create, update, and delete. The CacheMessage includes:

• index type

• type of operation, such as add, remove, or update

• ID of the entity to update in the index

• update time

The default capacity for the work queue size is 10,000. You can configure the capacity by updating the Sender queue size property (com.atlassian.assets.replication.work.queue.size) on the Assets configuration page. More about Assets configuration

The batch replication message contains the IDs of the objects to be added and deleted and the IDs of updated objects along with their update times. For example, if there’s an object creation with the ID 21229075 in the jiradctest3 node, you can see the following message in the logs:

2024-09-19 16:55:59,747+0200 assets-replicator-3 TRACE      [i.r.i.i.replication.poller.CacheMessageWorkQueuePoller] Batch message: AssetsBatchReplicationMessage(id=026ef180-78d5-487e-a8fb-35507714336f, sourceNode=jiradctest3, cacheType=OBJECT, addIds=[21229075, 21229057 ], updateIds={}, deleteIds=[], messageTimestamp=1726757759747, retryAttempts=0, lastRetryTimestamp=1726757759747)

On the other nodes, the message could be similar to this:

2024-09-19 16:56:00,729+0200 RMI TCP Connection(4298)-10.150.208.4 DEBUG      [i.r.i.index.replication.AssetsReplicationTransportManager] onRemove for event: 026ef180-78d5-487e-a8fb-35507714336f
2024-09-19 16:56:00,724+0200 RMI TCP Connection(5457)-10.150.208.4 DEBUG      [i.r.i.index.replication.AssetsReplicationTransportManager] Received event from node jiradctest3 from object cache with ID 026ef180-78d5-487e-a8fb-35507714336f
2024-09-19 16:56:00,724+0200 RMI TCP Connection(5457)-10.150.208.4 DEBUG      [i.r.i.index.replication.AssetsBatchReplicationMessageReceiver] Received message with ID 026ef180-78d5-487e-a8fb-35507714336f
2024-09-19 16:56:00,725+0200 assets-replication-receiver-0 TRACE      [i.r.i.i.replication.poller.AssetsBatchReplicationMessageWorkQueuePoller] Processing replication message with ID 026ef180-78d5-487e-a8fb-35507714336f

Replication happens by invalidation. Each node receives a remove message and independently loads updates from the database. More about Assets index validation property defaults

Dead-letter queue

A dead-letter queue (DLQ) is a holding area for messages that can't be processed successfully. By default, items on the retry queue are retried every minute for a total of 10 minutes. If you need, you can configure this in the Assets configuration.

When all retries fail, the failed update messages are moved to the DLQ. If necessary, you can replay (retry) items in the DLQ once you've resolved the root cause of the failure, or you can clear them if the issue can't be fixed or if a reindex has already been completed.

These endpoints let you either replay or clear the DLQ:

• GET rest/insight/1.0/deadletterqueue/replay that replays the DLQ in a non-cluster mode and sends a broadcast request to replay the DLQ of all nodes in a clustered mode.

• GET rest/insight/1.0/deadletterqueue/replay/{nodeId} that sends a unicast request to replay the DLQ of a specific node in a clustered mode. The nodeId is the node that should receive the request and replay its DLQ.

Reindexing Assets

Sometimes, your Assets data can fall out of synchronization with the database. This can happen due to direct alterations in the database or if the index replication process fails. To get everything back, you'll need to reindex Assets. Reindexing involves dropping all in-memory data and pulling fresh data from the database. You can perform the following reindex actions:

• Clean re-index where all objects will be removed from the index across all nodes, and then will be indexed again. This is recommended if you want to have a fresh state of the index. Once the indexing is in progress, you won't be able to cancel it, search for objects, or filter them. This reindexing will drop all IDs stored in the cache.

• Re-index (standard re-index) where all objects will stay in the index during the process, and Assets will index them again. You can search for objects during the process. This reindexing will keep the IDs and only retrieve data for the IDs that are already in the cache.

• Persists Assets index to file where you can manually persist (copy) the index on your disk. This is useful if you have a big Assets environment with a large number of objects and are planning to reinstall your Jira. With the index on your disk, Assets won't have to recreate it from scratch.

During a clean reindex, the local index for each type is cleared out, and a message is sent to all nodes to remove their copies of the index. This message includes the index type to be removed (like object) and the REMOVE_ALL operation. All local index files persisted on disk are deleted. The node that's processing the reindex will start rebuilding its index. You can configure the number of threads used in this process by adjusting the Assets parallelism property on the Assets configuration page. Then, all nodes will remove their local copy of the index and rebuild their indexes using the replication process.

Locking

During a reindex, you can't use Assets on the node that's executing the reindex. In the case of a clean re-index, Assets can't be used across the entire cluster. To manage this, a lock is applied to prevent any modification of Assets data. Once the reindex is completed, Assets will unlock.

Startup

On startup, if you've enabled Restore Assets index from file, the index will be recreated from the index files on disk. If you don’t enable this setting, the index will be re-created from the database. An integrity check of the Assets index files will be conducted against the database, which serves as the source of truth and verifies whether the data is consistent. If the data is inconsistent, an Assets reindex will be initiated.

You can see the reindex details in the logs that will look similar to the following example.

2024-11-11 10:15:27,870 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Control corrupted data done
2024-11-11 10:15:27,872 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] property: 'assets.version' set to: 20.2.0-SNAPSHOT
2024-11-11 10:15:27,873 InsightLauncherThread WARN anonymous     [i.r.i.i.m.InsightPersistedIndexManager] Assets Object Cache file not found <localhome>/caches/insight_indexes/objectschema
2024-11-11 10:15:27,876 InsightLauncherThread WARN anonymous     [i.r.i.i.m.InsightPersistedIndexManager] Assets Object Cache file not found <localhome>/caches/insight_indexes/objecttype
2024-11-11 10:15:27,877 InsightLauncherThread WARN anonymous     [i.r.i.i.m.InsightPersistedIndexManager] Assets Object Cache file not found <localhome>/caches/insight_indexes/objecttypeattribute
2024-11-11 10:15:27,877 InsightLauncherThread WARN anonymous     [i.r.i.i.m.InsightPersistedIndexManager] Assets Object Cache file not found <localhome>/caches/insight_indexes/objects
2024-11-11 10:15:27,877 InsightLauncherThread WARN anonymous     [i.r.i.i.m.InsightPersistedIndexManager] Assets Object Cache file not found <localhome>/caches/insight_indexes/objectConnection
2024-11-11 10:15:27,881 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Restore Assets cache done
2024-11-11 10:15:27,882 InsightLauncherThread WARN anonymous     [c.r.j.p.i.s.c.i.i.IndexIntegrityCheckerImpl] Integrity check failed, number of objects mismatched, database: 1, index: 0
2024-11-11 10:15:27,884 InsightLauncherThread WARN anonymous     [c.r.j.p.i.s.c.i.i.IndexIntegrityCheckerImpl] Integrity check failed, number of objects schemas mismatched, database: 1, index: 0
2024-11-11 10:15:27,885 InsightLauncherThread WARN anonymous     [c.r.j.p.i.s.c.i.i.IndexIntegrityCheckerImpl] Integrity check failed, number of objects types mismatched, database: 21, index: 0
2024-11-11 10:15:27,885 InsightLauncherThread WARN anonymous     [c.r.j.p.i.s.c.i.i.IndexIntegrityCheckerImpl] Integrity check failed, number of objects type attribute mismatched, database: 138, index: 0
2024-11-11 10:15:27,888 InsightLauncherThread WARN anonymous     [c.r.j.p.i.s.l.InsightLauncher] Cache integrity check failed: IndexIntegrity{objectJiraIssueIndexOk=true, objectSchemaIndexOk=false, objectTypeAttributeIndexOk=false, objectTypeIndexOk=false, objectIndexOk=false}
2024-11-11 10:15:27,893 InsightLauncherThread INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexServiceImpl] Numbers of objects in database 1
2024-11-11 10:15:27,895 InsightLauncherThread INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexServiceImpl] Starting Assets Reindex with a total work units of 161
2024-11-11 10:15:27,930 InsightLauncherThread INFO Anonymous user     [c.r.j.p.i.s.c.m.AsyncTaskHandlerImpl] Starting InsightMultiThreadedEnabled service AsyncTaskHandlerImpl with parallelism set to 10
2024-11-11 10:15:27,934 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Check Assets cache integrity done
2024-11-11 10:15:27,934 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Start scheduling object attributes
2024-11-11 10:15:27,934 insight-InsightThreadGroup-worker-thread-0 WARN Anonymous user     [c.r.j.p.i.s.c.i.ReindexServiceImpl] Starting Assets reindex in 10 threads
2024-11-11 10:15:27,934 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.InsightLockServiceImpl] Locking Assets insight_locked
2024-11-11 10:15:27,935 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Scheduling object attributes done.
2024-11-11 10:15:27,936 InsightLauncherThread INFO anonymous     [c.r.j.p.i.s.l.InsightLauncher] Scheduling import done.
2024-11-11 10:15:27,936 insight-InsightThreadGroup-worker-thread-0 WARN Anonymous user     [c.r.j.p.i.s.c.i.ClusterAwareReindexServiceImpl] Assets is locked for reindex
2024-11-11 10:15:27,943 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.c.i.ObjectTypeReindexStrategy] Getting object type IDs to index
2024-11-11 10:15:27,945 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.c.i.ObjectTypeReindexStrategy] Found 21 object type IDs to index
2024-11-11 10:15:27,946 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.c.i.ObjectTypeReindexStrategy] Started Assets reindex of object types
2024-11-11 10:15:27,974 insight-InsightThreadGroup-worker-thread-3 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:27,974 insight-InsightThreadGroup-worker-thread-3 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:27,974 insight-InsightThreadGroup-worker-thread-1 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,073 insight-InsightThreadGroup-worker-thread-8 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,074 insight-InsightThreadGroup-worker-thread-9 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,074 insight-InsightThreadGroup-worker-thread-5 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,075 insight-InsightThreadGroup-worker-thread-7 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,077 insight-InsightThreadGroup-worker-thread-6 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,078 insight-InsightThreadGroup-worker-thread-4 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,078 insight-InsightThreadGroup-worker-thread-2 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexObjectTypesJob] Object type index complete for thread
2024-11-11 10:15:28,078 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.c.i.ObjectTypeReindexStrategy] Reindex of object types completed in 135ms
2024-11-11 10:15:28,362 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.c.i.ReindexServiceImpl] Finishing progress OK ProgressId: resourceId: reindex, category: assets-reindex
2024-11-11 10:15:28,362 insight-InsightThreadGroup-worker-thread-0 INFO Anonymous user     [c.r.j.p.i.s.InsightLockServiceImpl] Unlocking Assets insight_locked
2024-11-11 10:15:28,364 insight-InsightThreadGroup-worker-thread-0 WARN Anonymous user     [c.r.j.p.i.s.c.i.ClusterAwareReindexServiceImpl] Assets is unlocked after reindex

構成

アセットのグローバル Jira 設定にアクセスするには、次の手順に従います。

1. Go to Administration, then Manage apps.

2. In the Assets section, go to Assets configuration.

3. At the bottom of the Assets configuration page, select Edit settings.
4. Go to the General Assets configuration section and change the needed configuration.

Restricting the index size

You can limit the number of objects in the Assets index to reduce memory consumption. However, this will affect performance for the objects that aren't in the index, as those objects will be queried directly from the database.

トラブルシューティング

If you run into indexing issues, refer to the information below.

Reindex inconsistent state

In some cases, data can end up in an inconsistent state in the database. When this happens, the data can't be referenced in the index, which can cause issues when you're trying to view it in the Assets application.

Starting with Jira Service Management 10.1, these errors display in the Assets indexing UI and logs. More about troubleshooting Assets index

Atlassian profiling and metrics

Assets collects and exposes some Java Management Extensions (JMX) metrics that you can use to monitor your assets replication stats. More about live monitoring using the JMX interface

ログ

Below you can see an overview of loggers (logging mechanisms) used in Assets to monitor and troubleshoot replication and reindexing processes.