要約

Several factors can contribute to Jira failing to start up and report plugin failures. This article lists the most common causes and the respective solutions.

This article applies to both system plugins and bundled or 3rd party plugins, like Automation for Jira, etc.

環境

Any version of Jira Data Center.

診断

Jira starts up but shows a Jira had problems starting up page when accessed through the Browser:

The page may report system plugins or 3rd party user installed plugins. Some root causes are the same.

原因

Common causes for this are:

1. Plugin startup timeout
2. Concurrent startup of Jira nodes
3. Java agents, bootdelegation or any other monitoring agent
4. Incompatible versions of atlassian-servlet-plugin or atlassian-authentication-plugin (JRASERVER-73257)
5. Older/incompatible versions of critical plugins or applications
6. Antivirus or file scanners on the home and install folders

1. Plugin startup timeout

Jira issues the initialization of all plugins and apps and waits for a configured amount of time before timing out and proceeding with the startup. The default 60 (seconds) threshold may not be sufficient and we'll see this key entry in the logs:

Unable to start the following plugins due to timeout

We can check if Jira's using the default 60 seconds or a different threshold by looking for "atlassian.plugins.enable.wait" in the startup logs:

$ egrep "atlassian.plugins.enable.wait\s*:" /jira-home/log/atlassian-jira.log
         atlassian.plugins.enable.wait                 : 120

If no lines are printed, Jira's using the default 60 seconds.

2. Concurrent startup

Starting Jira Data Center nodes concurrently is known to be a potential cause of issues like corrupt cache or plugins failing to start. This is fixed by another restart on the affected node (and no other restarts being performed at the same time).

You should avoid starting Jira in one node while another's still starting up. These two log entries are useful to determine when exactly Jira was started and when it's caches are all done building: 

$ tail -F /jira-home/log/atlassian-jira.log | egrep "exclusive use|Warmed cache"
2022-10-04 10:42:12,824-0300 localhost-startStop-1 INFO      [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/jira-home' is validated and locked for exclusive use by this instance.
2022-10-04 10:43:31,417-0300 Caesium-1-3 INFO      [c.a.jira.startup.CacheWarmerLauncher] Warmed cache(s) in 15923 ms.

Failures resulting from concurrent startups can also present NoClassDefFoundError entries in the atlassian-jira.log:

2022-01-01 10:00:35,013-0400 ThreadPoolAsyncTaskExecutor::Thread 2 ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.codebarrel.addons.automation'
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://199.0:0/META-INF/spring/plugin-context.xml]; nested exception is java.lang.NoClassDefFoundError: org/w3c/dom/Element

3. Java agents, bootdelegation or any other monitoring agent

Several monitoring tools use Java Agents configured in the monitored application's JVM startup parameters to allow monitoring data collection.

When this is the cause of the startup failure, we should see NoClassDefFoundError entries in atlassian-jira.log:

Caused by: java.lang.NoClassDefFoundError: jdk/internal/reflect/MethodAccessorImpl

The reported "not found" classes vary widely, and we can confirm the presence of any custom agent by looking for "agent" or "bootdelegation" keywords in the startup logs where it prints "JVM Input Arguments":

$ grep "JVM Input Arguments" /jira-home/log/atlassian-jira.log | egrep -i "agent|bootdelegation"

4. Incompatible versions of atlassian-servlet-plugin or atlassian-authentication-plugin

Jira Software versions above 8.13 and Jira Service Management versions above 4.13 may have system plugins failing to start.

This is further detailed in JRASERVER-73257 and can be confirmed by searching for the plugins in the atlassian-jira.log after a startup attempt:

$ egrep ": [a-z\.]+atlassian-servlet-plugin$" /jira-home/log/atlassian-jira.log -A3
         atlassian-servlet-plugin                      : com.atlassian.web.atlassian-servlet-plugin
              Version                                       : 5.3.0
              Status                                        : enabled
              Vendor                                        : Atlassian

Versions below 6.0.1 of atlassian-servlet-plugin are affected.

$ egrep ": [a-z\.]+atlassian-authentication-plugin$" /jira-home/log/atlassian-jira.log -A3
         SSO for Atlassian Data Center                 : com.atlassian.plugins.authentication.atlassian-authentication-plugin
              Version                                       : 4.2.0
              Status                                        : enabled
              Vendor                                        : Atlassian

Versions below 4.2.7 of atlassian-authentication-plugin are affected.

JRASERVER-73257 - 課題詳細を取得中... ステータス

5. Older/incompatible versions of critical plugins or applications

On this example, com.atlassian.jira.mobile.jira-mobile-rest plugin failed to load because there was an incompatible older version of Jira Software jars on installed-plugins directory.

For reference, the versions were:

• Jira Core 9.4.1 with Jira Service Management 5.4.1.
• Jira Software 8.13.0 was disabled but its Jars were still on installed-plugins .

        'com.atlassian.jira.jira-software-application' - 'Jira Software Application'  failed to load.
                Cannot start plugin: com.atlassian.jira.jira-software-application
                        Unable to resolve com.atlassian.jira.software-application [87](R 87.0): missing requirement [com.atlassian.jira.software-application [87](R 87.0)] osgi.wiring.package; (osgi.wiring.package=com.atlassian.jira.software.internal.obr.install) Unresolved requirements: [[com.atlassian.jira.software-application [87](R 87.0)] osgi.wiring.package; (osgi.wiring.package=com.atlassian.jira.software.internal.obr.install)]

                It was loaded from /var/atlassian/application-data/jira/plugins/installed-plugins/jira-software-application-8.13.0.jar

        'com.atlassian.jira.mobile.jira-mobile-rest' - 'Mobile Plugin for Jira Data Center and Server'  failed to load.
                Application context initialization for 'com.atlassian.jira.mobile.jira-mobile-rest' has timed out waiting for (|(&(objectClass=com.atlassian.greenhopper.service.rapid.view.ColumnService)(objectClass=com.atlassian.greenhopper.service.rapid.view.ColumnService))(&(objectClass=com.atlassian.greenhopper.api.customfield.ManagedCustomFieldsService)(objectClass=com.atlassian.greenhopper.api.customfield.ManagedCustomFieldsService))(&(objectClass=com.atlassian.greenhopper.service.rapid.ProjectRapidViewService)(objectClass=com.atlassian.greenhopper.service.rapid.ProjectRapidViewService))(&(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewHistoryService)(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewHistoryService))(&(objectClass=com.atlassian.greenhopper.service.issue.RapidViewIssueService)(objectClass=com.atlassian.greenhopper.service.issue.RapidViewIssueService))(&(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewQueryService)(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewQueryService))(&(objectClass=com.atlassian.greenhopper.service.rapid.view.RapidViewService)(objectClass=com.atlassian.greenhopper.service.rapid.view.RapidViewService))(&(objectClass=com.atlassian.greenhopper.service.sprint.SprintService)(objectClass=com.atlassian.greenhopper.service.sprint.SprintService)))

                It has the following missing service dependencies :
                         &rapidViewQueryService of type (&(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewQueryService)(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewQueryService))
                         &rapidViewIssueService of type (&(objectClass=com.atlassian.greenhopper.service.issue.RapidViewIssueService)(objectClass=com.atlassian.greenhopper.service.issue.RapidViewIssueService))
                         &rapidViewService of type (&(objectClass=com.atlassian.greenhopper.service.rapid.view.RapidViewService)(objectClass=com.atlassian.greenhopper.service.rapid.view.RapidViewService))
                         &managedCustomFieldsService of type (&(objectClass=com.atlassian.greenhopper.api.customfield.ManagedCustomFieldsService)(objectClass=com.atlassian.greenhopper.api.customfield.ManagedCustomFieldsService))
                         &sprintService of type (&(objectClass=com.atlassian.greenhopper.service.sprint.SprintService)(objectClass=com.atlassian.greenhopper.service.sprint.SprintService))
                         &rapidViewHistoryService of type (&(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewHistoryService)(objectClass=com.atlassian.greenhopper.service.rapid.RapidViewHistoryService))
                         &projectRapidViewService of type (&(objectClass=com.atlassian.greenhopper.service.rapid.ProjectRapidViewService)(objectClass=com.atlassian.greenhopper.service.rapid.ProjectRapidViewService))
                         &columnService of type (&(objectClass=com.atlassian.greenhopper.service.rapid.view.ColumnService)(objectClass=com.atlassian.greenhopper.service.rapid.view.ColumnService))

                It was loaded from /var/atlassian/application-data/jira/plugins/installed-plugins/plugin_17209659747072710362_jira-mobile-rest-4.1.0.jar

6. Antivirus or file scanners on the home and install folders

If you have any Antivirus or file scanner running on the Jira app servers, they may delay startup and eventually cause the startup to fail.

This may have an "intermittent" pattern, with Jira successfully starting up after a few attempts then failing again on another opportunity. There is no clear evidence on the logs of this specific issue (it may masquerade as a timeout issue or have no mention to timeouts at all).

This is usually observed in Windows app servers, though similar apps in Linux servers can also lead to startup failures.

ソリューション

Here are the respective solutions for each of the enumerated causes.

1. Plugin startup timeout

If the "due to timeout" message was present in the logs after a startup attempt, you may add the below parameter to Jira's JVM as further detailed in the KB article on Bundled plugin not available error thrown on Jira server login:

-Datlassian.plugins.enable.wait=300

The 300 value means Jira will wait for 5 minutes before timing out on plugin initialization. Please refer to the article for troubleshooting guidance on why plugin initialization's slower than usual.

2. Concurrent startup

Ideally, you should not start a Jira node while there's any other node still being started up.

We're aware of the challenge this poses to automated environments and LTS Jira 9.2 ships with significant cache management improvements that should mitigate concurrent startup issues.

Threads in the Community (link to keyword search) suggest explicit Import-Package entries in the plugins' pom.xml (something the plugin vendor should work on) may mitigate the issue — but still, concurrent startups should be avoided.

3. Java agents, bootdelegation or any other monitoring agent

For troubleshooting purposes, you may (backup and) remove the "agent" parameter from the JVM and restart Jira. It can be present in a number of forms:

-javaagent
-agentpath
-Datlassian.org.osgi.framework.bootdelegation

If Jira starts up alright, you may proceed with either:

• Contact the agent vendor for help. There may be incompatibilities with the agent version or other configuration needed. Please note such integrations are beyond the scope of Atlassian Support.
• If you were using the -Datlassian.org.osgi.framework.bootdelegation parameter, replace it by -Datlassian.org.osgi.framework.bootdelegation.extra as described in the article on Jira fails to start on Java 11 with plugin load errors when integrated with a monitoring agent.

Startup failures due to agents or custom class loaders are often related to race conditions during the JVM's initialization, so this error may look "intermittent" or start occurring even if "no changes were made to the instance".

4. Incompatible versions of atlassian-servlet-plugin or atlassian-authentication-plugin

If Jira still fails to start system plugins after applying all other solutions presented here, you may refer to the workarounds or fixes described in JRASERVER-73257.

Upgrading Jira Software to the latest 8.20 release (currently 8.20.13) or any version of Jira 9.0 and above should bundle the fixed dependencies. The same for Jira Service Management 4.20.13 and 5.0.

5. Older/incompatible versions of critical plugins or applications

Remove older and incompatible jars from installed-plugins , more specifically application jars such as jira-software-application-x.x.x.jar or jira-servicedesk-application-y.y.y.jar . There will be other jars related to the same application. For Jira Software you may have others such as:

jjira-bamboo-plugin-x.x.x.jar
jira-fisheye-plugin-x.x.x.jar
jira-greenhopper-plugin-x.x.x.jar
jira-software-monitor-plugin-x.x.x.jar
jira-software-plugin-x.x.x.jar
jira-transition-triggers-plugin-x.x.x.jar

• There could be other files, so you may engage support if you need assistance determining what to remove. The key is to look for the same version in the jar names that you want to remove.

Note: back up these files in case there are unexpected results and make sure you do not remove the current version of your apps.

6. Antivirus or file scanners on the home and install folders

We advise allow-listing Jira home and installation paths on the Antivirus or file scanner application as further explained on:

• Configure Anti-virus software for use with Jira server