問題

When runnings Fisheye and Crucible in AWS where $FISHEYE_INST is mounted to an AWS Elastic File System (EFS), indexing of repositories may stop when certain limits are reached.

atlassian-fisheye-YYYY-MM-DD.log に次のメッセージが表示される。

Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed 
out: NativeFSLock@/efs/var/cache/glass-100/idx2/write.lock: java.io.IOException:
 Disk quota exceeded

診断

環境

• Fisheye and Crucible running in AWS using EFS for $FISHEYE_INST

Diagnostic Steps

• The number of locks currently acquired can be seen with lslocks | wc -l
• The number of open files can be checked with sysctl fs.file-nr

原因

AWS EFS has limits on the number of active users that can have files open, the number of open files, and the number of locks that can be acquired. If one of these limits is exceeded, a Disk quota exceeded message appears in the logs. With Fisheye and Crucible, it's more likely that you'll hit the open file limit or number of locks that can be acquired. The limits are:

• Up to 128 active user accounts can have files open at once for an instance.

• Up to 32,768 files can be open at once for an instance.

• Each unique mount on the instance can acquire up to a total of 8,192 locks across 256 unique file-process pairs. For example, a single process can acquire one or more locks on 256 separate files, or eight processes can each acquire one or more locks on 32 files.

Information on the limits of EFS is available in AWS's Documentation at https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-fileop-errors.html#diskquotaerror

回避策

Since these are hard limits in EFS, there is limited ability to work around them. You may be able to reduce the number of open files or disable unused repositories to free up open files or locks.

ソリューション

Due to the way that Fisheye and Crucible use locks and open files, larger instances that are running into this limit will need to migrate to use AWS Elastic Block Store (EBS) which does not impose these same limits.