Troubleshooting SVN Indexing Related Issues in Fisheye/Crucible
This page lists the troubleshooting tips required to investigate some common SVN issues reported while indexing a repository in Fisheye/Crucible.
Recommendations for SVN integration
Latency of SVN communication
- Fisheye requires detailed information for every commit in configured SVN repository. The faster it can talk to SVN, the faster indexing will proceed.
- The http/s protocol has the slowest performance during the initial scan. The svn protocol (svn://) is faster and the file protocol (file:///) is the fastest.
- Therefore if you find your initial scan takes an extended amount of time (more than a day or two), you should consider switching over from the http/s protocol to the svn or file protocol to define the location of your SVN repository. (Use svnsync to mirror the repository onto the fisheye server, so that you can access it with the file protocol.). In order for SVN protocol to work you need to have set up an svnserve based server.
SVN Branch/Tag Structure
Ensure the SVN branch and tag structure has been selected correctly in Fisheye repository configuration. If Fisheye does not know which files are tags and branches, it will treat all files as trunk files. This can significantly increase the effective size of your repository and the initial scan time as well as impact runtime performance. This guide has a number of examples to help setup correct SVN branch and tag mapping in Fisheye.
Fisheye Index Structure
- Fisheye is a caching application. It caches all the repository information. Therefore, big repositories do take time to be indexed.
This index/repository cache is stored in the var directory defined in the Fisheye configuration, under a subdirectory named var/cache/name of the repository. For example, the indexes for the repository
FEare under var/cache/FE, with one subdirectory for each index:
- Metadata index is stored in
- Content index is stored in
- DiffText index is stored in
- Metadata index is stored in
Limit Indexing Scope
- You can limit the repository content that Fisheye will index by either:
- Setting a start revision to start indexing from a particular revision.
- Exclude certain file types/paths that are not required to be indexed.
- Turn OFF the store-diff info property. Some of the benefits are:
- Speed up indexing; Fisheye won't have to retrieve and index all diffs.
- Reduce disk footprint. The diff index files can become quite large, so disabling diffs can lead to significant savings.
- Reduce disk I/O, particularly write operations. If disk I/O is a performance bottleneck, disabling diff indexing could help.
- Reduce the load on your SVN server during indexing, but you'll only notice the difference if you've got a lot of commit activity going on.
Indexing Thread Configuration
- Check for the indexing thread configuration.
Fisheye uses one thread each for incremental and initial indexing for each repository. 3 is a recommended thread count. This means a maximum of 3 repositories will be processed concurrently for full re-indexing, and a maximum of 3 repositories will be processed concurrently for incremental indexing (the periodic indexing that occurs to keep Fisheye up to date with repository contents). Other repositories will be queued. Having more number of threads configured will add extra overhead on server when indexing and negatively impact performance.
Amount of heap allocated to the Fisheye instance
- The more indexing threads you allocate, the more heap (memory) that Fisheye will consume. Without sufficient memory, the Fisheye instance will spend a lot of time garbage collecting.
Number of processor cores available to Fisheye
- The more processors available to Fisheye, the faster indexing will go. Please refer to the hardware requirements for more details.
Block size for SVN indexing
- This setting determines the number of changesets that are processed at a time. A larger number reduces the number of calls to SVN, but consumes more memory. A lower number increases the number of calls to SVN, but reduces memory consumption.
Debugging Slow Indexing
Command Line Tool
- Monitor the size of your <FISHEYE_INST>/var/cache/name of the repository folder on disk as this should be similar to your SVN repository on the server. If there is a huge size mismatch, that is usually an indication of incorrect SVN branch and tag mapping in Fisheye. If there is a huge mismatch, you may use a command line tool shipped with the latest version of Fisheye, that can be downloaded from here. Please note that this does not require you to upgrade to use it, as this is command line tool. You can simply run it against your current version. To run the tool:
- Download the latest Fisheye version
- Unpack it to some folder (e.g. /tmp/fecru-2.10.1)
- Go into that folder cd /tmp/fecru-2.10.1
- Point FISHEYE_INST environment variable to the folder where your current instance home is, e.g. export FISHEYE_INST=/var/fisheye
- Run svn symbolic rule analyser on your repository, with a command ./bin/fisheyectl.sh svnrules <Repo Name> | tee <Repo Name>-svn-rules.log
This command may take a while as it would run svn log -v against the specified repository, but it would attempt to analyse all the copy operations in your repository and would advice where it is most likely that there is a missing svn symbolic rule definition. Once this has run see the end of the generated <Repo Name>-svn-rules.log file, it would contain summary of the 10 most common folders.
You don't need to stop your current instance to perform this check. The generated output would help to determine what the missing svn symbolic rules are. But please bear in mind the tool may report false positives, as sometimes it is genuine activity in the repository to copy some files or folders within the trunk.
- With debug logging enabled, look for log entries beginning with
. These entries correspond to indexing related tasks currently running in the application and the Thread No. corresponds to which thread is being used to complete the task on that repository. Using the timestamp on the log, you can then investigate the last time that Fisheye ran an incremental/initial index on a repository. and [
InitialPinger<Thread No.><Repository Name>]
- Debug logs will also report the exact commands being run on the repository at the time of initial and incremental indexing, for example:
2013-04-08 14:05:45,971 DEBUG [IncrementalPinger2 git_tutorial] fisheye.app LoggingProcessMonitor-onBeforeStart - Starting process: git fetch /Users/ganand/Git Tutorial/git_tutorial/work/hello/ +refs/*:refs/* . . . 2013-04-08 14:05:46,006 DEBUG [IncrementalPinger2 git_tutorial] fisheye.app LoggingProcessMonitor-onAfterFinished - Finished process: git fetch /Users/ganand/Git Tutorial/git_tutorial/work/hello/ +refs/*:refs/* took 33ms
In the above example, the "Starting process" and "Finished process" are the messages used to track the start and end of a command and how long did it take to complete - "took 33ms" in this case. This can be helpful in identifying some long running commands or if they do not complete at all when there are no corresponding "Finished process" log entries. You may then run the same commands via the command line directly on your server and compare the results.
- Logs will also lodge the time taken to index a particular repository, for example:
2013-04-08 14:06:05,833 INFO [InitialPinger3 git] fisheye.console BaseRepositoryScanner-ping - [git] starting initial scan of repository . . . 2013-04-08 14:06:11,817 INFO [InitialPinger3 git] fisheye.console BaseRepositoryScanner-ping - [git] finished initial scan of repository in 5s
- With debug logging enabled, you can also look for the exact command that Fisheye is failing at while indexing, for example:
2013-04-08 14:28:40,272 DEBUG [InitialPinger3 svn] fisheye.app SvnThrottledClient-start - Executing ls -r HEAD https://URL/public/@HEAD 2013-04-08 14:28:41,834 WARN [InitialPinger3 svn] fisheye.app SvnRepositoryTester-getServerRootURL - Unable to get Repository Root URL for svn:https://URL/public/ 2013-04-08 14:28:41,834 DEBUG [InitialPinger3 svn] fisheye.app SvnRepositoryTester-pingAndValidateAccess - Validating access to https://URL/public/ 2013-04-08 14:28:41,834 DEBUG [InitialPinger3 svn] fisheye.app SvnThrottledClient-start - Executing propget fisheye.access -r HEAD https://URL/public/@HEAD 2013-04-08 14:28:43,124 DEBUG [InitialPinger3 svn] fisheye.app RepositoryStatus-setMessage - Status change [svn]: Could not access https://URL/public/ 2013-04-08 14:28:43,124 ERROR [InitialPinger3 svn] fisheye.app SvnRepositoryTester-pingAndValidateAccess - Exception getting FishEye access control com.cenqua.fisheye.rep.RepositoryClientException: org.tigris.subversion.javahl.ClientException: svn: E175011: Repository moved permanently to '/public'; please relocate svn: E175002: PROPFIND request failed on '/public'
In the above example, the command is "svn ls -r HEAD". You can run it via the command line to check the result returned from your SVN server.