Mercurial-based tool installation issues in Galaxy 18.05

Recently I've encountered a subtle problem with tool installation after upgrading our local production instances to Galaxy release 18.05, which I'd like to document here in case it comes up again in future.

The problem manifests itself when attempting to install a tool from the main toolshed via the admin interface: after clicking Install, the tool installation status goes almost immediately to Error. Further inspection reveals that the tool repository hasn't been cloned to the local filesystem, and no dependencies are installed.

Frustratingly this fails to leave any error messages in the logs which might help to diagnose the cause. However attempting the install via the Galaxy API (using nebulizer) did return an error message:

Error cloning repository: [Errno 2] No such file or directory

I was able to track this down to the clone_repository function in  lib/tool_shed/util/hg_util.py, where it is issued when something goes wrong with the hg clone ... command used in the tool installation process. hg is the name of the Mercurial version control command, and essentially the problem was that this command couldn't be found by Galaxy.

Our local Galaxy installations are configured to use supervisor with uWSGI, with the Galaxy dependencies installed into a Python virtualenv. Since this virtualenv included Mercurial, I wondered why hg wasn't being picked up from there for the tool installation process.

Marius van den Beek offered some helpful insights via the galaxy-dev mailing list which clarified the situation:

Recent galaxy releases are using the `hg` command that should be automatically installed along with other galaxy dependencies.
If you're running galaxy in a virtualenv then that virtualenv should have the `hg` script in the bin folder.
Depending on how you start galaxy you may need to add the virtualenv's `bin` folder to the `PATH`.

Based on this it turned out that I needed to add an 'environment' parameter to the supervisor.ini for Galaxy file, which to specify the virtualenv to use and add its bin directory to the PATH - something like:

environment = VIRTUAL_ENV="/srv/galaxy/venv",PATH="/srv/galaxy/venv/bin:%(ENV_PATH)s"

(This parameter is mentioned in the installation documentation, in the Scaling and Load Balancing section, but only for configuring handler processes. However since our instances are using the uwsgi + mules strategy, it didn't occur to me that it would still be needed.)

Restarting Galaxy with the updated supervisor.ini file enabled tool installation to work without problems again.

Some closing asides:

• The problem can be masked if Mercurial is installed elsewhere on the system and is on the Galaxy user's PATH (for example /usr/bin/hg)
• If there is a system version but it is very old (for example Scientific Linux 7 has Mercurial 1.7) then it can cause a slightly different error in clone_repository, but the outcome and fix should be the same as above
• Since first encountering this issue I've come across a strange variant, whereby Mercurial is installed in the Galaxy virtualenv and supervisor is correctly configured but the tool installations still fail immediately. In this case for some unknown reason it turned out that the hg script in the virtualenv wasn't executable - adding 'execute' permission fixed this one.