Fixing "internal server error (500)" file upload failures

Users of one of our local Galaxy servers recently reported a problem with uploading files larger than a few tens of megabytes via the "Upload" interface, where the uploader would stop partway through with the message Warning: Internal server error (500).

For example:

By trial and error it was established that the maximum file size that the uploader could handle without this failure was around 65MB.

The particular server instance was running Galaxy release 19.05 and was configured to use nginx as the proxy; unfortunately there didn't appear to be any relevant error messages in the logs from either Galaxy or nginx.

However, looking at the sizes of the various logical volumes on the virtual machine hosting the Galaxy instance revealed a potential culprit:

# df -h /var
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/lv-var  2.0G  1.8G   65M  97% /var

i.e. the available space on the /var logical volume was around the same size as the maximum size for successful file upload. Additionally, monitoring the available space under /var while uploading a file it was possible to see it shrink (and then reset as the upload either completed or failed). So it appeared that this area was being used by nginx as temporary space for the file uploads before handing the data off to Galaxy.

The nginx configuration for this server didn't explicitly set this location, but the nginx documentation includes a directive called client_body_temp_path, which defines the directory for storing temporary files holding client request bodies.

Explicitly setting this directive (in the server block of the nginx configuration) to point to a location on the virtual machine (in this case under /tmp) with more available space seemed to fix the problem:

server {
    ...
    client_body_temp_path /tmp/nginx;
    ...