The problem is that the firefly-work/firefly/upload directory in the firefly docker container running on lsst-demo fills up with files sent by afwDisplay. There is a spawned shell script (/usr/local/tomcat-base/cleanup.sh) that attempts to clean up this directory, but it only executes once a day and cleans up files that are older than a week. Since this instance is being used as a live image monitor, hundreds or even thousands of 64 MB images are being uploaded every day, rapidly filling the 75GB filesystem.

One potential solution is to make the cleanup script much more aggressive, deleting files older than an hour or two and running as frequently as every hour.

I manually modified the cleanup.sh script inside the container to sleep for only 2 hours rather than 24 and remove anything in upload that is more than 120 minutes old.

Since this is modifying the live instance, it is not expected to survive the next redeployment.