Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28886

Disk fillling up regularly on firefly server docker image

    XMLWordPrintable

Details

    • Story
    • Status: To Do
    • Resolution: Unresolved
    • None
    • Firefly
    • None
    • Science User Interface
    • No

    Description

      I have no idea if that's even a reasonable description, but after viewing too many images with Firefly at NCSA (doesn't seem to be affecting the summit) we get Upload Errors due to the cache being filled.

      This is a problem for observing, and has been discussed a bit in #dm-suit-firefly-dev (Jan 21, Jan 22, and now Feb 18 2021)

      Attachments

        Activity

          ktl Kian-Tat Lim added a comment -

          The problem is that the firefly-work/firefly/upload directory in the firefly docker container running on lsst-demo fills up with files sent by afwDisplay. There is a spawned shell script (/usr/local/tomcat-base/cleanup.sh) that attempts to clean up this directory, but it only executes once a day and cleans up files that are older than a week. Since this instance is being used as a live image monitor, hundreds or even thousands of 64 MB images are being uploaded every day, rapidly filling the 75GB filesystem.

          One potential solution is to make the cleanup script much more aggressive, deleting files older than an hour or two and running as frequently as every hour.

          ktl Kian-Tat Lim added a comment - The problem is that the firefly-work/firefly/upload directory in the firefly docker container running on lsst-demo fills up with files sent by afwDisplay . There is a spawned shell script ( /usr/local/tomcat-base/cleanup.sh ) that attempts to clean up this directory, but it only executes once a day and cleans up files that are older than a week. Since this instance is being used as a live image monitor, hundreds or even thousands of 64 MB images are being uploaded every day, rapidly filling the 75GB filesystem. One potential solution is to make the cleanup script much more aggressive, deleting files older than an hour or two and running as frequently as every hour.
          ktl Kian-Tat Lim added a comment -

          I manually modified the cleanup.sh script inside the container to sleep for only 2 hours rather than 24 and remove anything in upload that is more than 120 minutes old.

          Since this is modifying the live instance, it is not expected to survive the next redeployment.

          ktl Kian-Tat Lim added a comment - I manually modified the cleanup.sh script inside the container to sleep for only 2 hours rather than 24 and remove anything in upload that is more than 120 minutes old. Since this is modifying the live instance, it is not expected to survive the next redeployment.

          People

            gpdf Gregory Dubois-Felsmann
            mfisherlevine Merlin Fisher-Levine
            Kian-Tat Lim, Merlin Fisher-Levine
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Jenkins

                No builds found.