Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23213

It's necessary to restart the butler to see newly ingested files

    XMLWordPrintable

Details

    • Architecture
    • No

    Description

      A butler pointing at a registry fails to see newly ingested data. This is a major nuisance when observing as you are always interested in the data that you have just taken, and thus have to restart the butler after every exposure.

      Attachments

        Activity

          tjenness Tim Jenness added a comment -

          Is this something to do with how shared sqlite database work when two processes are attached?

          (why is this a pipe_tasks issue and not a daf_persistence issue?)

          tjenness Tim Jenness added a comment - Is this something to do with how shared sqlite database work when two processes are attached? (why is this a pipe_tasks issue and not a daf_persistence issue?)
          tjenness Tim Jenness added a comment -

          and is NFS involved? The SQLite FAQ indicates that everything should work so long as OODS is writing to a local file (not using NFS) and other butler users aren't ever writing to the sqlite file.

          tjenness Tim Jenness added a comment - and is NFS involved? The SQLite FAQ indicates that everything should work so long as OODS is writing to a local file (not using NFS) and other butler users aren't ever writing to the sqlite file.
          ktl Kian-Tat Lim added a comment -

          This is due to a misfeature in the code that explicitly avoids sharing a SQLite file (which should work over NFSv4).

          ktl Kian-Tat Lim added a comment - This is due to a misfeature in the code that explicitly avoids sharing a SQLite file (which should work over NFSv4).
          ktl Kian-Tat Lim added a comment -

          With Gen3 rapidly approaching, the hope is that this is not needed for very long and so any performance impacts for large-scale ingestion (which I don't think are needed in the Gen2 world) will not be noticeable.

          ktl Kian-Tat Lim added a comment - With Gen3 rapidly approaching, the hope is that this is not needed for very long and so any performance impacts for large-scale ingestion (which I don't think are needed in the Gen2 world) will not be noticeable.
          tjenness Tim Jenness added a comment -

          Looks like a nice cleanup. The tempfile each time completely explains the problem.

          tjenness Tim Jenness added a comment - Looks like a nice cleanup. The tempfile each time completely explains the problem.
          ktl Kian-Tat Lim added a comment -

          Tested on lsst-dev with manual ingest and found to work properly.

          ktl Kian-Tat Lim added a comment - Tested on lsst-dev with manual ingest and found to work properly.

          People

            ktl Kian-Tat Lim
            rhl Robert Lupton
            Tim Jenness
            Kian-Tat Lim, Merlin Fisher-Levine, Robert Lupton, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.