Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21049

Ensure automatic ingest of BOT data taken at SLAC and rsync'ed to NCSA

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Team:
      Data Facility

      Description

      HI, we will soon (maybe second week of September) be starting to take new BOT data at SLAC, with the expectation that this data will be rsynch'ed to NCSA (the so called 9-raft testing at SLAC). This is a follow-on to early "2-ETU" testing at SLAC, where the data was rsynch'ed to NCSA, but as far as I know not ingested. I have seen discussion from various people which seems to indicate it should be possible to automate the ingest process (using Gen2 butler), so the goal of this ticket is to increase the probability that the right people are talking to each other to maximize the probability of this ingest working in time for 9-raft data (hence the large number of people added to the watchers list, sorry if I missed people, or added people who should not be involved). The 2-ETU data is as far as I know still not ingested, so it could also be used as a test case.

      One additional wrinkle, although we expect to have 9 science rafts and 4 corner rafts installed (+-1) due to disk space/performance reasons we may choose to not read out all rafts for all images, in which case there will be no FITS files for the rafts which are not read out. I am unclear if this is easy for the butler to handle.

        Attachments

          Activity

          Hide
          jchiang James Chiang added a comment -

          1) I don't know if we have plans to fix the headers for the data downloaded so far, so Tony Johnson may wish to comment.  New data (as of today at least) should have fixed headers.

           

          2) Yes, using the tickets/DM-21169 branch of obs_lsst should work with w_2019_37 or later.   I would recommend using that branch until it gets merged and that work is available in a new weekly.

          Show
          jchiang James Chiang added a comment - 1) I don't know if we have plans to fix the headers for the data downloaded so far, so Tony Johnson may wish to comment.  New data (as of today at least) should have fixed headers.   2) Yes, using the tickets/ DM-21169 branch of obs_lsst should work with w_2019_37 or later.   I would recommend using that branch until it gets merged and that work is available in a new weekly.
          Hide
          tjohnson Tony Johnson added a comment -

          I did not patch the headers on the older 9-raft data. It should  be fairly easy, but I am inclined to wait until someone asks for that data at NCSA before doing anything, I suspect the newer data is more useful anyway.

          Show
          tjohnson Tony Johnson added a comment - I did not patch the headers on the older 9-raft data. It should  be fairly easy, but I am inclined to wait until someone asks for that data at NCSA before doing anything, I suspect the newer data is more useful anyway.
          Hide
          emorganson Eric Morganson added a comment -

          Great. I will update our ingestion pipeline to use 21169 first thing Monday. I will also try to ingest the last few days of images as appropriate. I just got this half-setup on the NCSA machines, but don't have time to test it before I leave for the weekend.

          Show
          emorganson Eric Morganson added a comment - Great. I will update our ingestion pipeline to use 21169 first thing Monday. I will also try to ingest the last few days of images as appropriate. I just got this half-setup on the NCSA machines, but don't have time to test it before I leave for the weekend.
          Hide
          tjenness Tim Jenness added a comment -

          What's the completion criterion for this ticket?

          Show
          tjenness Tim Jenness added a comment - What's the completion criterion for this ticket?
          Hide
          tjohnson Tony Johnson added a comment - - edited

          I would say:
          1) Take BOT data with full 25-raft focal-plane
          2) Verify it is rsynced and ingested at NCSA, and can be analyzed

          Unfortunately 1) is not yet achieved, although we did come very close this past weekend, and could use the data (currently not in the standard rawData location) as a test if that is useful.

          Show
          tjohnson Tony Johnson added a comment - - edited I would say: 1) Take BOT data with full 25-raft focal-plane 2) Verify it is rsynced and ingested at NCSA, and can be analyzed Unfortunately 1) is not yet achieved, although we did come very close this past weekend, and could use the data (currently not in the standard rawData location) as a test if that is useful.

            People

            • Assignee:
              Unassigned
              Reporter:
              tjohnson Tony Johnson
              Watchers:
              Aaron Roodman, Eric Charles, Eric Morganson, Glenn Morris, James Chiang, Kian-Tat Lim, Margaret Gelman, Michelle Butler, Michelle Gower, Robert Gruendl, Robert Lupton, Seth Digel, Tim Jenness, Tony Johnson, Wil O'Mullane
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:

                Summary Panel