Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29266

Remove need for special raw ingest task for DECam/CFHT

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Currently because MegaPrime and DECam use multi-extension FITS files to store data for all detectors in a single file, we need to have a special ingest task. It's easy to miss this and end up ingesting a single detector.

      Investigate an alternative scheme where a file header is read from FITS as now, the instrument is determined and then a method on the instrument class can return additional information (such as headers indexed by HDU) if needed.

        Attachments

          Issue Links

          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29265 Create DECam override file for ap.verify.Gen3DatasetIngestTask

          • Invalid
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29332 Support DECam/MegaPrime in metadata index/sidecar files

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Looking at the way ingest works (in the absence of an index file or sidecar) it currently does:

            1. Read the primary header
            2. Ask afw to read the default header (which might not be the first one)
            3. Merge headers.
            4. Run metadata translation
            5. Work out the instrument from the ObservationInfo
            6. Get the instrument class.
            7. Populate the exposure record from ObservationInfo

            An alternative approach could be:

            1. Read the primary header
            2. Try to translate it partially to get the instrument name
            3. Get the instrument class
            4. Ask the instrument class to read the headers, returning multiple ones if necessary.
            5. Calculate the ObservationInfo for each header.

            or

            1. Read the primary header
            2. Ask the metadata translator which translator class it should use
            3. Ask the translator class to read the header and return multiples if needed.
            4. Translate each header into ObservationInfo
            5. Get instrument class

            The advantage of the second is that this would embed the multiple HDU knowledge into a place that would allow it to be leveraged by the astrometadata write-index and write-sidecar commands. Putting it in the Instrument class completely separates it from that and I don't think we gain anything by doing so.

            Show
            tjenness Tim Jenness added a comment - Looking at the way ingest works (in the absence of an index file or sidecar) it currently does: Read the primary header Ask afw to read the default header (which might not be the first one) Merge headers. Run metadata translation Work out the instrument from the ObservationInfo Get the instrument class. Populate the exposure record from ObservationInfo An alternative approach could be: Read the primary header Try to translate it partially to get the instrument name Get the instrument class Ask the instrument class to read the headers, returning multiple ones if necessary. Calculate the ObservationInfo for each header. or Read the primary header Ask the metadata translator which translator class it should use Ask the translator class to read the header and return multiples if needed. Translate each header into ObservationInfo Get instrument class The advantage of the second is that this would embed the multiple HDU knowledge into a place that would allow it to be leveraged by the astrometadata write-index and write-sidecar commands. Putting it in the Instrument class completely separates it from that and I don't think we gain anything by doing so.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            One final comment is that we probably don't want to read each header twice so whatever API the translator class/Instrument class uses it has to be able to say "primary HDU is fine, you can use that".

            Show
            tjenness Tim Jenness added a comment - One final comment is that we probably don't want to read each header twice so whatever API the translator class/Instrument class uses it has to be able to say "primary HDU is fine, you can use that".
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I have this all working and can ingest with DECam and CFHT without needing the specialist ingest task.

            I encountered a bug in astropy with compressed image headers (like we have in testdata_cfht) but I work around it.

            I think I can remove the special tasks on this ticket as well. Krzysztof Findeisen are there any uses of the DECam ingest task that I would not spot in a normal lsst_distrib jenkins build?

            Show
            tjenness Tim Jenness added a comment - I have this all working and can ingest with DECam and CFHT without needing the specialist ingest task. I encountered a bug in astropy with compressed image headers (like we have in testdata_cfht) but I work around it. I think I can remove the special tasks on this ticket as well. Krzysztof Findeisen are there any uses of the DECam ingest task that I would not spot in a normal lsst_distrib jenkins build?
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Yes, DM-29265 adds the DECam ingest task to an ap_verify config. The config wouldn't be used by anything other than full ap_verify runs. I don't know of any other references that wouldn't show up in an lsst org search.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Yes, DM-29265 adds the DECam ingest task to an ap_verify config. The config wouldn't be used by anything other than full ap_verify runs. I don't know of any other references that wouldn't show up in an lsst org search.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Krzysztof Findeisen would you mind doing the review? It's a bit of code in metadata translator and minor changes elsewhere. I do change astrometadata translate to report all HDUs now.

            I have left the stubs of the separate raw ingest tasks around with a deprecation notice.

            Show
            tjenness Tim Jenness added a comment - Krzysztof Findeisen would you mind doing the review? It's a bit of code in metadata translator and minor changes elsewhere. I do change astrometadata translate to report all HDUs now. I have left the stubs of the separate raw ingest tasks around with a deprecation notice.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Jim Bosch, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.