Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24402

Handle "curated" datasets more generically in Gen 2 ap_verify

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:

      Description

      ap_verify currently ingests three kinds of data: raws, calibs, and defects. This works well for DECam, but HSC also has transmission curve datasets (optics, filter, and sensor). The only way to ingest them in Gen 2 appears to be to call a command-line script, installTransmissionCurves.py. It's not obvious how to incorporate this into DatasetIngestTask, given that it's supposed to be instrument-agnostic.

      Find a way to handle transmission curves effectively in Gen 2 without breaking support for other instruments like DECam.

      (Transmission curves will not be an issue in Gen 3 if we provide all calibs through a pre-ingested repository, and if not, we can use the obs.base.Instrument.writeCuratedCalibrations method to ingest them generically.)

        Attachments

          Issue Links

            Activity

            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-24341 [ 433028 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-24259 [ DM-24259 ]
            krzys Krzysztof Findeisen made changes -
            Description {{ap_verify}} currently ingests three kinds of data: raws, calibs, and defects. This works well for DECam, but HSC also has transmission curve datasets (optics, filter, and sensor). The only way to ingest them in Gen 2 appears to be to call a command-line script, [installTransmissionCurves.py|https://github.com/lsst/obs_subaru/blob/master/bin.src/installTransmissionCurves.py]. It's not obvious how to incorporate this into {{DatasetIngestTask}}, given that it's supposed to be instrument-agnostic.

            Find a way to handle transmission curves effectively in Gen 2 without breaking support for other instruments like DECam.

            (Transmission curves will not be an issue in Gen 3 if we provide all calibs through a pre-ingested repository, and if not, we can use the {{obs.base.Instrument}} class to ingest them generically.)
            {{ap_verify}} currently ingests three kinds of data: raws, calibs, and defects. This works well for DECam, but HSC also has transmission curve datasets (optics, filter, and sensor). The only way to ingest them in Gen 2 appears to be to call a command-line script, [installTransmissionCurves.py|https://github.com/lsst/obs_subaru/blob/master/bin.src/installTransmissionCurves.py]. It's not obvious how to incorporate this into {{DatasetIngestTask}}, given that it's supposed to be instrument-agnostic.

            Find a way to handle transmission curves effectively in Gen 2 without breaking support for other instruments like DECam.

            (Transmission curves will not be an issue in Gen 3 if we provide all calibs through a pre-ingested repository, and if not, we can use the {{obs.base.Instrument.writeCuratedCalibrations}} method to ingest them generically.)
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-24260 [ DM-24260 ]
            swinbank John Swinbank made changes -
            Epic Link DM-24341 [ 433028 ] DM-25145 [ 435263 ]
            swinbank John Swinbank made changes -
            Epic Link DM-25145 [ 435263 ] DM-26803 [ 439755 ]
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            krzys Krzysztof Findeisen made changes -
            Labels gen2-only

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Eric Bellm, John Swinbank, Krzysztof Findeisen, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: