Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21864

Extend Gen3 butler support for obs_lsst

    XMLWordPrintable

Details

    • 8
    • Architecture

    Description

      obs_lsst now has basic Gen3 butler support (sufficient for ingest), but we'll need to expand that to get to the point where we can run single-frame processing (IsrTask, CharacterizeImageTask, and CalibrateTask) via Gen3 on imsim/phosim and IsrTask on all other instruments (or at least the ones for which this is possible in Gen2).  That will include, at least:

      • implementing writeCuratedCalibrations in the instrument class;
      • specializing (and modifying, as necessary) the instrument-specific hooks for the gen2to3 tool, at least to the point where we can write scripts like the ones in ci_hsc_gen3/bin.src to make a fully-usable Gen3 repo.

       

      Attachments

        Issue Links

          Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Description obs_decam now has basic Gen3 butler support (sufficient for ingest), but we'll need to expand that to get to the point where we can run single-frame processing (IsrTask, CharacterizeImageTask, and CalibrateTask) via Gen3 on imsim/phosim and IsrTask on all other instruments (or at least the ones for which this is possible in Gen2).  That will include, at least:
             * implementing {{writeCuratedCalibrations}} in the instrument class;
             * specializing (and modifying, as necessary) the instrument-specific hooks for the gen2to3 tool, at least to the point where we can write scripts like the ones in ci_hsc_gen3/bin.src to make a fully-usable Gen3 repo.

             
            obs_lsst now has basic Gen3 butler support (sufficient for ingest), but we'll need to expand that to get to the point where we can run single-frame processing (IsrTask, CharacterizeImageTask, and CalibrateTask) via Gen3 on imsim/phosim and IsrTask on all other instruments (or at least the ones for which this is possible in Gen2).  That will include, at least:
             * implementing {{writeCuratedCalibrations}} in the instrument class;
             * specializing (and modifying, as necessary) the instrument-specific hooks for the gen2to3 tool, at least to the point where we can write scripts like the ones in ci_hsc_gen3/bin.src to make a fully-usable Gen3 repo.

             
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            Parejkoj John Parejko made changes -
            Link This issue is blocked by DM-22655 [ DM-22655 ]
            tjenness Tim Jenness made changes -
            Watchers Bryce Kalmbach, Dino Bektesevic, Heather Kelly, Jim Bosch [ Bryce Kalmbach, Dino Bektesevic, Heather Kelly, Jim Bosch ] Bryce Kalmbach, Dino Bektesevic, Heather Kelly, Jim Bosch, John Parejko [ Bryce Kalmbach, Dino Bektesevic, Heather Kelly, Jim Bosch, John Parejko ]
            tjenness Tim Jenness added a comment -

            I think this ticket may be obsolete given what I've just done on DM-23778. writeCuratedCalibrations works and gen2to3 works. The scripts in ci_hsc_gen3 look like they would work as well if they could take an instrument class as a parameter.

            tjenness Tim Jenness added a comment - I think this ticket may be obsolete given what I've just done on DM-23778 . writeCuratedCalibrations works and gen2to3 works. The scripts in ci_hsc_gen3 look like they would work as well if they could take an instrument class as a parameter.
            tjenness Tim Jenness made changes -
            Link This issue is triggered by DM-23778 [ DM-23778 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggered by DM-23778 [ DM-23778 ]
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-23778 [ DM-23778 ]
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-23976 [ DM-23976 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Invalid [ 11005 ]
            tjenness Tim Jenness added a comment -

            The one problem remaining is not a gen3 problem as such, it's that the filters defined for LSSTCam do not match any of those used in BOT so gen3 raw ingest is impossible with BOT data. No-one is treating fixing this as a high priority so we may have to do it. See DM-21706.

            tjenness Tim Jenness added a comment - The one problem remaining is not a gen3 problem as such, it's that the filters defined for LSSTCam do not match any of those used in BOT so gen3 raw ingest is impossible with BOT data. No-one is treating fixing this as a high priority so we may have to do it. See DM-21706 .
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-21706 [ DM-21706 ]

            People

              tjenness Tim Jenness
              jbosch Jim Bosch
              Bryce Kalmbach, Dino Bektesevic, Heather Kelly, Jim Bosch, John Parejko, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.