Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16467

isrTask conversion to pipelineTask

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None

      Description

      isrTask needs to be updated to work with a Gen3 butler as a valid pipelineTask.  DM-15862 will unify the ISR processing, so this should make all cameras functional with a Gen3 butler.

        Attachments

        1. add_bfKernel.sh
          0.7 kB
          Christopher Waters
        2. add_defects.sh
          188 kB
          Christopher Waters
        3. calibtool.py
          9 kB
          Christopher Waters
        4. pickletool.py
          0.6 kB
          Christopher Waters

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment -

            Working from the DM-15862 ticket branch, I've created a DM-16467 with an incomplete solution.  Currently, there are three issues:

            1. raw data is loaded as a DecoratedImageU, but IsrTask expects it to be an Exposure.  Converting the received DecoratedImageU into an Exposure doesn't work, as the generated Exposure does not return a usable VisitInfo, and does not have a Detector associated.
            2. Calibration data is not correctly returned by the query to the gen3 database.  A query/subquery of the form "select * from Exposure JOIN Dataset ON (valid_first < datetime_begin AND valid_last > datetime_end) WHERE Exposure.exposure = 903988 AND detector = 16 and dataset_type_name = 'bias';" needs to be done to get the correct (in this case) bias.  This will eventually supersede the IsrTask.readIsrData/IsrTask.getIsrExposure code, which currently fetches ~12 images/products.
            3. Data queries that result in a single quanta to run (such as query two below, which restricts down to a single detector) fail to run as something in the code expects an array.  Removing the "Detector.detector=16" constraint allows it to continue to IsrTask.run().

            Beyond these two issues, I can use the debugger to check that the code is reaching IsrTask.run(), and the graphs constructed for the raw input exposure look plausible based on my tests on ci_hsc.

            Test commands for reference:

            stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" qgraph -t IsrTask --instrument-overrides --qgraph-dot ~/x.dot

            stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" run -t IsrTask --instrument-overrides

             

             

            Show
            czw Christopher Waters added a comment - Working from the DM-15862 ticket branch, I've created a DM-16467 with an incomplete solution.  Currently, there are three issues: raw data is loaded as a DecoratedImageU, but IsrTask expects it to be an Exposure.  Converting the received DecoratedImageU into an Exposure doesn't work, as the generated Exposure does not return a usable VisitInfo, and does not have a Detector associated. Calibration data is not correctly returned by the query to the gen3 database.  A query/subquery of the form "select * from Exposure JOIN Dataset ON (valid_first < datetime_begin AND valid_last > datetime_end) WHERE Exposure.exposure = 903988 AND detector = 16 and dataset_type_name = 'bias';" needs to be done to get the correct (in this case) bias.  This will eventually supersede the IsrTask.readIsrData/IsrTask.getIsrExposure code, which currently fetches ~12 images/products. Data queries that result in a single quanta to run (such as query two below, which restricts down to a single detector) fail to run as something in the code expects an array.  Removing the "Detector.detector=16" constraint allows it to continue to IsrTask.run(). Beyond these two issues, I can use the debugger to check that the code is reaching IsrTask.run(), and the graphs constructed for the raw input exposure look plausible based on my tests on ci_hsc. Test commands for reference: stac --clobber-config --clobber-versions -b ~/dev/ DM-16467 /ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" qgraph -t IsrTask --instrument-overrides --qgraph-dot ~/x.dot stac --clobber-config --clobber-versions -b ~/dev/ DM-16467 /ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" run -t IsrTask --instrument-overrides    
            Hide
            salnikov Andy Salnikov added a comment -

            Last point was caused by a bug in passing arguments to registry.associate() method. Fixed by DM-16481, already in pipe_supertask master, you can clone it if you prefer not to wait until next weekly release.

            Show
            salnikov Andy Salnikov added a comment - Last point was caused by a bug in passing arguments to registry.associate() method. Fixed by  DM-16481 , already in pipe_supertask master, you can clone it if you prefer not to wait until next weekly release.
            Hide
            salnikov Andy Salnikov added a comment -

            For calibrations we need to add special handling of ExposureRange unit to pre-flight. I have made DM-16482 to work on that, it is of non-trivial kind but I hope to close it this week.

            Show
            salnikov Andy Salnikov added a comment - For calibrations we need to add special handling of ExposureRange unit to pre-flight. I have made  DM-16482 to work on that, it is of non-trivial kind but I hope to close it this week.
            Hide
            salnikov Andy Salnikov added a comment -

            Christopher Waters, I have just merged daf_butler DM-16482 branch into master and that should take care of the calibration issue, flat/bias should now be correctly found and added to Quanta. Note that units for those should be:

            bias: Instrument, Detector, ExposureRange
            flat: Instrument, Detector, PhysicalFilter, ExposureRange
            

            Show
            salnikov Andy Salnikov added a comment - Christopher Waters , I have just merged daf_butler DM-16482 branch into master and that should take care of the calibration issue, flat/bias should now be correctly found and added to Quanta. Note that units for those should be: bias: Instrument, Detector, ExposureRange flat: Instrument, Detector, PhysicalFilter, ExposureRange
            Hide
            czw Christopher Waters added a comment -

            The complete ISR process does not work correctly at this time for this ticket branch:

            • thumbnails do not have the needed daf_butler formatter to be written (DM-17381)
            • fringe data for y-filter data is not read correctly (DM-17152)
            • crosstalk sources are not generated for inter-CCD sources (DM-17169)

            These issues do not prevent ci_hsc data from running, although additional calibration data must be ingested to add the brighter-fatter kernel and the HSC defects.  I have been adding those products using the attached python script, which will need to be superseded by a more permanent solution.  The commands to ingest products are:

             

             

            calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/bfKernel/brighter_fatter_kernel.pkl `pwd`/ci_hsc/DATA bfKernel --valid_first '1977-01-01T00:00:00.000001' --valid_last '2095-01-01T00:00:00.000001' --instrument HSC --init
            calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/2013-01-31/defects_0.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --init --detector 0
            calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/2013-01-31/defects_1.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --detector 1
            

            The gen2 behavior has not changed, and a comparison of output files shows that both generations have pixel-identical results, with the exception of changes in mask plane labeling.

             

            Show
            czw Christopher Waters added a comment - The complete ISR process does not work correctly at this time for this ticket branch: thumbnails do not have the needed daf_butler formatter to be written ( DM-17381 ) fringe data for y-filter data is not read correctly ( DM-17152 ) crosstalk sources are not generated for inter-CCD sources ( DM-17169 ) These issues do not prevent ci_hsc data from running, although additional calibration data must be ingested to add the brighter-fatter kernel and the HSC defects.  I have been adding those products using the attached python script, which will need to be superseded by a more permanent solution.  The commands to ingest products are:     calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/bfKernel/brighter_fatter_kernel.pkl `pwd`/ci_hsc/DATA bfKernel --valid_first '1977-01-01T00:00:00.000001' --valid_last '2095-01-01T00:00:00.000001' --instrument HSC --init calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/ 2013 - 01 - 31 /defects_0.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --init --detector 0 calibtool.py ./ci_hsc/DATA/gen3.sqlite3 `pwd`/ci_hsc/DATA/calib/defects/ 2013 - 01 - 31 /defects_1.fits `pwd`/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --detector 1 The gen2 behavior has not changed, and a comparison of output files shows that both generations have pixel-identical results, with the exception of changes in mask plane labeling.  
            Hide
            krughoff Simon Krughoff added a comment - - edited

            Looks pretty good to me. I have put some comments in the PR. They are mostly suggestion level.

            The most concerning thing to me is the hard coding of obs_subaru information, but, as is mentioned on this ticket, we are still shimming some aspect of things.

            Show
            krughoff Simon Krughoff added a comment - - edited Looks pretty good to me. I have put some comments in the PR. They are mostly suggestion level. The most concerning thing to me is the hard coding of obs_subaru information, but, as is mentioned on this ticket, we are still shimming some aspect of things.
            Hide
            czw Christopher Waters added a comment - - edited

            To be clear:  while testing ISR on `ci_hsc`, I often needed to start over from scratch and try again.  To make this easier and keep things consistent, I migrated as much as possible into script files.  This meant I could regenerate a usable `ci_hsc` gen3 repository for ISR by running:

             

            #Clear old repo:
            rm ./ci_hsc/DATA/gen3.sqlite3 ./ci_hsc/DATA/butler.yaml
            #Regenerate repo:
            cd ./ci_hsc && makeButlerRepo.py DATA && python ./bin/gen3.py
            #Ingest calibs:
            bash ./add_bfKernel.sh
            bash ./add_defects.sh
            

             The two bash scripts encapsulate the `calibtool` commands for both calib types, and converts the brighter-fatter kernel to a python3 pickle using `pickletool`.

            Show
            czw Christopher Waters added a comment - - edited To be clear:  while testing ISR on `ci_hsc`, I often needed to start over from scratch and try again.  To make this easier and keep things consistent, I migrated as much as possible into script files.  This meant I could regenerate a usable `ci_hsc` gen3 repository for ISR by running:   #Clear old repo: rm ./ci_hsc/DATA/gen3.sqlite3 ./ci_hsc/DATA/butler.yaml #Regenerate repo: cd ./ci_hsc && makeButlerRepo.py DATA && python ./bin/gen3.py #Ingest calibs: bash ./add_bfKernel.sh bash ./add_defects.sh  The two bash scripts encapsulate the `calibtool` commands for both calib types, and converts the brighter-fatter kernel to a python3 pickle using `pickletool`.

              People

              • Assignee:
                czw Christopher Waters
                Reporter:
                czw Christopher Waters
                Reviewers:
                Simon Krughoff
                Watchers:
                Andy Salnikov, Christopher Waters, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel