Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21171

Create task to add local PhotoCalib and WCS value to source table rows

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      As part of the ScienceDataModel we need to have calibration products available for use in the Functors in the SDM code. One solution to making these products available is to compute their local values and store them per row in the afw tables output by various SciencePipelines tasks. This ticket will implement a task to perform this operation.

        Attachments

          Issue Links

            Activity

            Hide
            cmorrison Chris Morrison added a comment -

            One Jenkins job crashed on the error below, which looks to be a file not found. All other jobs passed. Gonna merge as this looks like a Jenkins error not a code error. 

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30514/pipeline

            // meas_base: tickets.DM-21171-g6504b5248a ......................................................................................:::::  [2019-09-23T22:50:06.787874Z] ========================= 240 passed in 119.91 seconds =========================
            :::::  [2019-09-23T22:50:06.794642Z] Coverage.py warning: No data was collected. (no-data-collected)
            :::::  [2019-09-23T22:50:06.914227Z] Global pytest run completed successfully
            :::::  [2019-09-23T22:50:07.047914Z] scons: done building targets.
            :::::  [2019-09-23T22:50:07.161181Z] + '[' -d tests/.tests ']'
            :::::  [2019-09-23T22:50:07.162090Z] ++ wc -l
            :::::  [2019-09-23T22:50:07.162265Z] ++ ls 'tests/.tests/*.failed'
            :::::  [2019-09-23T22:50:07.210820Z] + '[' 0 -ne 0 ']'
            :::::  [2019-09-23T22:50:07.210870Z] + eupspkg PRODUCT=meas_base VERSION=tickets.DM-21171-g6504b5248a FLAVOR=generic install
            :::::  [2019-09-23T22:50:07.211230Z] /j/ws/stack-os-matrix/centos-7.devtoolset-6.py3/lsstsw/build/meas_base/_build.sh: line 91: /j/ws/stack-os-matrix/centos-7.devtoolset-6.py3/lsstsw/eups/2.1.5/bin/eupspkg: No such file or directory
            ERROR (229 sec).
            

            Show
            cmorrison Chris Morrison added a comment - One Jenkins job crashed on the error below, which looks to be a file not found. All other jobs passed. Gonna merge as this looks like a Jenkins error not a code error.  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30514/pipeline // meas_base: tickets.DM-21171-g6504b5248a ......................................................................................::::: [2019-09-23T22:50:06.787874Z] ========================= 240 passed in 119.91 seconds ========================= ::::: [ 2019 - 09 -23T22: 50 : 06 .794642Z] Coverage.py warning: No data was collected. (no-data-collected) ::::: [ 2019 - 09 -23T22: 50 : 06 .914227Z] Global pytest run completed successfully ::::: [ 2019 - 09 -23T22: 50 : 07 .047914Z] scons: done building targets. ::::: [ 2019 - 09 -23T22: 50 : 07 .161181Z] + '[' -d tests/.tests ']' ::::: [ 2019 - 09 -23T22: 50 : 07 .162090Z] ++ wc -l ::::: [ 2019 - 09 -23T22: 50 : 07 .162265Z] ++ ls 'tests/.tests/*.failed' ::::: [ 2019 - 09 -23T22: 50 : 07 .210820Z] + '[' 0 -ne 0 ']' ::::: [ 2019 - 09 -23T22: 50 : 07 .210870Z] + eupspkg PRODUCT=meas_base VERSION=tickets.DM- 21171 -g6504b5248a FLAVOR=generic install ::::: [ 2019 - 09 -23T22: 50 : 07 .211230Z] /j/ws/stack-os-matrix/centos- 7 .devtoolset- 6 .py3/lsstsw/build/meas_base/_build.sh: line 91 : /j/ws/stack-os-matrix/centos- 7 .devtoolset- 6 .py3/lsstsw/eups/ 2.1 . 5 /bin/eupspkg: No such file or directory ERROR ( 229 sec).
            Hide
            yusra Yusra AlSayyad added a comment -

            We did chat and Jim convinced me that this particular task shouldn't load any other datasets. So galactic extinction will get applied later.

            This task is just taking advantage of the fact that the WCS, PhotoCalib, PSF, VisitInfo etc... is already in memory and extracting what we need from it so that we don't have to fetch it all from disk again to do the transform. I CAN envision these as plugins, but I think this your task will do the trick as-is.

            Looks good. Remember that the config doEvalLocCalibration IS the part that people will write a lot and usually from memory. I have to run, and will think about, what abbreviation I would type to doEvaluateLocalCalib

            Show
            yusra Yusra AlSayyad added a comment - We did chat and Jim convinced me that this particular task shouldn't load any other datasets. So galactic extinction will get applied later. This task is just taking advantage of the fact that the WCS, PhotoCalib, PSF, VisitInfo etc... is already in memory and extracting what we need from it so that we don't have to fetch it all from disk again to do the transform. I CAN envision these as plugins, but I think this your task will do the trick as-is. Looks good. Remember that the config doEvalLocCalibration IS the part that people will write a lot and usually from memory. I have to run, and will think about, what abbreviation I would type to doEvaluateLocalCalib
            Hide
            cmorrison Chris Morrison added a comment -

            EvaluateLocalCalibration sounds good to me. I'll fix the comments and merge. Jim and yourself where talking about using this to store local extinctions as well? In that case would you want this task to evolve into something more like a measurement tasks with plugins or is it fine as is for the future and we just add new methods?

            Show
            cmorrison Chris Morrison added a comment - EvaluateLocalCalibration sounds good to me. I'll fix the comments and merge. Jim and yourself where talking about using this to store local extinctions as well? In that case would you want this task to evolve into something more like a measurement tasks with plugins or is it fine as is for the future and we just add new methods?
            Hide
            yusra Yusra AlSayyad added a comment -

            Pleasure to read. Let's change the name, before merging. I already like EvaluteLocalCalibration better than "StoreExposureCalibration." When the discussion on #dm-namingthings settles down, pick what you like from there.

            Show
            yusra Yusra AlSayyad added a comment - Pleasure to read. Let's change the name, before merging. I already like EvaluteLocalCalibration better than "StoreExposureCalibration." When the discussion on #dm-namingthings settles down, pick what you like from there.
            Hide
            cmorrison Chris Morrison added a comment -

            Jenkin: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30492/pipeline

            After review is complete I will make a ticket to vectorize/C++ the code if needed.

            Show
            cmorrison Chris Morrison added a comment - Jenkin: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30492/pipeline After review is complete I will make a ticket to vectorize/C++ the code if needed.

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Chris Morrison, Eric Bellm, Jim Bosch, John Swinbank, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: