Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21171

Create task to add local PhotoCalib and WCS value to source table rows

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      As part of the ScienceDataModel we need to have calibration products available for use in the Functors in the SDM code. One solution to making these products available is to compute their local values and store them per row in the afw tables output by various SciencePipelines tasks. This ticket will implement a task to perform this operation.

        Attachments

          Issue Links

            Activity

            cmorrison Chris Morrison created issue -
            cmorrison Chris Morrison made changes -
            Field Original Value New Value
            Component/s pipe_tasks [ 10726 ]
            Component/s meas_base [ 10750 ]
            cmorrison Chris Morrison made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            cmorrison Chris Morrison made changes -
            Link This issue is blocked by DM-21181 [ DM-21181 ]
            cmorrison Chris Morrison made changes -
            Reviewers Yusra AlSayyad [ yusra ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            cmorrison Chris Morrison added a comment -

            Jenkin: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30492/pipeline

            After review is complete I will make a ticket to vectorize/C++ the code if needed.

            Show
            cmorrison Chris Morrison added a comment - Jenkin: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30492/pipeline After review is complete I will make a ticket to vectorize/C++ the code if needed.
            Hide
            yusra Yusra AlSayyad added a comment -

            Pleasure to read. Let's change the name, before merging. I already like EvaluteLocalCalibration better than "StoreExposureCalibration." When the discussion on #dm-namingthings settles down, pick what you like from there.

            Show
            yusra Yusra AlSayyad added a comment - Pleasure to read. Let's change the name, before merging. I already like EvaluteLocalCalibration better than "StoreExposureCalibration." When the discussion on #dm-namingthings settles down, pick what you like from there.
            yusra Yusra AlSayyad made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            cmorrison Chris Morrison added a comment -

            EvaluateLocalCalibration sounds good to me. I'll fix the comments and merge. Jim and yourself where talking about using this to store local extinctions as well? In that case would you want this task to evolve into something more like a measurement tasks with plugins or is it fine as is for the future and we just add new methods?

            Show
            cmorrison Chris Morrison added a comment - EvaluateLocalCalibration sounds good to me. I'll fix the comments and merge. Jim and yourself where talking about using this to store local extinctions as well? In that case would you want this task to evolve into something more like a measurement tasks with plugins or is it fine as is for the future and we just add new methods?
            Hide
            yusra Yusra AlSayyad added a comment -

            We did chat and Jim convinced me that this particular task shouldn't load any other datasets. So galactic extinction will get applied later.

            This task is just taking advantage of the fact that the WCS, PhotoCalib, PSF, VisitInfo etc... is already in memory and extracting what we need from it so that we don't have to fetch it all from disk again to do the transform. I CAN envision these as plugins, but I think this your task will do the trick as-is.

            Looks good. Remember that the config doEvalLocCalibration IS the part that people will write a lot and usually from memory. I have to run, and will think about, what abbreviation I would type to doEvaluateLocalCalib

            Show
            yusra Yusra AlSayyad added a comment - We did chat and Jim convinced me that this particular task shouldn't load any other datasets. So galactic extinction will get applied later. This task is just taking advantage of the fact that the WCS, PhotoCalib, PSF, VisitInfo etc... is already in memory and extracting what we need from it so that we don't have to fetch it all from disk again to do the transform. I CAN envision these as plugins, but I think this your task will do the trick as-is. Looks good. Remember that the config doEvalLocCalibration IS the part that people will write a lot and usually from memory. I have to run, and will think about, what abbreviation I would type to doEvaluateLocalCalib
            Hide
            cmorrison Chris Morrison added a comment -

            One Jenkins job crashed on the error below, which looks to be a file not found. All other jobs passed. Gonna merge as this looks like a Jenkins error not a code error. 

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30514/pipeline

            // meas_base: tickets.DM-21171-g6504b5248a ......................................................................................:::::  [2019-09-23T22:50:06.787874Z] ========================= 240 passed in 119.91 seconds =========================
            :::::  [2019-09-23T22:50:06.794642Z] Coverage.py warning: No data was collected. (no-data-collected)
            :::::  [2019-09-23T22:50:06.914227Z] Global pytest run completed successfully
            :::::  [2019-09-23T22:50:07.047914Z] scons: done building targets.
            :::::  [2019-09-23T22:50:07.161181Z] + '[' -d tests/.tests ']'
            :::::  [2019-09-23T22:50:07.162090Z] ++ wc -l
            :::::  [2019-09-23T22:50:07.162265Z] ++ ls 'tests/.tests/*.failed'
            :::::  [2019-09-23T22:50:07.210820Z] + '[' 0 -ne 0 ']'
            :::::  [2019-09-23T22:50:07.210870Z] + eupspkg PRODUCT=meas_base VERSION=tickets.DM-21171-g6504b5248a FLAVOR=generic install
            :::::  [2019-09-23T22:50:07.211230Z] /j/ws/stack-os-matrix/centos-7.devtoolset-6.py3/lsstsw/build/meas_base/_build.sh: line 91: /j/ws/stack-os-matrix/centos-7.devtoolset-6.py3/lsstsw/eups/2.1.5/bin/eupspkg: No such file or directory
            ERROR (229 sec).
            

            Show
            cmorrison Chris Morrison added a comment - One Jenkins job crashed on the error below, which looks to be a file not found. All other jobs passed. Gonna merge as this looks like a Jenkins error not a code error.  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30514/pipeline // meas_base: tickets.DM-21171-g6504b5248a ......................................................................................::::: [2019-09-23T22:50:06.787874Z] ========================= 240 passed in 119.91 seconds ========================= ::::: [ 2019 - 09 -23T22: 50 : 06 .794642Z] Coverage.py warning: No data was collected. (no-data-collected) ::::: [ 2019 - 09 -23T22: 50 : 06 .914227Z] Global pytest run completed successfully ::::: [ 2019 - 09 -23T22: 50 : 07 .047914Z] scons: done building targets. ::::: [ 2019 - 09 -23T22: 50 : 07 .161181Z] + '[' -d tests/.tests ']' ::::: [ 2019 - 09 -23T22: 50 : 07 .162090Z] ++ wc -l ::::: [ 2019 - 09 -23T22: 50 : 07 .162265Z] ++ ls 'tests/.tests/*.failed' ::::: [ 2019 - 09 -23T22: 50 : 07 .210820Z] + '[' 0 -ne 0 ']' ::::: [ 2019 - 09 -23T22: 50 : 07 .210870Z] + eupspkg PRODUCT=meas_base VERSION=tickets.DM- 21171 -g6504b5248a FLAVOR=generic install ::::: [ 2019 - 09 -23T22: 50 : 07 .211230Z] /j/ws/stack-os-matrix/centos- 7 .devtoolset- 6 .py3/lsstsw/build/meas_base/_build.sh: line 91 : /j/ws/stack-os-matrix/centos- 7 .devtoolset- 6 .py3/lsstsw/eups/ 2.1 . 5 /bin/eupspkg: No such file or directory ERROR ( 229 sec).
            cmorrison Chris Morrison made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Labels SciencePipelines
            swinbank John Swinbank made changes -
            Epic Link DM-21442 [ 423049 ]
            swinbank John Swinbank made changes -
            Team Alert Production [ 10300 ]

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Chris Morrison, Eric Bellm, Jim Bosch, John Swinbank, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: