Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20695

Coadd in Gen3 doesn't have a Selector like the one in Gen2

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DRP S21a (Dec Jan)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Two patches (patch=28 & patch=72) in Gen3 assembleCoadd HSC-RC2 tract=9615 ran into the following error

      "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 777, in detectFootprints
          psf = self.getPsf(exposure, sigma=sigma)
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 503, in getPsf
          raise RuntimeError("Unable to determine PSF to use for detection: no sigma provided")
      RuntimeError: Unable to determine PSF to use for detection: no sigma provided
      

      Yusra AlSayyad identified that the Gen3 version doesn't have a Selector like how it is in Gen2. For patch=28 (Gen3) or '1,3' (Gen2), visit=27116 isn't selected for coadd and that warp isn't made in Gen2.
      From Slack:

      I’m 90% sure that the selector in `CoaddDriver` https://github.com/lsst/pipe_drivers/blob/master/python/lsst/pipe/drivers/coaddDriver.py#L268, (retargeted to `from lsst.pipe.tasks.selectImages import PsfWcsSelectImagesTask`) but am just shoving the dataRefs into the right dataId container that the selector likes… hang on

      Confirmed. `PsfWcsSelectImagesTask` filters out visit 27116 in `coaddDriver` and the Gen2 version of assembleCoadd if you ask it to use that selector.

       
      And attaching her useful snippet from Slack.

      One way to reproduce the error in Gen3 (but likely with Oracle permission issue) is 

      pipetask -b /project/hchiang2/gen3repos/w_2019_20/repo/butler.yaml -d "tract=9615 and patch=28 and abstract_filter='z'" -i G3M19c_000001_021 -o dummy run --skip-init-writes  -t "assembleCoadd.CompareWarpAssembleCoaddTask"
      

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Description Two patches (patch=28 & patch=72) in Gen3 assembleCoadd HSC-RC2 tract=9615 ran into the following error
            {noformat}
            "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 777, in detectFootprints
                psf = self.getPsf(exposure, sigma=sigma)
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 503, in getPsf
                raise RuntimeError("Unable to determine PSF to use for detection: no sigma provided")
            RuntimeError: Unable to determine PSF to use for detection: no sigma provided
            {noformat}

            [~yusra] identified that the Gen3 version doesn't have a Selector like how it is in Gen2. For patch=28 (Gen3) or '1,3' (Gen2), visit=27116 isn't selected for coadd and that warp isn't made in Gen2.
            From Slack:
            bq.
            I’m 90% sure that the selector in `CoaddDriver` https://github.com/lsst/pipe_drivers/blob/master/python/lsst/pipe/drivers/coaddDriver.py#L268, (retargeted to `from lsst.pipe.tasks.selectImages import PsfWcsSelectImagesTask`) but am just shoving the dataRefs into the right dataId container that the selector likes… hang on

            Confirmed. `PsfWcsSelectImagesTask` filters out visit 27116 in `coaddDriver` and the Gen2 version of assembleCoadd if you ask it to use that selector.

             
            And attaching her useful snippet from Slack.


            One way to reproduce the error in Gen3 (but likely with Oracle permission issue) is 
            {noformat}
            pipetask -b /project/hchiang2/gen3repos/w_2019_20/repo/butler.yaml -d "tract=9615 and patch=28 and abstract_filter='z'" -i G3M19c_000001_021 -o dummy run --skip-init-writes -t "assembleCoadd.CompareWarpAssembleCoaddTask"
            {noformat}
            Two patches (patch=28 & patch=72) in Gen3 assembleCoadd HSC-RC2 tract=9615 ran into the following error
            {noformat}"/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 777, in detectFootprints
                psf = self.getPsf(exposure, sigma=sigma)
              File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/18.0.0-5-ga38416e7+2/python/lsst/meas/algorithms/detection.py", line 503, in getPsf
                raise RuntimeError("Unable to determine PSF to use for detection: no sigma provided")
            RuntimeError: Unable to determine PSF to use for detection: no sigma provided
            {noformat}
            [~yusra] identified that the Gen3 version doesn't have a Selector like how it is in Gen2. For patch=28 (Gen3) or '1,3' (Gen2), visit=27116 isn't selected for coadd and that warp isn't made in Gen2.
             From Slack:
            {quote}I’m 90% sure that the selector in `CoaddDriver` [https://github.com/lsst/pipe_drivers/blob/master/python/lsst/pipe/drivers/coaddDriver.py#L268], (retargeted to `from lsst.pipe.tasks.selectImages import PsfWcsSelectImagesTask`) but am just shoving the dataRefs into the right dataId container that the selector likes… hang on

            Confirmed. `PsfWcsSelectImagesTask` filters out visit 27116 in `coaddDriver` and the Gen2 version of assembleCoadd if you ask it to use that selector.
            {quote}
             
             And attaching her useful snippet from Slack.

            One way to reproduce the error in Gen3 (but likely with Oracle permission issue) is 
            {noformat}pipetask -b /project/hchiang2/gen3repos/w_2019_20/repo/butler.yaml -d "tract=9615 and patch=28 and abstract_filter='z'" -i G3M19c_000001_021 -o dummy run --skip-init-writes -t "assembleCoadd.CompareWarpAssembleCoaddTask"
            {noformat}
            jbosch Jim Bosch made changes -
            Component/s pipe_tasks [ 10726 ]
            Team Data Release Production [ 10301 ]
            Labels gen3-middleware
            jbosch Jim Bosch made changes -
            Story Points 6
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            yusra Yusra AlSayyad made changes -
            Assignee Yusra AlSayyad [ yusra ]
            gruendl Robert Gruendl [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25781 ]
            yusra Yusra AlSayyad made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            yusra Yusra AlSayyad made changes -
            Epic Link PREOPS-281 [ 442892 ]
            Sprint DRP S21a (Dec Jan) [ 1071 ]
            Urgent? off
            yusra Yusra AlSayyad made changes -
            Reviewers Eric Bellm [ ebellm ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ebellm Eric Bellm made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            yusra Yusra AlSayyad made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-28885 [ DM-28885 ]
            jbosch Jim Bosch made changes -
            Link This issue is duplicated by DM-28885 [ DM-28885 ]

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, Hsin-Fang Chiang, Jim Bosch, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.