Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17062

Apply jointcal/FGCM in Gen3 ForcedPhotCcd

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      0
    • Epic Link:
    • Sprint:
      DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct), DRP S21a (Dec Jan), DRP S21b, DRP S22A, DRP S22B
    • Team:
      Data Release Production

      Description

      ci_hsc doesn't contain enough data to make running jointcal helpful. DM-17028 implements MakeWarpTask, but lacks the ability to apply jointcal outputs. 

      (This was split into a separate ticket because it is a lower priority than getting a full DRP on ci_hsc working)

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17028 Write PipelineTask to Make Warps (MakeCoaddTempExp conversion)

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23352 Add jointcal, skyCorr to forcedPhotCcd

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27613 Apply FGCM and jointcal in Gen3 MakeWarpTask

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            price Paul Price added a comment -

            Similar changes are required to the ForcedPhotCcdTask in meas_base.

            Gen3 doesn't work for this: I got the following error when adding a tract dimension for jointcal inputs.

            Failed to build graph: envelope(): incompatible function arguments. The following argument types are supported:
            		 
                1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet
            		 
            Invoked with: HtmPixelization(7), NotImplemented
            		 
            Traceback (most recent call last):
            		 
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/bin/pipetask", line 26, in <module>
            		 
                sys.exit(CmdLineFwk().parseAndRun())
            		 
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 156, in parseAndRun
            		 
                qgraph = self.makeGraph(pipeline, args)
            		 
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeGraph
            		 
                qgraph = graphBuilder.makeGraph(pipeline, inputCollections, outputCollection, args.data_query)
            		 
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 778, in makeGraph
            		 
                skipExisting=self.skipExisting)
            		 
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 668, in fillQuanta
            		 
                expand=True,
            		 
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/_registry.py", line 1301, in queryDatasets
            		 
                query = builder.finish()
            		 
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 403, in finish
            		 
                parameters = self._addWhereClause()
            		 
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 334, in _addWhereClause
            		 
                for begin, end in dimension.pixelization.envelope(self.summary.dataId.region):
            		 
            TypeError: envelope(): incompatible function arguments. The following argument types are supported:
            		 
                1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet
            		 
            Invoked with: HtmPixelization(7), NotImplemented

            Show
            price Paul Price added a comment - Similar changes are required to the ForcedPhotCcdTask in meas_base. Gen3 doesn't work for this: I got the following error when adding a tract dimension for jointcal inputs. Failed to build graph: envelope(): incompatible function arguments. The following argument types are supported: 1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet Invoked with: HtmPixelization(7), NotImplemented Traceback (most recent call last): File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/bin/pipetask", line 26, in <module> sys.exit(CmdLineFwk().parseAndRun()) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 156, in parseAndRun qgraph = self.makeGraph(pipeline, args) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeGraph qgraph = graphBuilder.makeGraph(pipeline, inputCollections, outputCollection, args.data_query) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 778, in makeGraph skipExisting=self.skipExisting) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 668, in fillQuanta expand=True, File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/_registry.py", line 1301, in queryDatasets query = builder.finish() File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 403, in finish parameters = self._addWhereClause() File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 334, in _addWhereClause for begin, end in dimension.pixelization.envelope(self.summary.dataId.region): TypeError: envelope(): incompatible function arguments. The following argument types are supported: 1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet Invoked with: HtmPixelization(7), NotImplemented
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Recent discussion on slack leads me to believe we can have the Gen3 code take ExposureCatalog s as input for visit-level compilations of PhotoCalib and SkyWcs which will reduce the number of files by a couple orders of magnitude for HSC and LSSTCam, as well as save disk space.

            Show
            erykoff Eli Rykoff added a comment - Recent discussion on slack leads me to believe we can have the Gen3 code take ExposureCatalog s as input for visit-level compilations of PhotoCalib and SkyWcs which will reduce the number of files by a couple orders of magnitude for HSC and LSSTCam, as well as save disk space.
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            FGCM/Jointcal being applied during makeWarp on DM-27613.

            Renaming this ticket to match the outstanding TO DOs in the stack:
            https://github.com/lsst/obs_subaru/blob/master/pipelines/DRP.yaml#L17
            https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py#L351

            Show
            yusra Yusra AlSayyad added a comment - FGCM/Jointcal being applied during makeWarp on DM-27613 . Renaming this ticket to match the outstanding TO DOs in the stack: https://github.com/lsst/obs_subaru/blob/master/pipelines/DRP.yaml#L17 https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py#L351
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Dove into this during pair coding, and now I'm pretty sure that the Gen2 code isn't applying these external calibrations at all, and never has (at first I thought maybe DM-19942 broke it, but if this task applied external calibrations at any point, it doesn't look like it was doing it immediately before that was merged).

            In fact, it probably shouldn't apply the photoCalib at all - it should report raw instrumental fluxes, and downstream (e.g. standardization) code should apply the photometric calibration). The same is mostly true of the the astrometric calibration, but we should also use those internally when transforming the reference positions (and shapes, where applicable). It should, however, apply the skyCorr backgrounds to the image before measuring.

            So, I'm inclined to say that while there is work to do on this ticket, it's not a Gen2<->Gen3 parity issue, and maybe it's something best tackled when or after the Gen2 version can be retired entirely. What have I missed here?

            Show
            jbosch Jim Bosch added a comment - Dove into this during pair coding, and now I'm pretty sure that the Gen2 code isn't applying these external calibrations at all, and never has (at first I thought maybe DM-19942 broke it, but if this task applied external calibrations at any point, it doesn't look like it was doing it immediately before that was merged). In fact, it probably shouldn't apply the photoCalib at all - it should report raw instrumental fluxes, and downstream (e.g. standardization) code should apply the photometric calibration). The same is mostly true of the the astrometric calibration, but we should also use those internally when transforming the reference positions (and shapes, where applicable). It should, however, apply the skyCorr backgrounds to the image before measuring. So, I'm inclined to say that while there is work to do on this ticket, it's not a Gen2<->Gen3 parity issue, and maybe it's something best tackled when or after the Gen2 version can be retired entirely. What have I missed here?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Confirmed my hypothesis at team meeting; this is worth doing, but not a parity issue, and I'm unassigning myself for now.

            Show
            jbosch Jim Bosch added a comment - Confirmed my hypothesis at team meeting; this is worth doing, but not a parity issue, and I'm unassigning myself for now.
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            This is getting done as part of DM-33982.

            Show
            erykoff Eli Rykoff added a comment - This is getting done as part of DM-33982 .

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Eli Rykoff, Hsin-Fang Chiang, Jim Bosch, John Parejko, Paul Price, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.