Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17062

Apply jointcal/FGCM in Gen3 ForcedPhotCcd

    XMLWordPrintable

Details

    • Story
    • Status: Invalid
    • Resolution: Done
    • None
    • None
    • None
    • 0
    • DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug), DRP F20-5 (Oct), DRP S21a (Dec Jan), DRP S21b, DRP S22A, DRP S22B
    • Data Release Production

    Description

      ci_hsc doesn't contain enough data to make running jointcal helpful. DM-17028 implements MakeWarpTask, but lacks the ability to apply jointcal outputs. 

      (This was split into a separate ticket because it is a lower priority than getting a full DRP on ci_hsc working)

      Attachments

        Issue Links

          Activity

            price Paul Price added a comment -

            Similar changes are required to the ForcedPhotCcdTask in meas_base.

            Gen3 doesn't work for this: I got the following error when adding a tract dimension for jointcal inputs.

            Failed to build graph: envelope(): incompatible function arguments. The following argument types are supported:
                1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet
            Invoked with: HtmPixelization(7), NotImplemented
            Traceback (most recent call last):
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/bin/pipetask", line 26, in <module>
                sys.exit(CmdLineFwk().parseAndRun())
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 156, in parseAndRun
                qgraph = self.makeGraph(pipeline, args)
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeGraph
                qgraph = graphBuilder.makeGraph(pipeline, inputCollections, outputCollection, args.data_query)
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 778, in makeGraph
                skipExisting=self.skipExisting)
              File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 668, in fillQuanta
                expand=True,
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/_registry.py", line 1301, in queryDatasets
                query = builder.finish()
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 403, in finish
                parameters = self._addWhereClause()
              File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 334, in _addWhereClause
                for begin, end in dimension.pixelization.envelope(self.summary.dataId.region):
            TypeError: envelope(): incompatible function arguments. The following argument types are supported:
                1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet
            Invoked with: HtmPixelization(7), NotImplemented
            

            price Paul Price added a comment - Similar changes are required to the ForcedPhotCcdTask in meas_base. Gen3 doesn't work for this: I got the following error when adding a tract dimension for jointcal inputs. Failed to build graph: envelope(): incompatible function arguments. The following argument types are supported: 1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet Invoked with: HtmPixelization(7), NotImplemented Traceback (most recent call last): File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/bin/pipetask", line 26, in <module> sys.exit(CmdLineFwk().parseAndRun()) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 156, in parseAndRun qgraph = self.makeGraph(pipeline, args) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_mpexec/19.0.0-7-g52b04eb+9/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeGraph qgraph = graphBuilder.makeGraph(pipeline, inputCollections, outputCollection, args.data_query) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 778, in makeGraph skipExisting=self.skipExisting) File "/projects/HSC/LSST/stack_tiger2-sumire.princeton.edu_20191212/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0-6-gb6b8b0a+9/python/lsst/pipe/base/graphBuilder.py", line 668, in fillQuanta expand=True, File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/_registry.py", line 1301, in queryDatasets query = builder.finish() File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 403, in finish parameters = self._addWhereClause() File "/home/pprice/LSST/daf_butler/python/lsst/daf/butler/registry/queries/_builder.py", line 334, in _addWhereClause for begin, end in dimension.pixelization.envelope(self.summary.dataId.region): TypeError: envelope(): incompatible function arguments. The following argument types are supported: 1. (self: lsst.sphgeom.pixelization.Pixelization, region: lsst.sphgeom.region.Region, maxRanges: int=0) -> lsst::sphgeom::RangeSet Invoked with: HtmPixelization(7), NotImplemented
            erykoff Eli Rykoff added a comment -

            Recent discussion on slack leads me to believe we can have the Gen3 code take ExposureCatalog s as input for visit-level compilations of PhotoCalib and SkyWcs which will reduce the number of files by a couple orders of magnitude for HSC and LSSTCam, as well as save disk space.

            erykoff Eli Rykoff added a comment - Recent discussion on slack leads me to believe we can have the Gen3 code take ExposureCatalog s as input for visit-level compilations of PhotoCalib and SkyWcs which will reduce the number of files by a couple orders of magnitude for HSC and LSSTCam, as well as save disk space.

            FGCM/Jointcal being applied during makeWarp on DM-27613.

            Renaming this ticket to match the outstanding TO DOs in the stack:
            https://github.com/lsst/obs_subaru/blob/master/pipelines/DRP.yaml#L17
            https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py#L351

            yusra Yusra AlSayyad added a comment - FGCM/Jointcal being applied during makeWarp on DM-27613 . Renaming this ticket to match the outstanding TO DOs in the stack: https://github.com/lsst/obs_subaru/blob/master/pipelines/DRP.yaml#L17 https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py#L351
            jbosch Jim Bosch added a comment -

            Dove into this during pair coding, and now I'm pretty sure that the Gen2 code isn't applying these external calibrations at all, and never has (at first I thought maybe DM-19942 broke it, but if this task applied external calibrations at any point, it doesn't look like it was doing it immediately before that was merged).

            In fact, it probably shouldn't apply the photoCalib at all - it should report raw instrumental fluxes, and downstream (e.g. standardization) code should apply the photometric calibration). The same is mostly true of the the astrometric calibration, but we should also use those internally when transforming the reference positions (and shapes, where applicable). It should, however, apply the skyCorr backgrounds to the image before measuring.

            So, I'm inclined to say that while there is work to do on this ticket, it's not a Gen2<->Gen3 parity issue, and maybe it's something best tackled when or after the Gen2 version can be retired entirely. What have I missed here?

            jbosch Jim Bosch added a comment - Dove into this during pair coding, and now I'm pretty sure that the Gen2 code isn't applying these external calibrations at all, and never has (at first I thought maybe DM-19942 broke it, but if this task applied external calibrations at any point, it doesn't look like it was doing it immediately before that was merged). In fact, it probably shouldn't apply the photoCalib at all - it should report raw instrumental fluxes, and downstream (e.g. standardization) code should apply the photometric calibration). The same is mostly true of the the astrometric calibration, but we should also use those internally when transforming the reference positions (and shapes, where applicable). It should, however, apply the skyCorr backgrounds to the image before measuring. So, I'm inclined to say that while there is work to do on this ticket, it's not a Gen2<->Gen3 parity issue, and maybe it's something best tackled when or after the Gen2 version can be retired entirely. What have I missed here?
            jbosch Jim Bosch added a comment -

            Confirmed my hypothesis at team meeting; this is worth doing, but not a parity issue, and I'm unassigning myself for now.

            jbosch Jim Bosch added a comment - Confirmed my hypothesis at team meeting; this is worth doing, but not a parity issue, and I'm unassigning myself for now.
            erykoff Eli Rykoff added a comment -

            This is getting done as part of DM-33982.

            erykoff Eli Rykoff added a comment - This is getting done as part of DM-33982 .

            People

              yusra Yusra AlSayyad
              yusra Yusra AlSayyad
              Eli Rykoff, Hsin-Fang Chiang, Jim Bosch, John Parejko, Paul Price, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.