Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15184

Reprocess RC2 with w_2018_32

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_32. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Running multiBandDriver.py with w_2018_32 gave the following error:

            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 1304, in __init__
                self.makeSubtask("measurement", schema=self.schema, algMetadata=self.algMetadata)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-10-g0b41441+2/python/lsst/pipe/base/task.py", line 300, in makeSubtask
                subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pex_config/16.0-2-g636dfb1/python/lsst/pex/config/configurableField.py", line 83, in apply
                return self.target(*args, config=self.value, **kw)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/sfm.py", line 267, in __init__
                self.initializePlugins(schema=self.schema)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/baseMeasurement.py", line 242, in initializePlugins
                self.plugins[name] = PluginClass(config, name, metadata=self.algMetadata, **kwds)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/wrappers.py", line 20, in __init__
                self.cpp = self.factory(config, name, schema, metadata)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/wrappers.py", line 246, in factory
                return AlgClass(config.makeControl(), name, schema, **kwargs)
            lsst.pex.exceptions.wrappers.NotFoundError:
              File "src/table/Schema.cc", line 225, in lsst::afw::table::SchemaItem<T> lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = int; std::string = std::basic_string<char>]
                Field or subfield withname 'deblend_nChild' not found with type 'I'. {0}
            lsst::pex::exceptions::NotFoundError: 'Field or subfield withname 'deblend_nChild' not found with type 'I'.'
            

            This was caused by DM-15104. The required change in the pipe_drivers package was forgotten when ticket merged before w_2018_32. The branch was later pushed to GitHub and is now in PR https://github.com/lsst/pipe_drivers/pull/62

            With the pipe_drivers branch, I got the following error:

              File "/home/hchiang2/stack/pipe_drivers/python/lsst/pipe/drivers/multiBandDriver.py", line 483, in runMergeMeasurements
                self.mergeCoaddMeasurements.runDataRef(dataRefList)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 504, in runDataRef
                mergedCatalog = self.run(catalogs, patchRefList[0])
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 1594, in run
                raise ValueError("Error in inputs to MergeCoaddMeasurements: source IDs do not match")
            ValueError: Error in inputs to MergeCoaddMeasurements: source IDs do not match
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Running multiBandDriver.py with w_2018_32 gave the following error: File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 1304, in __init__ self.makeSubtask("measurement", schema=self.schema, algMetadata=self.algMetadata) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0-10-g0b41441+2/python/lsst/pipe/base/task.py", line 300, in makeSubtask subtask = taskField.apply(name=name, parentTask=self, **keyArgs) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pex_config/16.0-2-g636dfb1/python/lsst/pex/config/configurableField.py", line 83, in apply return self.target(*args, config=self.value, **kw) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/sfm.py", line 267, in __init__ self.initializePlugins(schema=self.schema) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/baseMeasurement.py", line 242, in initializePlugins self.plugins[name] = PluginClass(config, name, metadata=self.algMetadata, **kwds) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/wrappers.py", line 20, in __init__ self.cpp = self.factory(config, name, schema, metadata) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_base/16.0-11-gf9130ea+3/python/lsst/meas/base/wrappers.py", line 246, in factory return AlgClass(config.makeControl(), name, schema, **kwargs) lsst.pex.exceptions.wrappers.NotFoundError: File "src/table/Schema.cc", line 225, in lsst::afw::table::SchemaItem<T> lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = int; std::string = std::basic_string<char>] Field or subfield withname 'deblend_nChild' not found with type 'I'. {0} lsst::pex::exceptions::NotFoundError: 'Field or subfield withname 'deblend_nChild' not found with type 'I'.' This was caused by DM-15104 . The required change in the pipe_drivers package was forgotten when ticket merged before w_2018_32. The branch was later pushed to GitHub and is now in PR https://github.com/lsst/pipe_drivers/pull/62 With the pipe_drivers branch, I got the following error: File "/home/hchiang2/stack/pipe_drivers/python/lsst/pipe/drivers/multiBandDriver.py", line 483, in runMergeMeasurements self.mergeCoaddMeasurements.runDataRef(dataRefList) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 504, in runDataRef mergedCatalog = self.run(catalogs, patchRefList[0]) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-9-gc30114dc+2/python/lsst/pipe/tasks/multiBand.py", line 1594, in run raise ValueError("Error in inputs to MergeCoaddMeasurements: source IDs do not match") ValueError: Error in inputs to MergeCoaddMeasurements: source IDs do not match
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            DM-15531 and the new branch of DM-15104 have been merged and solved the multiBandDriver errors.

            During w_2018_32 and w_2018_34 there was a breaking change of DM-14997, so simply running multiBandDriver using w_2018_34 stack on a w_2018_32-generated repo does not work: errors as follows

              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/16.0-4-g17aeff3/python/lsst/pipe/drivers/multiBandDriver.py", line 464, in runMeasurements
                self.measureCoaddSources.runDataRef(dataRef)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-12-g85846a17/python/lsst/pipe/tasks/multiBand.py", line 1361, in runDataRef
                exposure.getWcs())
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-12-g85846a17/python/lsst/pipe/tasks/propagateVisitFlags.py", line 191, in run
                matches = afwTable.matchRaDec(coaddSources, ccdSources[ccdSources.get(flag)], radius, mc)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/16.0-31-gd4f695684/python/lsst/afw/table/baseColumnView/baseColumnViewContinued.py", line 59, in __getitem__
                keyobj = self.schema.find(key).key
            KeyError: "Field with name 'calib_psf_candidate' not found"
            

            To continue this week 32 campaign, the new master of pipe_drivers at 17aeff3, and pipe_tasks as of w_2018_32 plus the DM-15531 patch were used. With this mix, multiBandDriver was able to finish.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - DM-15531 and the new branch of DM-15104 have been merged and solved the multiBandDriver errors. During w_2018_32 and w_2018_34 there was a breaking change of DM-14997 , so simply running multiBandDriver using w_2018_34 stack on a w_2018_32-generated repo does not work: errors as follows File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/16.0-4-g17aeff3/python/lsst/pipe/drivers/multiBandDriver.py", line 464, in runMeasurements self.measureCoaddSources.runDataRef(dataRef) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-12-g85846a17/python/lsst/pipe/tasks/multiBand.py", line 1361, in runDataRef exposure.getWcs()) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0-12-g85846a17/python/lsst/pipe/tasks/propagateVisitFlags.py", line 191, in run matches = afwTable.matchRaDec(coaddSources, ccdSources[ccdSources.get(flag)], radius, mc) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/16.0-31-gd4f695684/python/lsst/afw/table/baseColumnView/baseColumnViewContinued.py", line 59, in __getitem__ keyobj = self.schema.find(key).key KeyError: "Field with name 'calib_psf_candidate' not found" To continue this week 32 campaign, the new master of pipe_drivers at 17aeff3 , and pipe_tasks as of w_2018_32 plus the DM-15531 patch were used. With this mix, multiBandDriver was able to finish.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            coadd/multiband: the number of data products and empty patches are the same as w_2018_15/DM-14123 (this comment)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - coadd/multiband: the number of data products and empty patches are the same as w_2018_15/ DM-14123 ( this comment )
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_32/DM-15184/
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_32/DM-15184/logs
            The pipe_analysis and validate_drp runs are at DM-15519

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_32/ DM-15184 / The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_32/ DM-15184 /logs The pipe_analysis and validate_drp runs are at DM-15519
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In this run the optional packages such as meas_extensions_convolved and meas_extensions_photometryKron were not set up when the single frame processing was run, so the Kron magnitudes were not in the src catalogs, etc. Still closing this ticket for now; a new ticket will be filed if a new re-run is necessary.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In this run the optional packages such as meas_extensions_convolved and meas_extensions_photometryKron were not set up when the single frame processing was run, so the Kron magnitudes were not in the src catalogs, etc. Still closing this ticket for now; a new ticket will be filed if a new re-run is necessary.

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel