Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26119

Test dataset disassembly with ci_hsc_gen3

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Composite disassembly is not properly tested anywhere for afw Exposures. Run ci_hsc_gen3 with disassembly turned on and see what breaks.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            All of the changes on this ticket are good and should be reviewed and ci_hsc_gen3 does complete with them. A fundamental problem though is that when multiprocessing is enabled (with the -j option) it breaks because the FILTER singleton is not initialized. Normally this is done when an Instrument is instantiated but in multi processing this never happens.

            The error is:

            Traceback (most recent call last):
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/miniconda/envs/lsst-scipipe-973126a/lib/python3.7/multiprocessing/pool.py", line 121, in worker
                result = (True, func(*args, **kwds))
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 208, in _executePipelineTask
                return executor.execute(taskDef, quantum, butler)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 82, in execute
                self.runQuantum(task, quantum, taskDef, butler)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 224, in runQuantum
                task.runQuantum(butlerQC, inputRefs, outputRefs)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_tasks/20.0.0-12-g2a9f6943+2/python/lsst/pipe/tasks/calibrate.py", line 621, in runQuantum
                outputs = self.run(**inputs)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_tasks/20.0.0-12-g2a9f6943+2/python/lsst/pipe/tasks/calibrate.py", line 721, in run
                sourceCat=sourceCat,
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_base/20.0.0-6-g9c77118/python/lsst/pipe/base/timer.py", line 150, in wrapper
                res = func(self, *args, **keyArgs)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_astrom/20.0.0-1-gc96f8cb+5/python/lsst/meas/astrom/astrometry.py", line 152, in run
                res = self.solve(exposure=exposure, sourceCat=sourceCat)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_base/20.0.0-6-g9c77118/python/lsst/pipe/base/timer.py", line 150, in wrapper
                res = func(self, *args, **keyArgs)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_astrom/20.0.0-1-gc96f8cb+5/python/lsst/meas/astrom/astrometry.py", line 196, in solve
                epoch=expMd.epoch,
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 324, in loadPixelBox
                return self.loadRegion(outerSkyRegion, filtFunc=_filterFunction, epoch=epoch, filterName=filterName)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 434, in loadRegion
                fluxField = getRefFluxField(schema=expandedCat.schema, filterName=filterName)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 729, in getRefFluxField
                raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList)))
            RuntimeError: Could not find flux field(s) camFlux
            """
            The above exception was the direct cause of the following exception:
            Traceback (most recent call last):
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/bin/pipetask", line 26, in <module>
                sys.exit(CmdLineFwk().parseAndRun())
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 494, in parseAndRun
                return self.runPipeline(qgraph, taskFactory, args)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 694, in runPipeline
                executor.execute(graph, butler)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 71, in execute
                self._executeQuantaMP(quantaIter, butler)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 175, in _executeQuantaMP
                results[dep].get(self.timeout)
              File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/miniconda/envs/lsst-scipipe-973126a/lib/python3.7/multiprocessing/pool.py", line 657, in get
                raise self._value
            RuntimeError: Could not find flux field(s) camFlux
            

            and in single process mode the filter yaml look like:

            aliases:
            - W-S-R+
            - HSC-R
            canonicalName: r
            name: r
            properties:
              lambdaEff: 623.0
              lambdaMax: .nan
              lambdaMin: .nan
            

            but in multiprocessing they look like:

            aliases: []
            canonicalName: _unknown_
            name: r
            

            Show
            tjenness Tim Jenness added a comment - All of the changes on this ticket are good and should be reviewed and ci_hsc_gen3 does complete with them. A fundamental problem though is that when multiprocessing is enabled (with the -j option) it breaks because the FILTER singleton is not initialized. Normally this is done when an Instrument is instantiated but in multi processing this never happens. The error is: Traceback (most recent call last): File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/miniconda/envs/lsst-scipipe-973126a/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 208, in _executePipelineTask return executor.execute(taskDef, quantum, butler) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 82, in execute self.runQuantum(task, quantum, taskDef, butler) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 224, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_tasks/20.0.0-12-g2a9f6943+2/python/lsst/pipe/tasks/calibrate.py", line 621, in runQuantum outputs = self.run(**inputs) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_tasks/20.0.0-12-g2a9f6943+2/python/lsst/pipe/tasks/calibrate.py", line 721, in run sourceCat=sourceCat, File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_base/20.0.0-6-g9c77118/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_astrom/20.0.0-1-gc96f8cb+5/python/lsst/meas/astrom/astrometry.py", line 152, in run res = self.solve(exposure=exposure, sourceCat=sourceCat) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/pipe_base/20.0.0-6-g9c77118/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_astrom/20.0.0-1-gc96f8cb+5/python/lsst/meas/astrom/astrometry.py", line 196, in solve epoch=expMd.epoch, File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 324, in loadPixelBox return self.loadRegion(outerSkyRegion, filtFunc=_filterFunction, epoch=epoch, filterName=filterName) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 434, in loadRegion fluxField = getRefFluxField(schema=expandedCat.schema, filterName=filterName) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/meas_algorithms/20.0.0-4-g085c40a3+2/python/lsst/meas/algorithms/loadReferenceObjects.py", line 729, in getRefFluxField raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList))) RuntimeError: Could not find flux field(s) camFlux """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/bin/pipetask", line 26, in <module> sys.exit(CmdLineFwk().parseAndRun()) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 494, in parseAndRun return self.runPipeline(qgraph, taskFactory, args) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 694, in runPipeline executor.execute(graph, butler) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 71, in execute self._executeQuantaMP(quantaIter, butler) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/stack/973126a/DarwinX86/ctrl_mpexec/20.0.0-7-g518c986+1/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 175, in _executeQuantaMP results[dep].get(self.timeout) File "/Volumes/ExternalSSD/Users/timj/work/lsstsw/miniconda/envs/lsst-scipipe-973126a/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value RuntimeError: Could not find flux field(s) camFlux and in single process mode the filter yaml look like: aliases: - W-S-R+ - HSC-R canonicalName: r name: r properties: lambdaEff: 623.0 lambdaMax: .nan lambdaMin: .nan but in multiprocessing they look like: aliases: [] canonicalName: _unknown_ name: r
            Hide
            tjenness Tim Jenness added a comment -

            Follow up on this, if I clone the Exposure read/write behavior with Filter (by using the FILTER keyword and ignoring what happens to be in Filter) everything works. This raises two issues:

            1. The code in meas_astrom using HSC-R works but using r does not. I assume this is what we expect (John Parejko?)
            2. There is a lot of metadata manipulation in Exposure.writeFits that is not reproducible if each component in Exposure is written out separately. In many cases this does not matter because we are explicitly calling writeFits on each component so we don't need to rely on storing the values in metadata and then regenerating them from metadata. Somewhere in the pipeline the Filter is set to r but never gets over-ridden by the FILTER header value.

            I'm not sure how much of this is a problem. For now I will recreate how FILTER is handled in Exposure.writeFits inside ExposureAssembler. That can be removed when DM-26181 can be relied upon.

            Show
            tjenness Tim Jenness added a comment - Follow up on this, if I clone the Exposure read/write behavior with Filter (by using the FILTER keyword and ignoring what happens to be in Filter) everything works. This raises two issues: The code in meas_astrom using HSC-R works but using r does not. I assume this is what we expect ( John Parejko ?) There is a lot of metadata manipulation in Exposure.writeFits that is not reproducible if each component in Exposure is written out separately. In many cases this does not matter because we are explicitly calling writeFits on each component so we don't need to rely on storing the values in metadata and then regenerating them from metadata. Somewhere in the pipeline the Filter is set to r but never gets over-ridden by the FILTER header value. I'm not sure how much of this is a problem. For now I will recreate how FILTER is handled in Exposure.writeFits inside ExposureAssembler. That can be removed when DM-26181 can be relied upon.
            Hide
            Parejkoj John Parejko added a comment -

            I don't understand your point 1) above. "using HSC-R" - using it how and where?

            Show
            Parejkoj John Parejko added a comment - I don't understand your point 1) above. "using HSC-R" - using it how and where?
            Hide
            tjenness Tim Jenness added a comment -

            John Parejko I'm sorry. See the stack trace in an early comment involving meas_astrom and filter determination. Whatever code was using the results of that meas_astrom filter determination failed when it thought the filter was "r" but worked fine when it thought it was HSC-R even without the filters for HSC being registered. It seemed like a part of the code you were familiar with. I don't know exactly which part of the pipeline was using it but it was reading the filter from an icExp.

            Show
            tjenness Tim Jenness added a comment - John Parejko I'm sorry. See the stack trace in an early comment involving meas_astrom and filter determination. Whatever code was using the results of that meas_astrom filter determination failed when it thought the filter was "r" but worked fine when it thought it was HSC-R even without the filters for HSC being registered. It seemed like a part of the code you were familiar with. I don't know exactly which part of the pipeline was using it but it was reading the filter from an icExp.
            Hide
            krughoff Simon Krughoff added a comment -

            Looks good.

            Show
            krughoff Simon Krughoff added a comment - Looks good.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Simon Krughoff
              Watchers:
              Andy Salnikov, Jim Bosch, John Parejko, Michelle Gower, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.