Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14529

"RuntimeError: Input catalogs for source selection must be contiguous" in ci_hsc

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None

      Description

      ci_hsc is still broken after the obs_subaru config fix of DM-14526.

      https://ci.lsst.codes/blue/organizations/jenkins/science-pipelines%2Flsst_distrib/detail/lsst_distrib/255/pipeline

      self.writeMatches(patchRef, exposure, sources)
      :::::  [2018-05-20T05:38:13.987112Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/pipe_tasks/15.0-11-g8c8473f0/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches
      :::::  [2018-05-20T05:38:13.987118Z]     result = self.match.run(sources, exposure.getInfo().getFilter().getName())
      :::::  [2018-05-20T05:38:13.987171Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc+1/python/lsst/meas/astrom/directMatch.py", line 105, in run
      :::::  [2018-05-20T05:38:13.987178Z]     sourceSelection = self.sourceSelection.run(catalog)
      :::::  [2018-05-20T05:38:13.987183Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543+1/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run
      :::::  [2018-05-20T05:38:13.987188Z]     raise RuntimeError("Input catalogs for source selection must be contiguous.")
      :::::  [2018-05-20T05:38:13.987193Z] RuntimeError: Input catalogs for source selection must be contiguous.
      

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Risk Score 0
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            John Parejko do you have ideas about this error?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - John Parejko do you have ideas about this error?
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14339 [ DM-14339 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Labels SciencePipelines
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to RFC-335 [ RFC-335 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-9832 [ DM-9832 ]
            Hide
            erykoff Eli Rykoff added a comment -

            Basically, you have to check if a catalog is contiguous and if not do a deep copy before calling a sourceSelector. But I don't know exactly what idiom John Parejko chose as a recommendation after the discussion on slack: https://des-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=12336 and thereabouts. (The upshot is that making a deep copy of the catalog is a performance hit, but still much faster than doing source selection on a discontiguous catalog.)

            Show
            erykoff Eli Rykoff added a comment - Basically, you have to check if a catalog is contiguous and if not do a deep copy before calling a sourceSelector. But I don't know exactly what idiom John Parejko chose as a recommendation after the discussion on slack: https://des-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=12336 and thereabouts. (The upshot is that making a deep copy of the catalog is a performance hit, but still much faster than doing source selection on a discontiguous catalog.)
            Hide
            Parejkoj John Parejko added a comment -

            I'll run ci_hsc locally and try to figure this out.

            Show
            Parejkoj John Parejko added a comment - I'll run ci_hsc locally and try to figure this out.
            Parejkoj John Parejko made changes -
            Assignee John Parejko [ parejkoj ]
            Status To Do [ 10001 ] In Progress [ 3 ]
            Parejkoj John Parejko made changes -
            Comment [ I cannot seem to reproduce this locally, running {{scons}} on my setup copy of ci_hsc. ]
            Hide
            Parejkoj John Parejko added a comment -

            Here's a full traceback from a local run on my ci_hsc. It's failing in multiband, on the input catalog:

            $ /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise
            root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/measureCoaddSources.py'
            root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/hsc/measureCoaddSources.py'
            CameraMapper INFO: Loading exposure registry from /data/ci_hsc/DATA/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3
            root INFO: Running: /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise
            WARNING: You are using OpenBLAS with multiple threads (8), but have not
            specified the number of threads using one of the OpenBLAS environment variables:
            OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
            This may indicate that you are unintentionally using multiple threads, which may
            cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
            what you are doing and want threads enabled implicitly, set the environment
            variable LSST_ALLOW_IMPLICIT_THREADS.
            measureCoaddSources INFO: Read 4365 detections: DataId(initialdata={'tract': 0, 'patch': '5,4', 'filter': 'HSC-I'}, tag=set())
            measureCoaddSources.deblend INFO: Deblending 4365 sources
            measureCoaddSources.deblend INFO: Deblended: of 4365 sources, 1067 were deblended, creating 3994 children, total 8359 sources
            measureCoaddSources.measurement INFO: Measuring 8359 sources (4365 parents, 3994 children)
            measureCoaddSources.applyApCorr INFO: Applying aperture corrections to 23 flux fields
            measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs
            measureCoaddSources.propagateFlags INFO: Propagated 578 sources with flag calib_psfCandidate
            measureCoaddSources.propagateFlags INFO: Propagated 531 sources with flag calib_psfUsed
            Using matplotlib backend: Qt5Agg
            Traceback (most recent call last):
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py", line 3, in <module>
                MeasureMergedCoaddSourcesTask.parseAndRun()
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun
                resultList = taskRunner.run(parsedCmd)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 224, in run
                resultList = list(mapFunc(self, targetList))
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 389, in __call__
                result = task.run(dataRef, **kwargs)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1106, in run
                self.writeMatches(patchRef, exposure, sources)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches
                result = self.match.run(sources, exposure.getInfo().getFilter().getName())
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc/python/lsst/meas/astrom/directMatch.py", line 105, in run
                sourceSelection = self.sourceSelection.run(catalog)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run
                raise RuntimeError("Input catalogs for source selection must be contiguous.")
            RuntimeError: Input catalogs for source selection must be contiguous.
            > /home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py(106)run()
                105         if not sourceCat.isContiguous():
            --> 106             raise RuntimeError("Input catalogs for source selection must be contiguous.")
                107
            

            Show
            Parejkoj John Parejko added a comment - Here's a full traceback from a local run on my ci_hsc. It's failing in multiband, on the input catalog: $ /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/measureCoaddSources.py' root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/hsc/measureCoaddSources.py' CameraMapper INFO: Loading exposure registry from /data/ci_hsc/DATA/registry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 root INFO: Running: /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise WARNING: You are using OpenBLAS with multiple threads (8), but have not specified the number of threads using one of the OpenBLAS environment variables: OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS. This may indicate that you are unintentionally using multiple threads, which may cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know what you are doing and want threads enabled implicitly, set the environment variable LSST_ALLOW_IMPLICIT_THREADS. measureCoaddSources INFO: Read 4365 detections: DataId(initialdata={'tract': 0, 'patch': '5,4', 'filter': 'HSC-I'}, tag=set()) measureCoaddSources.deblend INFO: Deblending 4365 sources measureCoaddSources.deblend INFO: Deblended: of 4365 sources, 1067 were deblended, creating 3994 children, total 8359 sources measureCoaddSources.measurement INFO: Measuring 8359 sources (4365 parents, 3994 children) measureCoaddSources.applyApCorr INFO: Applying aperture corrections to 23 flux fields measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs measureCoaddSources.propagateFlags INFO: Propagated 578 sources with flag calib_psfCandidate measureCoaddSources.propagateFlags INFO: Propagated 531 sources with flag calib_psfUsed Using matplotlib backend: Qt5Agg Traceback (most recent call last): File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py", line 3, in <module> MeasureMergedCoaddSourcesTask.parseAndRun() File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 224, in run resultList = list(mapFunc(self, targetList)) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 389, in __call__ result = task.run(dataRef, **kwargs) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1106, in run self.writeMatches(patchRef, exposure, sources) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches result = self.match.run(sources, exposure.getInfo().getFilter().getName()) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc/python/lsst/meas/astrom/directMatch.py", line 105, in run sourceSelection = self.sourceSelection.run(catalog) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run raise RuntimeError("Input catalogs for source selection must be contiguous.") RuntimeError: Input catalogs for source selection must be contiguous. > /home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py(106)run() 105 if not sourceCat.isContiguous(): --> 106 raise RuntimeError("Input catalogs for source selection must be contiguous.") 107
            krughoff Simon Krughoff made changes -
            Reviewers Simon Krughoff [ krughoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            krughoff Simon Krughoff added a comment -

            Done.

            Show
            krughoff Simon Krughoff added a comment - Done.
            krughoff Simon Krughoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the weekend review.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thanks for the weekend review. Merged and done.
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Simon Krughoff
                Watchers:
                Eli Rykoff, Hsin-Fang Chiang, John Parejko, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel