Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14529

"RuntimeError: Input catalogs for source selection must be contiguous" in ci_hsc

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None

      Description

      ci_hsc is still broken after the obs_subaru config fix of DM-14526.

      https://ci.lsst.codes/blue/organizations/jenkins/science-pipelines%2Flsst_distrib/detail/lsst_distrib/255/pipeline

      self.writeMatches(patchRef, exposure, sources)
      :::::  [2018-05-20T05:38:13.987112Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/pipe_tasks/15.0-11-g8c8473f0/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches
      :::::  [2018-05-20T05:38:13.987118Z]     result = self.match.run(sources, exposure.getInfo().getFilter().getName())
      :::::  [2018-05-20T05:38:13.987171Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc+1/python/lsst/meas/astrom/directMatch.py", line 105, in run
      :::::  [2018-05-20T05:38:13.987178Z]     sourceSelection = self.sourceSelection.run(catalog)
      :::::  [2018-05-20T05:38:13.987183Z]   File "/home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.devtoolset-6.py3/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543+1/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run
      :::::  [2018-05-20T05:38:13.987188Z]     raise RuntimeError("Input catalogs for source selection must be contiguous.")
      :::::  [2018-05-20T05:38:13.987193Z] RuntimeError: Input catalogs for source selection must be contiguous.
      

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the weekend review.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thanks for the weekend review. Merged and done.
            Hide
            krughoff Simon Krughoff added a comment -

            Done.

            Show
            krughoff Simon Krughoff added a comment - Done.
            Hide
            Parejkoj John Parejko added a comment -

            Here's a full traceback from a local run on my ci_hsc. It's failing in multiband, on the input catalog:

            $ /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise
            root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/measureCoaddSources.py'
            root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/hsc/measureCoaddSources.py'
            CameraMapper INFO: Loading exposure registry from /data/ci_hsc/DATA/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3
            root INFO: Running: /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise
            WARNING: You are using OpenBLAS with multiple threads (8), but have not
            specified the number of threads using one of the OpenBLAS environment variables:
            OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
            This may indicate that you are unintentionally using multiple threads, which may
            cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
            what you are doing and want threads enabled implicitly, set the environment
            variable LSST_ALLOW_IMPLICIT_THREADS.
            measureCoaddSources INFO: Read 4365 detections: DataId(initialdata={'tract': 0, 'patch': '5,4', 'filter': 'HSC-I'}, tag=set())
            measureCoaddSources.deblend INFO: Deblending 4365 sources
            measureCoaddSources.deblend INFO: Deblended: of 4365 sources, 1067 were deblended, creating 3994 children, total 8359 sources
            measureCoaddSources.measurement INFO: Measuring 8359 sources (4365 parents, 3994 children)
            measureCoaddSources.applyApCorr INFO: Applying aperture corrections to 23 flux fields
            measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs
            measureCoaddSources.propagateFlags INFO: Propagated 578 sources with flag calib_psfCandidate
            measureCoaddSources.propagateFlags INFO: Propagated 531 sources with flag calib_psfUsed
            Using matplotlib backend: Qt5Agg
            Traceback (most recent call last):
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py", line 3, in <module>
                MeasureMergedCoaddSourcesTask.parseAndRun()
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun
                resultList = taskRunner.run(parsedCmd)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 224, in run
                resultList = list(mapFunc(self, targetList))
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 389, in __call__
                result = task.run(dataRef, **kwargs)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1106, in run
                self.writeMatches(patchRef, exposure, sources)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches
                result = self.match.run(sources, exposure.getInfo().getFilter().getName())
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc/python/lsst/meas/astrom/directMatch.py", line 105, in run
                sourceSelection = self.sourceSelection.run(catalog)
              File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run
                raise RuntimeError("Input catalogs for source selection must be contiguous.")
            RuntimeError: Input catalogs for source selection must be contiguous.
            > /home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py(106)run()
                105         if not sourceCat.isContiguous():
            --> 106             raise RuntimeError("Input catalogs for source selection must be contiguous.")
                107
            

            Show
            Parejkoj John Parejko added a comment - Here's a full traceback from a local run on my ci_hsc. It's failing in multiband, on the input catalog: $ /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/measureCoaddSources.py' root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/hsc/measureCoaddSources.py' CameraMapper INFO: Loading exposure registry from /data/ci_hsc/DATA/registry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 root INFO: Running: /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise WARNING: You are using OpenBLAS with multiple threads (8), but have not specified the number of threads using one of the OpenBLAS environment variables: OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS. This may indicate that you are unintentionally using multiple threads, which may cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know what you are doing and want threads enabled implicitly, set the environment variable LSST_ALLOW_IMPLICIT_THREADS. measureCoaddSources INFO: Read 4365 detections: DataId(initialdata={'tract': 0, 'patch': '5,4', 'filter': 'HSC-I'}, tag=set()) measureCoaddSources.deblend INFO: Deblending 4365 sources measureCoaddSources.deblend INFO: Deblended: of 4365 sources, 1067 were deblended, creating 3994 children, total 8359 sources measureCoaddSources.measurement INFO: Measuring 8359 sources (4365 parents, 3994 children) measureCoaddSources.applyApCorr INFO: Applying aperture corrections to 23 flux fields measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs measureCoaddSources.propagateFlags INFO: Propagated 578 sources with flag calib_psfCandidate measureCoaddSources.propagateFlags INFO: Propagated 531 sources with flag calib_psfUsed Using matplotlib backend: Qt5Agg Traceback (most recent call last): File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py", line 3, in <module> MeasureMergedCoaddSourcesTask.parseAndRun() File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 224, in run resultList = list(mapFunc(self, targetList)) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 389, in __call__ result = task.run(dataRef, **kwargs) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1106, in run self.writeMatches(patchRef, exposure, sources) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches result = self.match.run(sources, exposure.getInfo().getFilter().getName()) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc/python/lsst/meas/astrom/directMatch.py", line 105, in run sourceSelection = self.sourceSelection.run(catalog) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run raise RuntimeError("Input catalogs for source selection must be contiguous.") RuntimeError: Input catalogs for source selection must be contiguous. > /home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py(106)run() 105 if not sourceCat.isContiguous(): --> 106 raise RuntimeError("Input catalogs for source selection must be contiguous.") 107
            Hide
            Parejkoj John Parejko added a comment -

            I'll run ci_hsc locally and try to figure this out.

            Show
            Parejkoj John Parejko added a comment - I'll run ci_hsc locally and try to figure this out.
            Hide
            erykoff Eli Rykoff added a comment -

            Basically, you have to check if a catalog is contiguous and if not do a deep copy before calling a sourceSelector. But I don't know exactly what idiom John Parejko chose as a recommendation after the discussion on slack: https://des-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=12336 and thereabouts. (The upshot is that making a deep copy of the catalog is a performance hit, but still much faster than doing source selection on a discontiguous catalog.)

            Show
            erykoff Eli Rykoff added a comment - Basically, you have to check if a catalog is contiguous and if not do a deep copy before calling a sourceSelector. But I don't know exactly what idiom John Parejko chose as a recommendation after the discussion on slack: https://des-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=12336 and thereabouts. (The upshot is that making a deep copy of the catalog is a performance hit, but still much faster than doing source selection on a discontiguous catalog.)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            John Parejko do you have ideas about this error?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - John Parejko do you have ideas about this error?

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Simon Krughoff
                Watchers:
                Eli Rykoff, Hsin-Fang Chiang, John Parejko, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel