Hsin-Fang Chiang added a comment - 20/May/18 9:52 AM John Parejko do you have ideas about this error?

Eli Rykoff added a comment - 20/May/18 10:21 AM Basically, you have to check if a catalog is contiguous and if not do a deep copy before calling a sourceSelector. But I don't know exactly what idiom John Parejko chose as a recommendation after the discussion on slack: https://des-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=12336 and thereabouts. (The upshot is that making a deep copy of the catalog is a performance hit, but still much faster than doing source selection on a discontiguous catalog.)

John Parejko added a comment - 20/May/18 11:22 AM I'll run ci_hsc locally and try to figure this out.

John Parejko added a comment - 20/May/18 1:09 PM Here's a full traceback from a local run on my ci_hsc. It's failing in multiband, on the input catalog: $ /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/measureCoaddSources.py' root INFO: Loading config overrride file '/home/parejkoj/lsst/cameras/obs_subaru/config/hsc/measureCoaddSources.py' CameraMapper INFO: Loading exposure registry from /data/ci_hsc/DATA/registry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /data/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 root INFO: Running: /home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py /data/ci_hsc/DATA --rerun ci_hsc --id tract=0 patch=5,4 filter=HSC-I --doraise WARNING: You are using OpenBLAS with multiple threads (8), but have not specified the number of threads using one of the OpenBLAS environment variables: OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS. This may indicate that you are unintentionally using multiple threads, which may cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know what you are doing and want threads enabled implicitly, set the environment variable LSST_ALLOW_IMPLICIT_THREADS. measureCoaddSources INFO: Read 4365 detections: DataId(initialdata={'tract': 0, 'patch': '5,4', 'filter': 'HSC-I'}, tag=set()) measureCoaddSources.deblend INFO: Deblending 4365 sources measureCoaddSources.deblend INFO: Deblended: of 4365 sources, 1067 were deblended, creating 3994 children, total 8359 sources measureCoaddSources.measurement INFO: Measuring 8359 sources (4365 parents, 3994 children) measureCoaddSources.applyApCorr INFO: Applying aperture corrections to 23 flux fields measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs measureCoaddSources.propagateFlags INFO: Propagated 578 sources with flag calib_psfCandidate measureCoaddSources.propagateFlags INFO: Propagated 531 sources with flag calib_psfUsed Using matplotlib backend: Qt5Agg Traceback (most recent call last): File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/bin/measureCoaddSources.py", line 3, in <module> MeasureMergedCoaddSourcesTask.parseAndRun() File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 575, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 224, in run resultList = list(mapFunc(self, targetList)) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_base/15.0-6-gfa9b38f/python/lsst/pipe/base/cmdLineTask.py", line 389, in __call__ result = task.run(dataRef, **kwargs) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1106, in run self.writeMatches(patchRef, exposure, sources) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/pipe_tasks/15.0-10-g91db0633/python/lsst/pipe/tasks/multiBand.py", line 1139, in writeMatches result = self.match.run(sources, exposure.getInfo().getFilter().getName()) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_astrom/15.0-3-g52118bc/python/lsst/meas/astrom/directMatch.py", line 105, in run sourceSelection = self.sourceSelection.run(catalog) File "/home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py", line 106, in run raise RuntimeError("Input catalogs for source selection must be contiguous.") RuntimeError: Input catalogs for source selection must be contiguous. > /home/parejkoj/lsst/lsstsw/stack/Linux64/meas_algorithms/15.0-11-g7db6e543/python/lsst/meas/algorithms/sourceSelector.py(106)run() 105 if not sourceCat.isContiguous(): --> 106 raise RuntimeError("Input catalogs for source selection must be contiguous.") 107

Simon Krughoff added a comment - 20/May/18 6:27 PM Done.

John Parejko added a comment - 20/May/18 10:56 PM Thanks for the weekend review. Merged and done.