Hsin-Fang Chiang added a comment - 26/Mar/18 2:30 PM For singleFrameDriver , the failed CCDs are the same as w_2018_10 in DM-13647 .

Hsin-Fang Chiang added a comment - 26/Mar/18 2:32 PM - edited In the logs of mosaic and coadd, messages such as daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:calexp_md Data ID:DataId(initialdata={ 'ccd' : 6 , 'visit' : 23596 , 'tract' : 9813 , 'filter' : 'NB0921' }, tag=set()) at location lsst.afw.image.ExposureF at FitsStorage( 01172 /NB0921/corr/CORR- 0023596 - 006 .fits) are seen, as reported and fixed in DM-13888 .

Hsin-Fang Chiang added a comment - 27/Mar/18 8:56 AM Compared to w_2018_10 or earlier, fewer IndexError: cannot do a non-empty take from an empty axes are seen ( DM-13517 ). Only one at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' 196477 WARN 2018-03-24T19:25:47.007-0500 coaddDriver: lsst-verify-worker29:196477: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. 196477 INFO 2018-03-24T19:25:47.100-0500 coaddDriver: lsst-verify-worker29:196477: Traceback: Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+14/python/lsst/ctrl/pool/parallel.py", line 512, in logOperation yield File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b0-hsc-22-g014233a/python/lsst/pipe/drivers/coaddDriver.py", line 321, in coadd detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-55-g35e83f69+1/python/lsst/pipe/tasks/multiBand.py", line 307, in runDetection detections = self.detection.makeSourceCatalog(table, exposure, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+60/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/detection.py", line 323, in run clearMask=clearMask, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 198, in detectFootprints threshResults = self.calculateThreshold(exposure, seed, sigma=sigma) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 123, in calculateThreshold lq, uq = np.percentile(fluxes, [25.0, 75.0]) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4269, in percentile interpolation=interpolation) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4011, in _ureduce r = func(a, **kwargs) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4386, in _percentile x1 = take(ap, indices_below, axis=axis) * weights_below File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc return getattr(obj, method)(*args, **kwds) IndexError: cannot do a non-empty take from an empty axes.

Hsin-Fang Chiang added a comment - 27/Mar/18 10:45 AM - edited For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 77 in HSC-Z, and 76 in NB0921; in total 463 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 Compared to the w_2018_10 run ( DM-13647 ), deepCoadd_calexp and other coadd/multiband products are created in more edge patches, as fewer "IndexError" are encountered as in the previous comment. This is likely related to DM-13741 . In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 79 patches in COSMOS tract 9813.

Hsin-Fang Chiang added a comment - 27/Mar/18 11:37 AM Similar to the last few RC2 runs, the multiband processing for the COSMOS tract did not finish in one go and was continued with the --reuse-outputs-from all option. The first attempt died likely because it went out of memory while deblending at multiBandDriver.measureCoaddSources . lsst-verify-worker22 hit 100% memory usage before the job was killed:

Hsin-Fang Chiang added a comment - 27/Mar/18 12:37 PM The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_12/ DM-13795 / . The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_12/ DM-13795 /logs/ . The pipe_analysis run is in DM-13893 . The validate_drp run is in DM-13894 .