Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13795

Reprocess RC2 with w_2018_12

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_12. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For singleFrameDriver, the failed CCDs are the same as w_2018_10 in DM-13647.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For singleFrameDriver , the failed CCDs are the same as w_2018_10 in DM-13647 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            In the logs of mosaic and coadd, messages such as

            daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:calexp_md Data ID:DataId(initialdata={'ccd': 6, 'visit': 23596, 'tract': 9813, 'filter': 'NB0921'}, tag=set()) at location lsst.afw.image.ExposureF at FitsStorage(01172/NB0921/corr/CORR-0023596-006.fits)
            

            are seen, as reported and fixed in DM-13888.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited In the logs of mosaic and coadd, messages such as daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:calexp_md Data ID:DataId(initialdata={ 'ccd' : 6 , 'visit' : 23596 , 'tract' : 9813 , 'filter' : 'NB0921' }, tag=set()) at location lsst.afw.image.ExposureF at FitsStorage( 01172 /NB0921/corr/CORR- 0023596 - 006 .fits) are seen, as reported and fixed in DM-13888 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Compared to w_2018_10 or earlier, fewer IndexError: cannot do a non-empty take from an empty axes are seen (DM-13517). Only one at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'

            196477 WARN  2018-03-24T19:25:47.007-0500 coaddDriver: lsst-verify-worker29:196477: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            196477 INFO  2018-03-24T19:25:47.100-0500 coaddDriver: lsst-verify-worker29:196477: Traceback:
            Traceback (most recent call last):
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+14/python/lsst/ctrl/pool/parallel.py", line 512, in logOperation
                yield
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b0-hsc-22-g014233a/python/lsst/pipe/drivers/coaddDriver.py", line 321, in coadd
                detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-55-g35e83f69+1/python/lsst/pipe/tasks/multiBand.py", line 307, in runDetection
                detections = self.detection.makeSourceCatalog(table, exposure, expId=expId)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+60/python/lsst/pipe/base/timer.py", line 150, in wrapper
                res = func(self, *args, **keyArgs)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/detection.py", line 323, in run
                clearMask=clearMask, expId=expId)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 198, in detectFootprints
                threshResults = self.calculateThreshold(exposure, seed, sigma=sigma)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 123, in calculateThreshold
                lq, uq = np.percentile(fluxes, [25.0, 75.0])
              File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4269, in percentile
                interpolation=interpolation)
              File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4011, in _ureduce
                r = func(a, **kwargs)
              File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4386, in _percentile
                x1 = take(ap, indices_below, axis=axis) * weights_below
              File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take
                return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
              File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
                return getattr(obj, method)(*args, **kwds)
            IndexError: cannot do a non-empty take from an empty axes.
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Compared to w_2018_10 or earlier, fewer IndexError: cannot do a non-empty take from an empty axes are seen ( DM-13517 ). Only one at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' 196477 WARN 2018-03-24T19:25:47.007-0500 coaddDriver: lsst-verify-worker29:196477: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. 196477 INFO 2018-03-24T19:25:47.100-0500 coaddDriver: lsst-verify-worker29:196477: Traceback: Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+14/python/lsst/ctrl/pool/parallel.py", line 512, in logOperation yield File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b0-hsc-22-g014233a/python/lsst/pipe/drivers/coaddDriver.py", line 321, in coadd detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-55-g35e83f69+1/python/lsst/pipe/tasks/multiBand.py", line 307, in runDetection detections = self.detection.makeSourceCatalog(table, exposure, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+60/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/detection.py", line 323, in run clearMask=clearMask, expId=expId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 198, in detectFootprints threshResults = self.calculateThreshold(exposure, seed, sigma=sigma) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-21-ge7d40960+1/python/lsst/meas/algorithms/dynamicDetection.py", line 123, in calculateThreshold lq, uq = np.percentile(fluxes, [25.0, 75.0]) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4269, in percentile interpolation=interpolation) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4011, in _ureduce r = func(a, **kwargs) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4386, in _percentile x1 = take(ap, indices_below, axis=axis) * weights_below File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode) File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc return getattr(obj, method)(*args, **kwds) IndexError: cannot do a non-empty take from an empty axes.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 77 in HSC-Z, and 76 in NB0921; in total 463 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            Compared to the w_2018_10 run (DM-13647), deepCoadd_calexp and other coadd/multiband products are created in more edge patches, as fewer "IndexError" are encountered as in the previous comment. This is likely related to DM-13741.

            In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in 4 WIDE tracts, 79 patches in COSMOS tract 9813.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 77 in HSC-Z, and 76 in NB0921; in total 463 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 Compared to the w_2018_10 run ( DM-13647 ), deepCoadd_calexp and other coadd/multiband products are created in more edge patches, as fewer "IndexError" are encountered as in the previous comment. This is likely related to DM-13741 . In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 79 patches in COSMOS tract 9813.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Similar to the last few RC2 runs, the multiband processing for the COSMOS tract did not finish in one go and was continued with the --reuse-outputs-from all option.

            The first attempt died likely because it went out of memory while deblending at multiBandDriver.measureCoaddSources. lsst-verify-worker22 hit 100% memory usage before the job was killed:

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Similar to the last few RC2 runs, the multiband processing for the COSMOS tract did not finish in one go and was continued with the --reuse-outputs-from all option. The first attempt died likely because it went out of memory while deblending at multiBandDriver.measureCoaddSources . lsst-verify-worker22 hit 100% memory usage before the job was killed:
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_12/DM-13795/.
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_12/DM-13795/logs/.

            The pipe_analysis run is in DM-13893.
            The validate_drp run is in DM-13894.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_12/ DM-13795 / . The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_12/ DM-13795 /logs/ . The pipe_analysis run is in DM-13893 . The validate_drp run is in DM-13894 .

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.