Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13463

Reprocess RC1 with w_2018_03

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC1 dataset, as defined in DMTR-31 Sect. 4.1 , using Stack w_2018_03. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, coaddDriver.py, and multiBandDriver.py (DM-10129/DM-11020).

      Summarize the total node-hours needed.

        Attachments

          Issue Links

            Activity

            Hide
            sthrush Samantha Thrush added a comment -

            All of the runs have finished successfully, as confirmed by Slurm logs and butler and all of the files are located at /project/hsc_rc/w_2018_03/DM-13463.

            When I ran singleFrameDriver.py, I encountered 29 FATAL errors from the following ccd/visit pairs:

            Wide: {visit: 7344, ccd: 67}, {visit:19468, ccd:69}, {visit:9708, ccd:99}, {visit:9736, ccd:67}, {visit:17738, ccd:69}, {visit:17750, ccd:58}, {visit:9868, ccd 76}, {visit:11582, ccd:76}, {visit:6478, ccd:99}, {visit:6528, ccd:24}, {visit:6528, ccd:67}, {visit:6528, ccd:59}

            Cosmos: {visit:11698, ccd:68}, {visit:17934, ccd:1}, {visit:28376, ccd: 69}, {visit:28382, ccd:101}, {visit:28392, ccd:101}, {visit:28396, ccd:102}, {visit:28398, ccd:95}, {visit:28398, ccd:101}, {visit:28400, ccd:53}, {visit:28400, ccd:61}, {visit:28400, ccd:95}, {visit:28400, ccd:101}, {visit:28400, ccd:100}, {visit:23596, ccd:6}, {visit:280, ccd:22}, {visit:280, ccd:13}, {visit:280, ccd:103}

            The errors encountered were:

            • 5 - RuntimeError: Unable to match sources
            • 16 - " InvalidParameterError:
              File "src/PsfexPsf.cc", line 221, in virtual std::shared_ptr<lsst::afw::image::Image<double> > lsst::meas::extensions::psfex::PsfexPsf::_doComputeImage(const Point2D&, const lsst::afw::image::Color&, const Point2D&) const
              Only spatial variation (ndim == 2) is supported; saw 0 {0}"
            • 4 - RuntimeError: No objects passed our cuts for consideration as psf stars.
            • 4 - RuntimeError: No matches to use for photocal 

            No other FATAL errors were found in the logs for makeSkyMap.py,  mosaic.pycoaddDriver.py, and multiBandDriver.py.  The log files can be found in /project/hsc_rc/w_2018_03/DM-13463/logs. If you are interested in seeing the script used to run this reprocessing, it can be found in /project/hsc_rc/w_2018_03/DM-13463.  

            Below is a table detailing the various pertinent information of the run, especially the node-hours used.  To summarize this table quickly, this run used a total of 209.2774 node-hours.  

            Code Visit set Time Elapsed JobID Nodes Node-Hours
            makeSkyMap  All 00:01:04 108407 0.0178
            singleFrameDriver WideG 2:08:06 108408 3 6.405 
              WideR 00:32:27 108425 1.6518 
              WideI 00:59:10 108426  2.9583
              WideZ 00:46:00 108427   2.3001
              WideY 02:08:34 108428   6.4284
              CosmoG 01:03:00 108429   3.15
              CosmoR 01:07:30 108430  3.375
              CosmoI 01:57:20 108431  5.8668
              CosmoZ 02:10:44 108432   6.5367
              CosmoN 01:47:12 108434   5.3601
              CosmoY 03:43:50 108433  11.1918
            mosaic WideG tract 8766 & WideG tract 8767 00:13:11+00:10:29 108520 & 108518   1  0.3944
              WideR tract 8766 & WideR tract 8767 00:05:36+00:05:26 108521 & 108522  1  0.1839
              WideI tract 8766 & WideI tract 8767 00:11:11+00:13:19 108523 & 108524  0.4083
              WideZ tract 8766 & WideZ tract 8767 00:10:37+00:10:39 108525 & 108526  0.3544
              WideY tract 8766 & WideY tract 8767 00:07:59+00:08:15 108527 & 108528  0.2706
              CosmoG 00:16:06 108529  0.2683
              CosmoR 00:15:36 108530  0.26
              CosmoI 00:26:07 108531  0.4353
              CosmoZ 01:01:46 108541  1.0294
              CosmoY 01:22:15  108540  1.3708
              CosmoN 00:23:35 108534  0.3931
            coaddDriver WideG 03:14:57 108535  3.2492
              WideR 01:38:13 108536  1.6369
              WideI 02:49:40 108537  2.8278
              WideZ 02:50:18 108538   2.8383
              WideY 02:11:07 108539  2.1853
              CosmoG 03:11:12 108587  3.1867
              CosmoR 02:49:07 108588   2.8186
              CosmoI 06:13:38 108590  6.2272
              CosmoZ 10:14:37 108591   10.2436
              CosmoN 05:17:35 108589   5.2931
              CosmoY 15:26:42 108592   15.445
            multiBandDriver Cosmos 07:13:59 108612 10  72.331
              Wide 10:11:32 108586   20.3844
            Show
            sthrush Samantha Thrush added a comment - All of the runs have finished successfully, as confirmed by Slurm logs and butler and all of the files are located at /project/hsc_rc/w_2018_03/ DM-13463 . When I ran singleFrameDriver.py , I encountered 29 FATAL errors from the following ccd/visit pairs: Wide: {visit: 7344, ccd: 67}, {visit:19468, ccd:69}, {visit:9708, ccd:99}, {visit:9736, ccd:67}, {visit:17738, ccd:69}, {visit:17750, ccd:58}, {visit:9868, ccd 76}, {visit:11582, ccd:76}, {visit:6478, ccd:99}, {visit:6528, ccd:24}, {visit:6528, ccd:67}, {visit:6528, ccd:59} Cosmos: {visit:11698, ccd:68}, {visit:17934, ccd:1}, {visit:28376, ccd: 69}, {visit:28382, ccd:101}, {visit:28392, ccd:101}, {visit:28396, ccd:102}, {visit:28398, ccd:95}, {visit:28398, ccd:101}, {visit:28400, ccd:53}, {visit:28400, ccd:61}, {visit:28400, ccd:95}, {visit:28400, ccd:101}, {visit:28400, ccd:100}, {visit:23596, ccd:6}, {visit:280, ccd:22}, {visit:280, ccd:13}, {visit:280, ccd:103} The errors encountered were: 5 - RuntimeError: Unable to match sources 16 - " InvalidParameterError: File "src/PsfexPsf.cc", line 221, in virtual std::shared_ptr<lsst::afw::image::Image<double> > lsst::meas::extensions::psfex::PsfexPsf::_doComputeImage(const Point2D&, const lsst::afw::image::Color&, const Point2D&) const Only spatial variation (ndim == 2) is supported; saw 0 {0}" 4 - RuntimeError: No objects passed our cuts for consideration as psf stars. 4 - RuntimeError: No matches to use for photocal  No other FATAL errors were found in the logs for makeSkyMap.py ,   mosaic.py ,  coaddDriver.py , and  multiBandDriver.py .  The log files can be found in  /project/hsc_rc/w_2018_03/ DM-13463 /logs . If you are interested in seeing the script used to run this reprocessing, it can be found in /project/hsc_rc/w_2018_03/ DM-13463 .   Below is a table detailing the various pertinent information of the run, especially the node-hours used.  To summarize this table quickly, this run used a total of  209.2774 node-hours.    Code Visit set Time Elapsed JobID Nodes Node-Hours makeSkyMap  All 00:01:04 108407 1  0.0178 singleFrameDriver WideG 2:08:06 108408 3 6.405    WideR 00:32:27 108425 3  1.6518    WideI 00:59:10 108426  3  2.9583   WideZ 00:46:00 108427  3   2.3001   WideY 02:08:34 108428  3   6.4284   CosmoG 01:03:00 108429  3   3.15   CosmoR 01:07:30 108430 3   3.375   CosmoI 01:57:20 108431 3   5.8668   CosmoZ 02:10:44 108432  3   6.5367   CosmoN 01:47:12 108434  3   5.3601   CosmoY 03:43:50 108433 3   11.1918 mosaic WideG tract 8766 & WideG tract 8767 00:13:11+00:10:29 108520 & 108518   1  0.3944   WideR tract 8766 & WideR tract 8767 00:05:36+00:05:26 108521 & 108522  1  0.1839   WideI tract 8766 & WideI tract 8767 00:11:11+00:13:19 108523 & 108524 1   0.4083   WideZ tract 8766 & WideZ tract 8767 00:10:37+00:10:39 108525 & 108526 1   0.3544   WideY tract 8766 & WideY tract 8767 00:07:59+00:08:15 108527 & 108528 1   0.2706   CosmoG 00:16:06 108529 1   0.2683   CosmoR 00:15:36 108530 1   0.26   CosmoI 00:26:07 108531 1   0.4353   CosmoZ 01:01:46 108541 1   1.0294   CosmoY 01:22:15  108540 1   1.3708   CosmoN 00:23:35 108534 1   0.3931 coaddDriver WideG 03:14:57 108535 1   3.2492   WideR 01:38:13 108536 1   1.6369   WideI 02:49:40 108537 1   2.8278   WideZ 02:50:18 108538  1   2.8383   WideY 02:11:07 108539 1   2.1853   CosmoG 03:11:12 108587 1   3.1867   CosmoR 02:49:07 108588  1   2.8186   CosmoI 06:13:38 108590 1   6.2272   CosmoZ 10:14:37 108591  1   10.2436   CosmoN 05:17:35 108589  1   5.2931   CosmoY 15:26:42 108592  1   15.445 multiBandDriver Cosmos 07:13:59 108612 10  72.331   Wide 10:11:32 108586  2   20.3844
            Hide
            sthrush Samantha Thrush added a comment - - edited

            After checking through the coadd log files, I have found that quite a few visits were removed because the median e residual was too large and their scaled size scatter was too large.  A list of these visits is included below.

            coadd visit list & filter number of "Removing visits" found
            Cosmos-G 53
            Cosmos-I 86
            Cosmos-N 16
            Cosmos-R 294
            Cosmos-Y 937
            Cosmos-Z 264
            Wide-G 81
            Wide-I 132
            Wide-R 27
            Wide-Y 127
            Wide-Z 33

            This information was found with the command grep -c 'Removing visit' coadd*

            The number of occurances for Cosmo-Y seem incredibly high, but after looking at the log files in detail, it would seem that if a visit & ccd pair are complained about once and put on the "removing visits" list, then they are complained about more than twice in the log file.  I am looking further into why this happened. 

            Show
            sthrush Samantha Thrush added a comment - - edited After checking through the coadd log files, I have found that quite a few visits were removed because the median e residual was too large and their scaled size scatter was too large.  A list of these visits is included below. coadd visit list & filter number of "Removing visits" found Cosmos-G 53 Cosmos-I 86 Cosmos-N 16 Cosmos-R 294 Cosmos-Y 937 Cosmos-Z 264 Wide-G 81 Wide-I 132 Wide-R 27 Wide-Y 127 Wide-Z 33 This information was found with the command  grep -c 'Removing visit' coadd* .  The number of occurances for Cosmo-Y seem incredibly high, but after looking at the log files in detail, it would seem that if a visit & ccd pair are complained about once and put on the "removing visits" list, then they are complained about more than twice in the log file.  I am looking further into why this happened. 
            Hide
            sthrush Samantha Thrush added a comment -

            After running wc -l to find calexp files in the deepCoadd-results directory, I found that the Wide tracts 8767 and 8766 both have 81 patches in all 5 filters, thus aren't missing any patches.  However, tract 9813 is missing some patches in each filter:

            • HSC-G: 78 patches (missing:{0,0}, {0,8}, {8,0})
            • HSC-I: 79 patches (missing:{0,0}, {0,8})
            • HSC-R: 74 patches (missing:{0,0},{0,1},{0,7},{0,8},{1,8},{8,0},{8,8})
            • HSC-Z: 79 patches (missing:{0,0}, {0,8})
            • HSC-Y: 79 patches (missing: {0,0}, {0,8})
            • NB0921: 76 patches (missing: {0,0}, {0,7}, {0,8}, {1,8}, {8,8})

            This closely mirrors the results shown in DM-11345 for all 9813 filters except for HSC-Z (that run was missing the {8,8} patch in addition to those mentioned above).

            After checking to see which patches are included in deepCoadd-results/merged (created by multiBandDriver.py), I found that tracts 8766 and 8767 included all 81 patches, while tract 9813 included 79 patches (missing: {0,0}, {0,8}).

            Show
            sthrush Samantha Thrush added a comment - After running wc -l to find calexp files in the deepCoadd-results directory, I found that the Wide tracts 8767 and 8766 both have 81 patches in all 5 filters, thus aren't missing any patches.  However, tract 9813 is missing some patches in each filter: HSC-G: 78 patches (missing:{0,0}, {0,8}, {8,0}) HSC-I: 79 patches (missing:{0,0}, {0,8}) HSC-R: 74 patches (missing:{0,0},{0,1},{0,7},{0,8},{1,8},{8,0},{8,8}) HSC-Z: 79 patches (missing:{0,0}, {0,8}) HSC-Y: 79 patches (missing: {0,0}, {0,8}) NB0921: 76 patches (missing: {0,0}, {0,7}, {0,8}, {1,8}, {8,8}) This closely mirrors the results shown in DM-11345 for all 9813 filters except for HSC-Z (that run was missing the {8,8} patch in addition to those mentioned above). After checking to see which patches are included in deepCoadd-results/merged (created by multiBandDriver.py), I found that tracts 8766 and 8767 included all 81 patches, while tract 9813 included 79 patches (missing: {0,0}, {0,8}).
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The repeated visit/ccd IDs of "Removing visit" is known and expected; there would be one record from each patch if I understand correctly.

            Note that the COSMOS visit list in DM-11345 is different from the one you use here. Some problematic visits have been removed for RC2.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The repeated visit/ccd IDs of "Removing visit" is known and expected; there would be one record from each patch if I understand correctly. Note that the COSMOS visit list in DM-11345 is different from the one you use here. Some problematic visits have been removed for RC2.
            Hide
            sthrush Samantha Thrush added a comment - - edited

            Ah, ok, that would make sense.    In that case, is there anything else that I should include here before I close this ticket?

            Show
            sthrush Samantha Thrush added a comment - - edited Ah, ok, that would make sense.    In that case, is there anything else that I should include here before I close this ticket?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            You can close the ticket if you have verified the run and are happy with the results.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - You can close the ticket if you have verified the run and are happy with the results.
            Hide
            sthrush Samantha Thrush added a comment -

            Ok.  I'm just running coaddDriver for Cosmos Y one more time just to confirm that the "removed visits" seen above isn't a complete fluke.  

            Show
            sthrush Samantha Thrush added a comment - Ok.  I'm just running coaddDriver for Cosmos Y one more time just to confirm that the "removed visits" seen above isn't a complete fluke.  
            Hide
            sthrush Samantha Thrush added a comment -

            I've confirmed that the number of "Removing visits" is not a one-time fluke for CosmosY and am happy with my results.

            Show
            sthrush Samantha Thrush added a comment - I've confirmed that the number of "Removing visits" is not a one-time fluke for CosmosY and am happy with my results.

              People

              • Assignee:
                sthrush Samantha Thrush
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Samantha Thrush
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: