Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11345

Redefine HSC "RC" dataset for bi-weeklies processing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP F17-5, DRP F17-6, DRP S18-1, DRP S18-2
    • Team:
      Data Release Production

      Description

      We are currently processing what is known as the HSC RC dataset, where RC ==> Release Candidate, on a bi-weekly basis (for all even numbered weeklies). See https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC+dataset for full details. The purpose of this processing is to have a fully processed dataset as the pipeline evolves and be able track any quality/performance changes (for better or for worse - to properly praise/bless the former and catch & fix/revert the latter). Clearly, the dataset involved should be representative of the widest range of data quality/observing conditions expected, and is currently available to us through public HSC data. The RC dataset we have been using is not satisfactory in this respect. This issue is to remedy this situation by redefining (and perhaps renaming? RC ==> CI ??) the set of HSC tracts/visits to include in our bi-weeklies processing.

      A list of known data "characteristics" that this dataset should cover include tracts that suffer from the known issues to date, including:

      • a very well-behaved tract
      • a tract that includes visits that have proven challenging for astrometric fitting (see DM-11090)
      • a tract that includes visits that have proven challenging for photometric fitting (see DM-11927)
      • a tract including "too-good-seeing" visits (see DM-9855)
      • a tract including a wide range in seeing conditions (to test PSF-matching, see DM-8289)
      • tracts covering a range of "challenges" for deblending (possibly those identified in DM-9168)
      • a tract at the survey's edge (to check for issues arising due to patchy coverage - verify if 8766 WIDE still fills this bill)
      • a tract including galaxies suitable for assessing total magnitudes for "big" galaxies (list of examples along with "best" total magnitude estimates to be compared against being compiled by HSC member Song Huang)
      • a tract with observations in both the old i and new i2 HSC filters (see RFC-390)
      • PLEASE ADD YOUR FAVORITE PATHOLOGY HERE

      We will have to keep this dataset to a reasonable size (perhaps Hsin-Fang Chiang could comment on what "reasonable" means to her in numbers?) While we will certainly want to add pathological areas as we find/gain access to them (e.g. crowded fields), the set we define and agree upon here will be considered the canonical one until a need to expand/redefine is identified.

        Attachments

        1. PsfDist_UD_COSMOS_9813.png
          PsfDist_UD_COSMOS_9813.png
          21 kB
        2. PsfDist_WIDE_GAMMA15H_9615.png
          PsfDist_WIDE_GAMMA15H_9615.png
          19 kB
        3. PsfDist_WIDE_VVDS_9697.png
          PsfDist_WIDE_VVDS_9697.png
          18 kB
        4. tracts_patches_UD_COSMOS_HSC-I.png
          tracts_patches_UD_COSMOS_HSC-I.png
          97 kB
        5. visits_UD_COSMOS_9813_I.png
          visits_UD_COSMOS_9813_I.png
          270 kB
        6. visits_W_VVDS_9697_I.png
          visits_W_VVDS_9697_I.png
          317 kB
        7. visits_WIDE_GAMA15H_9615_I.png
          visits_WIDE_GAMA15H_9615_I.png
          286 kB
        8. W-GAMA15H_i_area.png
          W-GAMA15H_i_area.png
          53 kB
        9. WIDE_GAMA15H_9615_1,5_1,6-gri.png
          WIDE_GAMA15H_9615_1,5_1,6-gri.png
          2.96 MB
        10. W-VVDS_i_area.png
          W-VVDS_i_area.png
          51 kB

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            This discussion seems relevant to the question about how constant "the" RC dataset is which was raised in RFC-362.

            Show
            swinbank John Swinbank added a comment - This discussion seems relevant to the question about how constant "the" RC dataset is which was raised in RFC-362 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            +1 for having a distinct name for the newly defined dataset. I'm neutral to the name "CI" with a very mild concern it could be confused with ci_hsc, but I don't have a better proposal.

            In the short term (a few months / before we start using new tools officially) I hope it's sufficient to reprocess no more than a few tracts, processed repo < 10TB, in the bi-weeklies. But that will change quickly (next cycle).

            So I would say go ahead to define you minimal dataset that does what you need and perhaps set priorities. Then we will discuss the cadence of reprocessing them, storage provisioning, resources (machines+humans), etc. I recall previous discussions that we may have a small set for (bi-)weeklies and a medium set for monthlies; that might be a good idea.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - +1 for having a distinct name for the newly defined dataset. I'm neutral to the name "CI" with a very mild concern it could be confused with ci_hsc , but I don't have a better proposal. In the short term (a few months / before we start using new tools officially) I hope it's sufficient to reprocess no more than a few tracts, processed repo < 10TB, in the bi-weeklies. But that will change quickly (next cycle). So I would say go ahead to define you minimal dataset that does what you need and perhaps set priorities. Then we will discuss the cadence of reprocessing them, storage provisioning, resources (machines+humans), etc. I recall previous discussions that we may have a small set for (bi-)weeklies and a medium set for monthlies; that might be a good idea.
            Hide
            lauren Lauren MacArthur added a comment -

            Great, thanks Hsin-Fang Chiang!

            Show
            lauren Lauren MacArthur added a comment - Great, thanks Hsin-Fang Chiang !
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Looking at tract 9697 in WIDE-VVDS as a good candidate for patches with too-good-seeing. A query at https://hsc-release.mtk.nao.ac.jp/datasearch/catalog_jobs reveals 5 patches with large offset and/or scatter from the stellar locus:

            -- Identify bad patches with poor photometric accuracy in the Wide layer
             
            SELECT
              ra, dec, tract, patch,
              gri_offset, gri_scatter, riz_offset, riz_scatter, izy_offset, izy_scatter
            FROM
              pdr1_wide.patch_qa
            WHERE
              tract=9697 AND (ABS(gri_offset) > 0.05 OR gri_scatter > 0.07 OR ABS(riz_offset) > 0.05 OR riz_scatter > 0.07 OR ABS(izy_offset) > 0.05 OR izy_scatter > 0.07)
            

            5 records found.
                                                  gri               riz             izy
               ra       dec    tract  patch offset scatter    offset  scatter   offset  scatter
            338.1510  -0.00044  9697  200  0.0408  0.06825  -0.04192  0.09607  0.026753  0.02702
            338.5222   1.48801  9697    8  0.0591  0.03436  -0.02820  0.02155  0.017231  0.01477
            338.3377  -0.00044  9697  100  0.0318  0.05818  -0.05938  0.06735  0.029136  0.02105
            337.5911   0.18386  9697  501  0.0548  0.04894  -0.04252  0.03836  0.018961  0.02168
            337.0334   0.37049  9697  802  0.0500  0.03944  -0.02718  0.02248  0.019254  0.01457
            

            Show
            lauren Lauren MacArthur added a comment - - edited Looking at tract 9697 in WIDE-VVDS as a good candidate for patches with too-good-seeing. A query at https://hsc-release.mtk.nao.ac.jp/datasearch/catalog_jobs reveals 5 patches with large offset and/or scatter from the stellar locus: -- Identify bad patches with poor photometric accuracy in the Wide layer   SELECT ra, dec, tract, patch, gri_offset, gri_scatter, riz_offset, riz_scatter, izy_offset, izy_scatter FROM pdr1_wide.patch_qa WHERE tract=9697 AND (ABS(gri_offset) > 0.05 OR gri_scatter > 0.07 OR ABS(riz_offset) > 0.05 OR riz_scatter > 0.07 OR ABS(izy_offset) > 0.05 OR izy_scatter > 0.07) 5 records found. gri riz izy ra dec tract patch offset scatter offset scatter offset scatter 338.1510 -0.00044 9697 200 0.0408 0.06825 -0.04192 0.09607 0.026753 0.02702 338.5222 1.48801 9697 8 0.0591 0.03436 -0.02820 0.02155 0.017231 0.01477 338.3377 -0.00044 9697 100 0.0318 0.05818 -0.05938 0.06735 0.029136 0.02105 337.5911 0.18386 9697 501 0.0548 0.04894 -0.04252 0.03836 0.018961 0.02168 337.0334 0.37049 9697 802 0.0500 0.03944 -0.02718 0.02248 0.019254 0.01457
            Hide
            lauren Lauren MacArthur added a comment - - edited
            Field Tract Filter
            1. Visits
            Visit List
            WIDE_VVDS 9697 HSC-G 22 6320^34338^34342^34362^34366^34382^34384^34400^34402^34412^34414^34422^34424^34448^34450^34464^34468^34478^34480^34482^34484^34486
            WIDE_VVDS 9697 HSC-R 22 7138^34640^34644^34648^34652^34664^34670^34672^34674^34676^34686^34688^34690^34698^34706^34708^34712^34714^34734^34758^34760^34772
            WIDE_VVDS 9697 HSC-I 33 35870^35890^35892^35906^35936^35950^35974^36114^36118^36140^36144^36148^36158^36160^36170^36172^36180^36182^36190^36192^36202^36204^36212^36214^36216^36218^36234^36236^36238^36240^36258^36260^36262
            WIDE_VVDS 9697 HSC-Z 33 36404^36408^36412^36416^36424^36426^36428^36430^36432^36434^36438^36442^36444^36446^36448^36456^36458^36460^36466^36474^36476^36480^36488^36490^36492^36494^36498^36504^36506^36508^38938^38944^38950
            WIDE_VVDS 9697 HSC-Y 33 34874^34942^34944^34946^36726^36730^36738^36750^36754^36756^36758^36762^36768^36772^36774^36776^36778^36788^36790^36792^36794^36800^36802^36808^36810^36812^36818^36820^36828^36830^36834^36836^36838
            WIDE_VVDS 9697 TOTAL 143 Size: 1.7T
                     
            WIDE_GAMA15H 9615 HSC-G 17 26024^26028^26032^26036^26044^26046^26048^26050^26058^26060^26062^26070^26072^26074^26080^26084^26094
            WIDE_GAMA15H 9615 HSC-R 17 23864^23868^23872^23876^23884^23886^23888^23890^23898^23900^23902^23910^23912^23914^23920^23924^28976
            WIDE_GAMA15H 9615 HSC-I 26 1258^1262^1270^1274^1278^1280^1282^1286^1288^1290^1294^1300^1302^1306^1308^1310^1314^1316^1324^1326^1330^24494^24504^24522^24536^24538
            WIDE_GAMA15H 9615 HSC-Z 26 23212^23216^23224^23226^23228^23232^23234^23242^23250^23256^23258^27090^27094^27106^27108^27116^27118^27120^27126^27128^27130^27134^27136^27146^27148^27156
            WIDE_GAMA15H 9615 HSC-Y 26 380^384^388^404^408^424^426^436^440^442^446^452^456^458^462^464^468^470^472^474^478^27032^27034^27042^27066^27068
            WIDE_GAMA15H 9615 TOTAL 112 Size: 1.4T
                     
            UD_COSMOS 9813 HSC-G 17 11690^11692^11694^11696^11698^11700^11702^11704^11706^11708^11710^11712^29324^29326^29336^29340^29350
            UD_COSMOS 9813 HSC-R 16 1202^1204^1206^1208^1210^1212^1214^1216^1218^1220^23692^23694^23704^23706^23716^23718
            UD_COSMOS 9813 HSC-I 33 1228^1230^1232^1238^1240^1242^1244^1246^1248^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482^30484^30486^30488^30490^30492^30494^30496^30498^30500^30502^30504
            UD_COSMOS 9813 HSC-Z 31 1166^1168^1170^1172^1174^1176^1178^1180^1182^1184^1186^1188^1190^1192^1194^17900^17902^17904^17906^17908^17926^17928^17930^17932^17934^17944^17946^17948^17950^17952^17962
            UD_COSMOS 9813 HSC-Y 52 318^322^324^326^328^330^332^344^346^348^350^352^354^356^358^360^362^1868^1870^1872^1874^1876^1880^1882^11718^11720^11722^11724^11726^11728^11730^11732^11734^11736^11738^11740^22602^22604^22606^22608^22626^22628^22630^22632^22642^22644^22646^22648^22658^22660^22662^22664
            UD_COSMOS 9813 NB0921 28 23038^23040^23042^23044^23046^23048^23050^23052^23054^23056^23594^23596^23598^23600^23602^23604^23606^24298^24300^24302^24304^24306^24308^24310^25810^25812^25814^25816
            UD_COSMOS 9813 TOTAL 177 Size: 3.2T
            Show
            lauren Lauren MacArthur added a comment - - edited Field Tract Filter Visits Visit List WIDE_VVDS 9697 HSC-G 22 6320^34338^34342^34362^34366^34382^34384^34400^34402^34412^34414^34422^34424^34448^34450^34464^34468^34478^34480^34482^34484^34486 WIDE_VVDS 9697 HSC-R 22 7138^34640^34644^34648^34652^34664^34670^34672^34674^34676^34686^34688^34690^34698^34706^34708^34712^34714^34734^34758^34760^34772 WIDE_VVDS 9697 HSC-I 33 35870^35890^35892^35906^35936^35950^35974^36114^36118^36140^36144^36148^36158^36160^36170^36172^36180^36182^36190^36192^36202^36204^36212^36214^36216^36218^36234^36236^36238^36240^36258^36260^36262 WIDE_VVDS 9697 HSC-Z 33 36404^36408^36412^36416^36424^36426^36428^36430^36432^36434^36438^36442^36444^36446^36448^36456^36458^36460^36466^36474^36476^36480^36488^36490^36492^36494^36498^36504^36506^36508^38938^38944^38950 WIDE_VVDS 9697 HSC-Y 33 34874^34942^34944^34946^36726^36730^36738^36750^36754^36756^36758^36762^36768^36772^36774^36776^36778^36788^36790^36792^36794^36800^36802^36808^36810^36812^36818^36820^36828^36830^36834^36836^36838 WIDE_VVDS 9697 TOTAL 143 Size: 1.7T           WIDE_GAMA15H 9615 HSC-G 17 26024^26028^26032^26036^26044^26046^26048^26050^26058^26060^26062^26070^26072^26074^26080^26084^26094 WIDE_GAMA15H 9615 HSC-R 17 23864^23868^23872^23876^23884^23886^23888^23890^23898^23900^23902^23910^23912^23914^23920^23924^28976 WIDE_GAMA15H 9615 HSC-I 26 1258^1262^1270^1274^1278^1280^1282^1286^1288^1290^1294^1300^1302^1306^1308^1310^1314^1316^1324^1326^1330^24494^24504^24522^24536^24538 WIDE_GAMA15H 9615 HSC-Z 26 23212^23216^23224^23226^23228^23232^23234^23242^23250^23256^23258^27090^27094^27106^27108^27116^27118^27120^27126^27128^27130^27134^27136^27146^27148^27156 WIDE_GAMA15H 9615 HSC-Y 26 380^384^388^404^408^424^426^436^440^442^446^452^456^458^462^464^468^470^472^474^478^27032^27034^27042^27066^27068 WIDE_GAMA15H 9615 TOTAL 112 Size: 1.4T           UD_COSMOS 9813 HSC-G 17 11690^11692^11694^11696^11698^11700^11702^11704^11706^11708^11710^11712^29324^29326^29336^29340^29350 UD_COSMOS 9813 HSC-R 16 1202^1204^1206^1208^1210^1212^1214^1216^1218^1220^23692^23694^23704^23706^23716^23718 UD_COSMOS 9813 HSC-I 33 1228^1230^1232^1238^1240^1242^1244^1246^1248^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482^30484^30486^30488^30490^30492^30494^30496^30498^30500^30502^30504 UD_COSMOS 9813 HSC-Z 31 1166^1168^1170^1172^1174^1176^1178^1180^1182^1184^1186^1188^1190^1192^1194^17900^17902^17904^17906^17908^17926^17928^17930^17932^17934^17944^17946^17948^17950^17952^17962 UD_COSMOS 9813 HSC-Y 52 318^322^324^326^328^330^332^344^346^348^350^352^354^356^358^360^362^1868^1870^1872^1874^1876^1880^1882^11718^11720^11722^11724^11726^11728^11730^11732^11734^11736^11738^11740^22602^22604^22606^22608^22626^22628^22630^22632^22642^22644^22646^22648^22658^22660^22662^22664 UD_COSMOS 9813 NB0921 28 23038^23040^23042^23044^23046^23048^23050^23052^23054^23056^23594^23596^23598^23600^23602^23604^23606^24298^24300^24302^24304^24306^24308^24310^25810^25812^25814^25816 UD_COSMOS 9813 TOTAL 177 Size: 3.2T
            Hide
            lauren Lauren MacArthur added a comment - - edited

            The above comprises the new dataset for weekly processing. The total size of the processing data products is 8.9T. This is under Hsin-Fang Chiang's 10T limit mentioned above, but we plan to start running the qa scripts in pipe_analysis as part of the bi-weekly processing, so the plots generated will increase the size of the data products. I am running the scripts now and will post updated total size numbers when I have them. UPDATE: The numbers have been updated and reflect the size of the w_2017_51 processing (which includes compression) and all but the visit level plots which are running now (so they will grow a bit larger, but at 6.3T, we are well within our limit, so all is good on that front).

            I believe all of the criteria above are met modulo the fact that we have decided to abandon (for now) including a tract overlapping with the HSC-I2 data currently on lsst-dev. These data are from commissioning runs so have only 30s exposures (science images in HSC-I are 270-300s) and an odd dither pattern such that there is very little area with "full" (15x30s) depth. Additionally, the PDR1 for this field only includes UDEEP data whose footprint straddles several tracts (i.e. is not centered on a single tract), so is not ideal for this context. We will revisit this issue in the future (possibly when more public HSC-I2 become available).

            Show
            lauren Lauren MacArthur added a comment - - edited The above comprises the new dataset for weekly processing. The total size of the processing data products is 8.9T. This is under Hsin-Fang Chiang 's 10T limit mentioned above, but we plan to start running the qa scripts in pipe_analysis as part of the bi-weekly processing, so the plots generated will increase the size of the data products. I am running the scripts now and will post updated total size numbers when I have them. UPDATE : The numbers have been updated and reflect the size of the w_2017_51 processing (which includes compression) and all but the visit level plots which are running now (so they will grow a bit larger, but at 6.3T, we are well within our limit, so all is good on that front). I believe all of the criteria above are met modulo the fact that we have decided to abandon (for now) including a tract overlapping with the HSC-I2 data currently on lsst-dev . These data are from commissioning runs so have only 30s exposures (science images in HSC-I are 270-300s) and an odd dither pattern such that there is very little area with "full" (15x30s) depth. Additionally, the PDR1 for this field only includes UDEEP data whose footprint straddles several tracts (i.e. is not centered on a single tract), so is not ideal for this context. We will revisit this issue in the future (possibly when more public HSC-I2 become available).
            Hide
            lauren Lauren MacArthur added a comment -

            In making what were to be the final quality checks on these fields, I discovered the issue of DM-12882, so reprocessing must be done. This is running now with the w_2017_48 along with master of meas_mosaic (to get the fix).

            Show
            lauren Lauren MacArthur added a comment - In making what were to be the final quality checks on these fields, I discovered the issue of DM-12882 , so reprocessing must be done. This is running now with the w_2017_48 along with master of meas_mosaic (to get the fix).
            Hide
            swinbank John Swinbank added a comment -

            Hey folks, what's the status with this ticket?

            Show
            swinbank John Swinbank added a comment - Hey folks, what's the status with this ticket?
            Hide
            lauren Lauren MacArthur added a comment -

            Sorry, got side tracked with a bunch of other "issues".  I hope to get this into review by tomorrow evening.

            Show
            lauren Lauren MacArthur added a comment - Sorry, got side tracked with a bunch of other "issues".  I hope to get this into review by tomorrow evening.
            Hide
            lauren Lauren MacArthur added a comment -

            I have attached the seeing FWHM distributions from the SFP measurement for each Visit+CCD.  They are all plotted on the same scale for ease of comparison.  Indeed, these 3 tracts display a range in seeing distributions.  In particular, the VVDS field does indeed include "too-good-seeing" visits, particularly in the HSC-I filter that extends down to FWHM~0.35" and has a narrow distribution, extending only to ~0.7".  In contrast, the UD-COSMOS field kicks in at ~0.45" and is much more extended, out to ~1.25".

            Show
            lauren Lauren MacArthur added a comment - I have attached the seeing FWHM distributions from the SFP measurement for each Visit+CCD.  They are all plotted on the same scale for ease of comparison.  Indeed, these 3 tracts display a range in seeing distributions.  In particular, the VVDS field does indeed include "too-good-seeing" visits, particularly in the HSC-I filter that extends down to FWHM~0.35" and has a narrow distribution, extending only to ~0.7".  In contrast, the UD-COSMOS field kicks in at ~0.45" and is much more extended, out to ~1.25".
            Hide
            lauren Lauren MacArthur added a comment -

            In case it's of interest, here are some rough estimates of the limiting magnitudes in a few patches per tract for the coadds.

             

            WIDE_VVDS_9697:

            Rough estimate of limiting HSC-G mag = 24.67 (detThresh = 5.0) for tract 9697 patch 4,4
            Rough estimate of limiting HSC-R mag = 24.87 (detThresh = 5.0) for tract 9697 patch 4,4
            Rough estimate of limiting HSC-I mag = 25.11 (detThresh = 5.0) for tract 9697 patch 4,4
            Rough estimate of limiting HSC-Z mag = 23.84 (detThresh = 5.0) for tract 9697 patch 4,4
            Rough estimate of limiting HSC-Y mag = 23.17 (detThresh = 5.0) for tract 9697 patch 4,4
             
            Rough estimate of limiting HSC-G mag = 24.61 (detThresh = 5.0) for tract 9697 patch 0,0
            Rough estimate of limiting HSC-R mag = 25.11 (detThresh = 5.0) for tract 9697 patch 0,0
            Rough estimate of limiting HSC-I mag = 24.89 (detThresh = 5.0) for tract 9697 patch 0,0
            Rough estimate of limiting HSC-Z mag = 23.91 (detThresh = 5.0) for tract 9697 patch 0,0
            Rough estimate of limiting HSC-Y mag = 23.19 (detThresh = 5.0) for tract 9697 patch 0,0
             
            Rough estimate of limiting HSC-G mag = 24.53 (detThresh = 5.0) for tract 9697 patch 8,8
            Rough estimate of limiting HSC-R mag = 24.94 (detThresh = 5.0) for tract 9697 patch 8,8
            Rough estimate of limiting HSC-I mag = 25.19 (detThresh = 5.0) for tract 9697 patch 8,8
            Rough estimate of limiting HSC-Z mag = 23.82 (detThresh = 5.0) for tract 9697 patch 8,8
            Rough estimate of limiting HSC-Y mag = 23.09 (detThresh = 5.0) for tract 9697 patch 8,8
            

            WIDE_GAMA15H_9615:

            Rough estimate of limiting HSC-G mag = 25.05 (detThresh = 5.0) for tract 9615 patch 4,4
            Rough estimate of limiting HSC-R mag = 25.35 (detThresh = 5.0) for tract 9615 patch 4,4
            Rough estimate of limiting HSC-I mag = 25.09 (detThresh = 5.0) for tract 9615 patch 4,4
            Rough estimate of limiting HSC-Z mag = 23.86 (detThresh = 5.0) for tract 9615 patch 4,4
            Rough estimate of limiting HSC-Y mag = 22.98 (detThresh = 5.0) for tract 9615 patch 4,4
             
            Rough estimate of limiting HSC-G mag = 24.66 (detThresh = 5.0) for tract 9615 patch 8,8
            Rough estimate of limiting HSC-R mag = 25.06 (detThresh = 5.0) for tract 9615 patch 8,8
            Rough estimate of limiting HSC-I mag = 24.40 (detThresh = 5.0) for tract 9615 patch 8,8
            Rough estimate of limiting HSC-Z mag = 23.75 (detThresh = 5.0) for tract 9615 patch 8,8
            Rough estimate of limiting HSC-Y mag = 22.99 (detThresh = 5.0) for tract 9615 patch 8,8
             
            Rough estimate of limiting HSC-G mag = 25.15 (detThresh = 5.0) for tract 9615 patch 0,0
            Rough estimate of limiting HSC-R mag = 24.99 (detThresh = 5.0) for tract 9615 patch 0,0
            Rough estimate of limiting HSC-I mag = 24.95 (detThresh = 5.0) for tract 9615 patch 0,0
            Rough estimate of limiting HSC-Z mag = 23.70 (detThresh = 5.0) for tract 9615 patch 0,0
            Rough estimate of limiting HSC-Y mag = 22.87 (detThresh = 5.0) for tract 9615 patch 0,0
            

            UD_COSMOS_9813:

            Rough estimate of limiting HSC-G mag = 25.59 (detThresh = 5.0) for tract 9813 patch 4,4
            Rough estimate of limiting HSC-R mag = 26.09 (detThresh = 5.0) for tract 9813 patch 4,4
            Rough estimate of limiting HSC-I mag = 25.55 (detThresh = 5.0) for tract 9813 patch 4,4
            Rough estimate of limiting HSC-Z mag = 25.18 (detThresh = 5.0) for tract 9813 patch 4,4
            Rough estimate of limiting HSC-Y mag = 23.93 (detThresh = 5.0) for tract 9813 patch 4,4
            Rough estimate of limiting NB921 mag = 24.40 (detThresh = 5.0) for tract 9813 patch 4,4
             
            Rough estimate of limiting HSC-G mag = 25.04 (detThresh = 5.0) for tract 9813 patch 7,7
            Rough estimate of limiting HSC-R mag = 25.86 (detThresh = 5.0) for tract 9813 patch 7,7
            Rough estimate of limiting HSC-I mag = 25.18 (detThresh = 5.0) for tract 9813 patch 7,7
            Rough estimate of limiting HSC-Z mag = 24.78 (detThresh = 5.0) for tract 9813 patch 7,7
            Rough estimate of limiting HSC-Y mag = 23.52 (detThresh = 5.0) for tract 9813 patch 7,7
            Rough estimate of limiting NB921 mag = 23.77 (detThresh = 5.0) for tract 9813 patch 7,7
             
            Rough estimate of limiting HSC-G mag = 24.47 (detThresh = 5.0) for tract 9813 patch 1,1
            Rough estimate of limiting HSC-R mag = 25.32 (detThresh = 5.0) for tract 9813 patch 1,1
            Rough estimate of limiting HSC-I mag = 24.48 (detThresh = 5.0) for tract 9813 patch 1,1
            Rough estimate of limiting HSC-Z mag = 23.91 (detThresh = 5.0) for tract 9813 patch 1,1
            Rough estimate of limiting HSC-Y mag = 22.54 (detThresh = 5.0) for tract 9813 patch 1,1
            Rough estimate of limiting NB921 mag = 23.27 (detThresh = 5.0) for tract 9813 patch 1,1
            

            Show
            lauren Lauren MacArthur added a comment - In case it's of interest, here are some rough estimates of the limiting magnitudes in a few patches per tract for the coadds.   WIDE_VVDS_9697: Rough estimate of limiting HSC-G mag = 24.67 (detThresh = 5.0) for tract 9697 patch 4,4 Rough estimate of limiting HSC-R mag = 24.87 (detThresh = 5.0) for tract 9697 patch 4,4 Rough estimate of limiting HSC-I mag = 25.11 (detThresh = 5.0) for tract 9697 patch 4,4 Rough estimate of limiting HSC-Z mag = 23.84 (detThresh = 5.0) for tract 9697 patch 4,4 Rough estimate of limiting HSC-Y mag = 23.17 (detThresh = 5.0) for tract 9697 patch 4,4   Rough estimate of limiting HSC-G mag = 24.61 (detThresh = 5.0) for tract 9697 patch 0,0 Rough estimate of limiting HSC-R mag = 25.11 (detThresh = 5.0) for tract 9697 patch 0,0 Rough estimate of limiting HSC-I mag = 24.89 (detThresh = 5.0) for tract 9697 patch 0,0 Rough estimate of limiting HSC-Z mag = 23.91 (detThresh = 5.0) for tract 9697 patch 0,0 Rough estimate of limiting HSC-Y mag = 23.19 (detThresh = 5.0) for tract 9697 patch 0,0   Rough estimate of limiting HSC-G mag = 24.53 (detThresh = 5.0) for tract 9697 patch 8,8 Rough estimate of limiting HSC-R mag = 24.94 (detThresh = 5.0) for tract 9697 patch 8,8 Rough estimate of limiting HSC-I mag = 25.19 (detThresh = 5.0) for tract 9697 patch 8,8 Rough estimate of limiting HSC-Z mag = 23.82 (detThresh = 5.0) for tract 9697 patch 8,8 Rough estimate of limiting HSC-Y mag = 23.09 (detThresh = 5.0) for tract 9697 patch 8,8 WIDE_GAMA15H_9615: Rough estimate of limiting HSC-G mag = 25.05 (detThresh = 5.0) for tract 9615 patch 4,4 Rough estimate of limiting HSC-R mag = 25.35 (detThresh = 5.0) for tract 9615 patch 4,4 Rough estimate of limiting HSC-I mag = 25.09 (detThresh = 5.0) for tract 9615 patch 4,4 Rough estimate of limiting HSC-Z mag = 23.86 (detThresh = 5.0) for tract 9615 patch 4,4 Rough estimate of limiting HSC-Y mag = 22.98 (detThresh = 5.0) for tract 9615 patch 4,4 Rough estimate of limiting HSC-G mag = 24.66 (detThresh = 5.0) for tract 9615 patch 8,8 Rough estimate of limiting HSC-R mag = 25.06 (detThresh = 5.0) for tract 9615 patch 8,8 Rough estimate of limiting HSC-I mag = 24.40 (detThresh = 5.0) for tract 9615 patch 8,8 Rough estimate of limiting HSC-Z mag = 23.75 (detThresh = 5.0) for tract 9615 patch 8,8 Rough estimate of limiting HSC-Y mag = 22.99 (detThresh = 5.0) for tract 9615 patch 8,8 Rough estimate of limiting HSC-G mag = 25.15 (detThresh = 5.0) for tract 9615 patch 0,0 Rough estimate of limiting HSC-R mag = 24.99 (detThresh = 5.0) for tract 9615 patch 0,0 Rough estimate of limiting HSC-I mag = 24.95 (detThresh = 5.0) for tract 9615 patch 0,0 Rough estimate of limiting HSC-Z mag = 23.70 (detThresh = 5.0) for tract 9615 patch 0,0 Rough estimate of limiting HSC-Y mag = 22.87 (detThresh = 5.0) for tract 9615 patch 0,0 UD_COSMOS_9813: Rough estimate of limiting HSC-G mag = 25.59 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting HSC-R mag = 26.09 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting HSC-I mag = 25.55 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting HSC-Z mag = 25.18 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting HSC-Y mag = 23.93 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting NB921 mag = 24.40 (detThresh = 5.0) for tract 9813 patch 4,4 Rough estimate of limiting HSC-G mag = 25.04 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting HSC-R mag = 25.86 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting HSC-I mag = 25.18 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting HSC-Z mag = 24.78 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting HSC-Y mag = 23.52 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting NB921 mag = 23.77 (detThresh = 5.0) for tract 9813 patch 7,7 Rough estimate of limiting HSC-G mag = 24.47 (detThresh = 5.0) for tract 9813 patch 1,1 Rough estimate of limiting HSC-R mag = 25.32 (detThresh = 5.0) for tract 9813 patch 1,1 Rough estimate of limiting HSC-I mag = 24.48 (detThresh = 5.0) for tract 9813 patch 1,1 Rough estimate of limiting HSC-Z mag = 23.91 (detThresh = 5.0) for tract 9813 patch 1,1 Rough estimate of limiting HSC-Y mag = 22.54 (detThresh = 5.0) for tract 9813 patch 1,1 Rough estimate of limiting NB921 mag = 23.27 (detThresh = 5.0) for tract 9813 patch 1,1
            Hide
            lauren Lauren MacArthur added a comment - - edited

            I've also been making and looking at 3-colour images of the coadds.  Here is an example from the WIDE_GAMA15H field that includes a cluster and a large spiral (which overlaps two patches, hence the discontinuity...I just screen-grabbed a manual alignment of the two patch images, so will be even more of a challenge to get right.  It is on of the "big galaxies" in Song Huang's list):

            Show
            lauren Lauren MacArthur added a comment - - edited I've also been making and looking at 3-colour images of the coadds.  Here is an example from the WIDE_GAMA15H field that includes a cluster and a large spiral (which overlaps two patches, hence the discontinuity...I just screen-grabbed a manual alignment of the two patch images, so will be even more of a challenge to get right.  It is on of the "big galaxies" in Song Huang's list):
            Hide
            lauren Lauren MacArthur added a comment -

            Also attached are the visit pointings and their overlap with the tract for HSC-I.

            Show
            lauren Lauren MacArthur added a comment - Also attached are the visit pointings and their overlap with the tract for HSC-I.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            singleFrameDriver [grep on "Failed" in singleFrameDriver logs]:
            Total failures = 8 ccds
            ===============================================================
            WIDE_VVDS_9697:

            Failed on dataId={'ccd': 70, 'filter': 'HSC-G', 'visit': 34414, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'expTime': 150.0}: RuntimeError: Unable to match sources
            Failed on dataId={'ccd': 78, 'filter': 'HSC-G', 'visit': 34412, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'expTime': 150.0}: RuntimeError: Unable to match sources
            Failed on dataId={'ccd':  6, 'filter': 'HSC-I', 'visit': 36216, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'expTime': 200.0}: RuntimeError: No matches to use for photocal
            Failed on dataId={'ccd': 12, 'filter': 'HSC-I', 'visit': 35892, 'field': 'SSP_WIDE', 'dateObs': '2015-07-20', 'pointing': 1296, 'expTime': 200.0}: RuntimeError: No matches to use for photocal
            Failed on dataId={'ccd': 21, 'filter': 'HSC-I', 'visit': 36118, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'expTime': 200.0}: RuntimeError: No matches to use for photocal
            

            UD_COSMOS_9813:

            Failed on dataId={'ccd': 68, 'filter': 'HSC-G', 'visit': 11698, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-11-18', 'pointing': 1052, 'expTime': 300.0}: RuntimeError: No matches to use for photocal
            Failed on dataId={'ccd':  1, 'filter':'NB0921', 'visit': 23600, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-18', 'pointing': 1172, 'expTime': 900.0}: RuntimeError: No matches to use for photocal
            Failed on dataId={'ccd':  1, 'filter': 'HSC-Z', 'visit': 17934, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-16', 'pointing': 1111, 'expTime': 300.0}: RuntimeError: No matches to use for photocal
            

            coaddDriver.select [grep on "Removing visit" in coaddDriver logs]:
            =================================================================
            NOTE: multiple entries for ccds overlapping multiple patches
            Reasons for removal include:
            because median e residual too large: 0.008402 vs 0.007000
            because scaled size scatter is too large: 0.010015 vs 0.009000
            =================================================================

            WIDE_VVDS_9697_G:  12
            WIDE_VVDS_9697_R:  25
            WIDE_VVDS_9697_I: 562 [~600 ccds with seeing FWHM < 0.425"]
            WIDE_VVDS_9697_Z:  44 [~150 ccds with seeing FWHM < 0.45"]
            WIDE_VVDS_9697_Y: 124 [~50 ccds with seeing FWHM < 0.45"]
             
            WIDE_GAMA15H_9615_G:  14
            WIDE_GAMA15H_9615_R: 124 [~150 ccds with seeing FWHM < 0.45"]
            WIDE_GAMA15H_9615_I:  30
            WIDE_GAMA15H_9615_Z:   6
            WIDE_GAMA15H_9615_Y:  22
             
            UD_COSMOS_9813_G:      43
            UD_COSMOS_9813_R:     278 [nothing obvious in PSF distribution alone]
            UD_COSMOS_9813_I:      79
            UD_COSMOS_9813_Z:      77
            UD_COSMOS_9813_Y:      28
            UD_COSMOS_9813_NB0921: 28
            

            Number of patches [wc of calexps in deepCoadd-results dir]:
            ================================================

            WIDE_VVDS_9697_G: 81
            WIDE_VVDS_9697_R: 81
            WIDE_VVDS_9697_I: 81
            WIDE_VVDS_9697_Z: 81
            WIDE_VVDS_9697_Y: 81
             
            WIDE_GAMA15H_9615_G: 81
            WIDE_GAMA15H_9615_R: 81
            WIDE_GAMA15H_9615_I: 81
            WIDE_GAMA15H_9615_Z: 81
            WIDE_GAMA15H_9615_Y: 81
             
                                       [missing patches]
            UD_COSMOS_9813_G:       78 [0,0 0,8 8,0]
            UD_COSMOS_9813_R:       74 [0,0 0,1 0,7 0,8 1,8 8,0 8,8]
            UD_COSMOS_9813_I:       79 [0,0 0,8]
            UD_COSMOS_9813_Z:       78 [0,0 0,8 8,8]
            UD_COSMOS_9813_Y:       79 [0,0 0,8]
            UD_COSMOS_9813_NB0921:  76 [0,0 0,7 0,8 1,8 8,8]
            

            Show
            lauren Lauren MacArthur added a comment - - edited singleFrameDriver [grep on "Failed" in singleFrameDriver logs] : Total failures = 8 ccds =============================================================== WIDE_VVDS_9697 : Failed on dataId={'ccd': 70, 'filter': 'HSC-G', 'visit': 34414, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'expTime': 150.0}: RuntimeError: Unable to match sources Failed on dataId={'ccd': 78, 'filter': 'HSC-G', 'visit': 34412, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'expTime': 150.0}: RuntimeError: Unable to match sources Failed on dataId={'ccd': 6, 'filter': 'HSC-I', 'visit': 36216, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'expTime': 200.0}: RuntimeError: No matches to use for photocal Failed on dataId={'ccd': 12, 'filter': 'HSC-I', 'visit': 35892, 'field': 'SSP_WIDE', 'dateObs': '2015-07-20', 'pointing': 1296, 'expTime': 200.0}: RuntimeError: No matches to use for photocal Failed on dataId={'ccd': 21, 'filter': 'HSC-I', 'visit': 36118, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'expTime': 200.0}: RuntimeError: No matches to use for photocal UD_COSMOS_9813 : Failed on dataId={'ccd': 68, 'filter': 'HSC-G', 'visit': 11698, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-11-18', 'pointing': 1052, 'expTime': 300.0}: RuntimeError: No matches to use for photocal Failed on dataId={'ccd': 1, 'filter':'NB0921', 'visit': 23600, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-18', 'pointing': 1172, 'expTime': 900.0}: RuntimeError: No matches to use for photocal Failed on dataId={'ccd': 1, 'filter': 'HSC-Z', 'visit': 17934, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-16', 'pointing': 1111, 'expTime': 300.0}: RuntimeError: No matches to use for photocal coaddDriver.select [grep on "Removing visit" in coaddDriver logs] : ================================================================= NOTE: multiple entries for ccds overlapping multiple patches Reasons for removal include: because median e residual too large: 0.008402 vs 0.007000 because scaled size scatter is too large: 0.010015 vs 0.009000 ================================================================= WIDE_VVDS_9697_G: 12 WIDE_VVDS_9697_R: 25 WIDE_VVDS_9697_I: 562 [~600 ccds with seeing FWHM < 0.425"] WIDE_VVDS_9697_Z: 44 [~150 ccds with seeing FWHM < 0.45"] WIDE_VVDS_9697_Y: 124 [~50 ccds with seeing FWHM < 0.45"]   WIDE_GAMA15H_9615_G: 14 WIDE_GAMA15H_9615_R: 124 [~150 ccds with seeing FWHM < 0.45"] WIDE_GAMA15H_9615_I: 30 WIDE_GAMA15H_9615_Z: 6 WIDE_GAMA15H_9615_Y: 22   UD_COSMOS_9813_G: 43 UD_COSMOS_9813_R: 278 [nothing obvious in PSF distribution alone] UD_COSMOS_9813_I: 79 UD_COSMOS_9813_Z: 77 UD_COSMOS_9813_Y: 28 UD_COSMOS_9813_NB0921: 28 Number of patches [wc of calexps in deepCoadd-results dir] : ================================================ WIDE_VVDS_9697_G: 81 WIDE_VVDS_9697_R: 81 WIDE_VVDS_9697_I: 81 WIDE_VVDS_9697_Z: 81 WIDE_VVDS_9697_Y: 81   WIDE_GAMA15H_9615_G: 81 WIDE_GAMA15H_9615_R: 81 WIDE_GAMA15H_9615_I: 81 WIDE_GAMA15H_9615_Z: 81 WIDE_GAMA15H_9615_Y: 81   [missing patches] UD_COSMOS_9813_G: 78 [0,0 0,8 8,0] UD_COSMOS_9813_R: 74 [0,0 0,1 0,7 0,8 1,8 8,0 8,8] UD_COSMOS_9813_I: 79 [0,0 0,8] UD_COSMOS_9813_Z: 78 [0,0 0,8 8,8] UD_COSMOS_9813_Y: 79 [0,0 0,8] UD_COSMOS_9813_NB0921: 76 [0,0 0,7 0,8 1,8 8,8]
            Hide
            lauren Lauren MacArthur added a comment -

            Reductions from the w_2017_51 weekly live in /datasets/hsc/repo/rerun/private/lauren/DM-11345/w51/ on lsat-dev

            The coadd and color QA scripts have been run and parquet tables created, so those can be perused (most easily with the tools of Tim Morton). The visit level scripts are running now.

            Show
            lauren Lauren MacArthur added a comment - Reductions from the w_2017_51 weekly live in /datasets/hsc/repo/rerun/private/lauren/ DM-11345 /w51/ on lsat-dev The coadd and color QA scripts have been run and parquet tables created, so those can be perused (most easily with the tools of Tim Morton ). The visit level scripts are running now.
            Hide
            lauren Lauren MacArthur added a comment -

            I've attached the survey footprints (taken from https://hsc-release.mtk.nao.ac.jp/doc/index.php/data/).  Note that WIDE_GAMA15H_9615 satisfies our "survey's edge" criterion.

             

            Show
            lauren Lauren MacArthur added a comment - I've attached the survey footprints (taken from https://hsc-release.mtk.nao.ac.jp/doc/index.php/data/ ).  Note that WIDE_GAMA15H_9615 satisfies our "survey's edge" criterion.  
            Hide
            lauren Lauren MacArthur added a comment -

            I have included as much info about the proposed "RC2" dataset to ensure all our criteria (delineated above) are met. Jim, would you mind having a look to see if you're satisfied or would like to see any more characteristics (or clarifications of what I've presented) before signing off. I image Robert Lupton, Paul Price, and Bob Armstrong might want to weigh in as well.

            Also, I've had a discussion with Hsin-Fang Chiang about changing from the current RC dataset she is running as the bi-weekly dataset to the one defined here (as well as adding the running of the QA scripts to her workflow). If there are no major concerns or issues, this switch-over could happen as early as the w_2018_06 release bi-weekly run.

            Show
            lauren Lauren MacArthur added a comment - I have included as much info about the proposed "RC2" dataset to ensure all our criteria (delineated above) are met. Jim, would you mind having a look to see if you're satisfied or would like to see any more characteristics (or clarifications of what I've presented) before signing off. I image Robert Lupton , Paul Price , and Bob Armstrong might want to weigh in as well. Also, I've had a discussion with Hsin-Fang Chiang about changing from the current RC dataset she is running as the bi-weekly dataset to the one defined here (as well as adding the running of the QA scripts to her workflow). If there are no major concerns or issues, this switch-over could happen as early as the w_2018_06 release bi-weekly run.
            Hide
            jbosch Jim Bosch added a comment -

            Looks great; you've certainly done due diligence on these!

            I think the two remaining questions (which should not block closing this ticket) are:

            • What do we call the new dataset?
            • When do we start using it.

            I'll (re-)start those discussions on #dm-hsc-reprocessing in Slack to see if we can get some resolution on those issues.

            Show
            jbosch Jim Bosch added a comment - Looks great; you've certainly done due diligence on these! I think the two remaining questions (which should not block closing this ticket) are: What do we call the new dataset? When do we start using it. I'll (re-)start those discussions on #dm-hsc-reprocessing in Slack to see if we can get some resolution on those issues.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks, Jim!

            Show
            lauren Lauren MacArthur added a comment - Thanks, Jim!

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Jim Bosch
                Watchers:
                Bob Armstrong, Dino Bektesevic, Hsin-Fang Chiang, Ian Sullivan, Jeffrey Carlin, Jim Bosch, John Swinbank, Lauren MacArthur, Paul Price, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel