Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10128

Obtain missing public HSC RC data and add them to /datasets/hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Some of the HSC RC dataset were not included in HSC-PDR1/DM-9683//datasets/hsc/raw/ssp_pdr1 although some of them are publicly available.

      Obtain them from NAOJ if possible, so to allow processing the exactly same RC dataset and facilitate QA.

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-10084 [ DM-10084 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Downloading these 62 visits (*112 ccd each = 6944 files) from SMOKA:

            274^276^278^280^282^284^286^288^290^292^294^296^298^300^302^306^308^310^312^314^316^320^334^342^364^366^368^370^1236^1858^1860^1862^1878^9864^9890^11742^28354^28356^28358^28360^28362^28364^28366^28368^28370^28372^28374^28376^28378^28380^28382^28384^28386^28388^28390^28392^28394^28396^28398^28400^28402^29352

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Downloading these 62 visits (*112 ccd each = 6944 files) from SMOKA: 274^276^278^280^282^284^286^288^290^292^294^296^298^300^302^306^308^310^312^314^316^320^334^342^364^366^368^370^1236^1858^1860^1862^1878^9864^9890^11742^28354^28356^28358^28360^28362^28364^28366^28368^28370^28372^28374^28376^28378^28380^28382^28384^28386^28388^28390^28392^28394^28396^28398^28400^28402^29352
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggered by RFC-323 [ RFC-323 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues, may you please create /datasets/hsc/raw/ssp_extra for me?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues , may you please create /datasets/hsc/raw/ssp_extra for me?
            Hide
            gdaues Greg Daues added a comment -

            This directory /datasets/hsc/raw/ssp_extra has been created; it is ready to receive data from 'hchiang2'.

            Show
            gdaues Greg Daues added a comment - This directory /datasets/hsc/raw/ssp_extra has been created; it is ready to receive data from 'hchiang2'.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Sorry, Greg Daues, may you please also give me the permission of editing /datasets/hsc/repo/registry.sqlite3 and adding new items inside /datasets/hsc/repo/SSP_WIDE/ and /datasets/hsc/repo/SSP_UDEEP_COSMOS? Thank you!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Sorry, Greg Daues , may you please also give me the permission of editing /datasets/hsc/repo/registry.sqlite3 and adding new items inside /datasets/hsc/repo/SSP_WIDE/ and /datasets/hsc/repo/SSP_UDEEP_COSMOS ? Thank you!
            Hide
            gdaues Greg Daues added a comment -

            These should now have immutable flag turned off, so I think you can proceed.

            Show
            gdaues Greg Daues added a comment - These should now have immutable flag turned off, so I think you can proceed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            These 6944 files have been copied to /datasets/hsc/raw/ssp_extra, and then ingested into the butler repo using stack w_2017_15.

            $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_UDEEP_COSMOS' -s
            filter  field                expTime visit
            HSC-G   SSP_UDEEP_COSMOS      5460.0 1886^11688^11690..11712:2^29324^29326^29336^29340^29350^29352
            HSC-I   SSP_UDEEP_COSMOS      9960.0 1226..1232:2^1236..1248:2^19656..19662:2^19680..19684:2^19694..19698:2^19708..19712:2^30482..30504:2
            HSC-R   SSP_UDEEP_COSMOS      5490.0 1200..1222:2^23690..23694:2^23704^23706^23716^23718
            HSC-Y   SSP_UDEEP_COSMOS     23550.0 274..302:2^306..334:2^340..370:2^1856..1862:2^1868..1882:2^11716..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2
            HSC-Z   SSP_UDEEP_COSMOS     16410.0 1164..1194:2^17898..17908:2^17926..17934:2^17944..17952:2^17962^28354^28356..28402:2
            NB0921  SSP_UDEEP_COSMOS     25230.0 23036..23056:2^23594..23606:2^24298..24310:2^25810..25816:2
             
            $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_WIDE' -s
            filter  field                expTime visit
            HSC-G   SSP_WIDE             115530.0 6292..6350:2^9838..9872:2^9880..9908:2^9912..9918:2^11562..11624:2^11630..11640:2^11646..11656:2^11664..11684:2^15186..15196:2^15210..15214:2^15222..15226:2^15234..15238:2^15264..15268:2^15272..15278:2^15282..15318:2^15324..15400:2^25824..25882:2^25886..26014:2^26018..26110:2^29362..29480:2^29484..29502:2^29508^29510^29514^29518^29520..29526:2^29588..29614:2^29618^29620^29624..29630:2^34176..34204:2^34260..34328:2^34332..34344:2^34348..34356:4^34358..34486:2^38206..38266:2^38274..38300:2^38304..38334:2^38338..38354:2^38358^38362^38364^38368^38370..38406:2^38414..38440:2^38444..38470:2^42142..42176:2^42180..42190:2^42194..42342:2^42382..42560:2^46872^46876^46878..46948:2
            HSC-I   SSP_WIDE             161100.0 1252..1318:2^1322..1346:2^7212..7238:2^7286..7314:2^7318..7322:2^7338..7360:2^7364..7374:2^7378..7404:2^7408..7416:2^7424^14078^14080..14104:2^14116..14150:2^14160..14182:2^14192..14214:2^16220..16272:2^16284^16292^16294..16380:2^19378^19382^19384..19432:2^19444..19486:2^19490..19622:2^19628..19654:2^24360..24426:2^24430..24434:2^24438..24494:2^24504..24556:2^24938..25080:2^25084^25088^25090..25150:2^30508..30528:2^30534..30550:2^30554..30594:2^30672..30726:2^35832..35918:2^35922..35954:2^35958..35974:2^36112..36122:2^36126..36266:2^37976..38072:2^45848..45854:2^45858..45876:2^45890..45916:2^45920..45932:2^45936..46008:2^46016..46024:2
            HSC-R   SSP_WIDE             102300.0 7110..7154:2^7158^7164^7166^7168^7204..7208:2^11382..11400:2^11404..11450:2^11458..11480:2^11486..11508:2^11514..11556:2^14836..14848:2^14868..14912:2^14920..14928:2^14932..14990:2^16154..16168:2^16176..16182:2^16190..16196:2^16204..16210:2^23610..23618:2^23622..23670:2^23728..23930:2^28902..28906:2^28910..28930:2^28940..28996:2^29058^29060^29064..29108:2^34030..34098:2^34582..34630:2^34634..34664:2^34668..34774:2^34808..34816:2^40752..40844:2^40848..40972:2^41010..41178:2^45158..45206:2^45210..45230:2^45234^45238^45240^45242^45246..45252:2^45256..45284:2^45288..45320:2^45324^45340^45346..45374:2
            HSC-Y   SSP_WIDE             161800.0 374..432:2^436..480:2^6370..6410:2^6460..6500:2^6512^6522^6524..6546:2^6556..6568:2^13130..13156:2^13166..13172:2^13178..13198:2^13208..13226:2^13236..13254:2^14294..14374:2^14380..14408:2^16054..16062:4^16064^16066^16082..16110:2^16118..16132:2^22438..22540:2^22544..22550:2^22554..22566:2^22572^22584^22586..22598:2^26884..26986:2^26994..27002:2^27006..27084:2^29898..29916:2^29928..29974:2^29978^29998^30000^30004^30006^30028^30036^30038^30050^30074^30076..30080:2^30084^30086^30090..30122:2^30128..30138:2^30142..30148:2^30152..30174:2^30178^30184^30186..30192:2^30204^32258^32260..32360:2^32364..32416:2^32468..32506:2^34870..34916:2^34920..34948:2^34954^36600^36604^36608^36610..36628:2^36632..36720:2^36724..36838:2^36874..36880:2^37520..37574:2^37610..37628:2^39312..39368:2^44008..44036:2^44044..44060:2^44064^44066^44074..44106:2^44110..44166:2^44176..44208:2
            HSC-Z   SSP_WIDE             173210.0 8794..8806:2^8810..8816:2^9596..9626:2^9682..9690:2^9694..9790:2^13272..13288:2^13304^13306^13340^13342^13356..13452:2^15078^15080^15084..15182:2^17654..17686:2^17690..17726:2^17730..17754:2^17758..17850:2^17854..17894:2^23108..23260:2^27088..27160:2^28432^28438^28440^31262^31264..31270:2^31274..31282:4^31284..31350:2^31370..31448:2^31468..31530:2^33762..33776:2^33780..33846:2^33850..33868:2^33872..33952:2^36040..36058:2^36062..36108:2^36324..36340:2^36344..36372:2^36376^36378^36382..36398:2^36402..36508:2^38876..39028:2^44444..44512:2^44516..44536:2^44542^44560^44590^44614^44616^44620^44622..44686:2^44690..44772:2^47042^47056^47060..47072:4^47074^47076^47080^47092^47094^47098^47112^47116^47118..47126:2^47130^47134^47136^47140^47142..47148:2^47154
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - These 6944 files have been copied to /datasets/hsc/raw/ssp_extra , and then ingested into the butler repo using stack w_2017_15 . $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_UDEEP_COSMOS' -s filter field expTime visit HSC-G SSP_UDEEP_COSMOS 5460.0 1886 ^ 11688 ^ 11690 .. 11712 : 2 ^ 29324 ^ 29326 ^ 29336 ^ 29340 ^ 29350 ^ 29352 HSC-I SSP_UDEEP_COSMOS 9960.0 1226 .. 1232 : 2 ^ 1236 .. 1248 : 2 ^ 19656 .. 19662 : 2 ^ 19680 .. 19684 : 2 ^ 19694 .. 19698 : 2 ^ 19708 .. 19712 : 2 ^ 30482 .. 30504 : 2 HSC-R SSP_UDEEP_COSMOS 5490.0 1200 .. 1222 : 2 ^ 23690 .. 23694 : 2 ^ 23704 ^ 23706 ^ 23716 ^ 23718 HSC-Y SSP_UDEEP_COSMOS 23550.0 274 .. 302 : 2 ^ 306 .. 334 : 2 ^ 340 .. 370 : 2 ^ 1856 .. 1862 : 2 ^ 1868 .. 1882 : 2 ^ 11716 .. 11742 : 2 ^ 22602 .. 22608 : 2 ^ 22626 .. 22632 : 2 ^ 22642 .. 22648 : 2 ^ 22658 .. 22664 : 2 HSC-Z SSP_UDEEP_COSMOS 16410.0 1164 .. 1194 : 2 ^ 17898 .. 17908 : 2 ^ 17926 .. 17934 : 2 ^ 17944 .. 17952 : 2 ^ 17962 ^ 28354 ^ 28356 .. 28402 : 2 NB0921 SSP_UDEEP_COSMOS 25230.0 23036 .. 23056 : 2 ^ 23594 .. 23606 : 2 ^ 24298 .. 24310 : 2 ^ 25810 .. 25816 : 2   $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_WIDE' -s filter field expTime visit HSC-G SSP_WIDE 115530.0 6292 .. 6350 : 2 ^ 9838 .. 9872 : 2 ^ 9880 .. 9908 : 2 ^ 9912 .. 9918 : 2 ^ 11562 .. 11624 : 2 ^ 11630 .. 11640 : 2 ^ 11646 .. 11656 : 2 ^ 11664 .. 11684 : 2 ^ 15186 .. 15196 : 2 ^ 15210 .. 15214 : 2 ^ 15222 .. 15226 : 2 ^ 15234 .. 15238 : 2 ^ 15264 .. 15268 : 2 ^ 15272 .. 15278 : 2 ^ 15282 .. 15318 : 2 ^ 15324 .. 15400 : 2 ^ 25824 .. 25882 : 2 ^ 25886 .. 26014 : 2 ^ 26018 .. 26110 : 2 ^ 29362 .. 29480 : 2 ^ 29484 .. 29502 : 2 ^ 29508 ^ 29510 ^ 29514 ^ 29518 ^ 29520 .. 29526 : 2 ^ 29588 .. 29614 : 2 ^ 29618 ^ 29620 ^ 29624 .. 29630 : 2 ^ 34176 .. 34204 : 2 ^ 34260 .. 34328 : 2 ^ 34332 .. 34344 : 2 ^ 34348 .. 34356 : 4 ^ 34358 .. 34486 : 2 ^ 38206 .. 38266 : 2 ^ 38274 .. 38300 : 2 ^ 38304 .. 38334 : 2 ^ 38338 .. 38354 : 2 ^ 38358 ^ 38362 ^ 38364 ^ 38368 ^ 38370 .. 38406 : 2 ^ 38414 .. 38440 : 2 ^ 38444 .. 38470 : 2 ^ 42142 .. 42176 : 2 ^ 42180 .. 42190 : 2 ^ 42194 .. 42342 : 2 ^ 42382 .. 42560 : 2 ^ 46872 ^ 46876 ^ 46878 .. 46948 : 2 HSC-I SSP_WIDE 161100.0 1252 .. 1318 : 2 ^ 1322 .. 1346 : 2 ^ 7212 .. 7238 : 2 ^ 7286 .. 7314 : 2 ^ 7318 .. 7322 : 2 ^ 7338 .. 7360 : 2 ^ 7364 .. 7374 : 2 ^ 7378 .. 7404 : 2 ^ 7408 .. 7416 : 2 ^ 7424 ^ 14078 ^ 14080 .. 14104 : 2 ^ 14116 .. 14150 : 2 ^ 14160 .. 14182 : 2 ^ 14192 .. 14214 : 2 ^ 16220 .. 16272 : 2 ^ 16284 ^ 16292 ^ 16294 .. 16380 : 2 ^ 19378 ^ 19382 ^ 19384 .. 19432 : 2 ^ 19444 .. 19486 : 2 ^ 19490 .. 19622 : 2 ^ 19628 .. 19654 : 2 ^ 24360 .. 24426 : 2 ^ 24430 .. 24434 : 2 ^ 24438 .. 24494 : 2 ^ 24504 .. 24556 : 2 ^ 24938 .. 25080 : 2 ^ 25084 ^ 25088 ^ 25090 .. 25150 : 2 ^ 30508 .. 30528 : 2 ^ 30534 .. 30550 : 2 ^ 30554 .. 30594 : 2 ^ 30672 .. 30726 : 2 ^ 35832 .. 35918 : 2 ^ 35922 .. 35954 : 2 ^ 35958 .. 35974 : 2 ^ 36112 .. 36122 : 2 ^ 36126 .. 36266 : 2 ^ 37976 .. 38072 : 2 ^ 45848 .. 45854 : 2 ^ 45858 .. 45876 : 2 ^ 45890 .. 45916 : 2 ^ 45920 .. 45932 : 2 ^ 45936 .. 46008 : 2 ^ 46016 .. 46024 : 2 HSC-R SSP_WIDE 102300.0 7110 .. 7154 : 2 ^ 7158 ^ 7164 ^ 7166 ^ 7168 ^ 7204 .. 7208 : 2 ^ 11382 .. 11400 : 2 ^ 11404 .. 11450 : 2 ^ 11458 .. 11480 : 2 ^ 11486 .. 11508 : 2 ^ 11514 .. 11556 : 2 ^ 14836 .. 14848 : 2 ^ 14868 .. 14912 : 2 ^ 14920 .. 14928 : 2 ^ 14932 .. 14990 : 2 ^ 16154 .. 16168 : 2 ^ 16176 .. 16182 : 2 ^ 16190 .. 16196 : 2 ^ 16204 .. 16210 : 2 ^ 23610 .. 23618 : 2 ^ 23622 .. 23670 : 2 ^ 23728 .. 23930 : 2 ^ 28902 .. 28906 : 2 ^ 28910 .. 28930 : 2 ^ 28940 .. 28996 : 2 ^ 29058 ^ 29060 ^ 29064 .. 29108 : 2 ^ 34030 .. 34098 : 2 ^ 34582 .. 34630 : 2 ^ 34634 .. 34664 : 2 ^ 34668 .. 34774 : 2 ^ 34808 .. 34816 : 2 ^ 40752 .. 40844 : 2 ^ 40848 .. 40972 : 2 ^ 41010 .. 41178 : 2 ^ 45158 .. 45206 : 2 ^ 45210 .. 45230 : 2 ^ 45234 ^ 45238 ^ 45240 ^ 45242 ^ 45246 .. 45252 : 2 ^ 45256 .. 45284 : 2 ^ 45288 .. 45320 : 2 ^ 45324 ^ 45340 ^ 45346 .. 45374 : 2 HSC-Y SSP_WIDE 161800.0 374 .. 432 : 2 ^ 436 .. 480 : 2 ^ 6370 .. 6410 : 2 ^ 6460 .. 6500 : 2 ^ 6512 ^ 6522 ^ 6524 .. 6546 : 2 ^ 6556 .. 6568 : 2 ^ 13130 .. 13156 : 2 ^ 13166 .. 13172 : 2 ^ 13178 .. 13198 : 2 ^ 13208 .. 13226 : 2 ^ 13236 .. 13254 : 2 ^ 14294 .. 14374 : 2 ^ 14380 .. 14408 : 2 ^ 16054 .. 16062 : 4 ^ 16064 ^ 16066 ^ 16082 .. 16110 : 2 ^ 16118 .. 16132 : 2 ^ 22438 .. 22540 : 2 ^ 22544 .. 22550 : 2 ^ 22554 .. 22566 : 2 ^ 22572 ^ 22584 ^ 22586 .. 22598 : 2 ^ 26884 .. 26986 : 2 ^ 26994 .. 27002 : 2 ^ 27006 .. 27084 : 2 ^ 29898 .. 29916 : 2 ^ 29928 .. 29974 : 2 ^ 29978 ^ 29998 ^ 30000 ^ 30004 ^ 30006 ^ 30028 ^ 30036 ^ 30038 ^ 30050 ^ 30074 ^ 30076 .. 30080 : 2 ^ 30084 ^ 30086 ^ 30090 .. 30122 : 2 ^ 30128 .. 30138 : 2 ^ 30142 .. 30148 : 2 ^ 30152 .. 30174 : 2 ^ 30178 ^ 30184 ^ 30186 .. 30192 : 2 ^ 30204 ^ 32258 ^ 32260 .. 32360 : 2 ^ 32364 .. 32416 : 2 ^ 32468 .. 32506 : 2 ^ 34870 .. 34916 : 2 ^ 34920 .. 34948 : 2 ^ 34954 ^ 36600 ^ 36604 ^ 36608 ^ 36610 .. 36628 : 2 ^ 36632 .. 36720 : 2 ^ 36724 .. 36838 : 2 ^ 36874 .. 36880 : 2 ^ 37520 .. 37574 : 2 ^ 37610 .. 37628 : 2 ^ 39312 .. 39368 : 2 ^ 44008 .. 44036 : 2 ^ 44044 .. 44060 : 2 ^ 44064 ^ 44066 ^ 44074 .. 44106 : 2 ^ 44110 .. 44166 : 2 ^ 44176 .. 44208 : 2 HSC-Z SSP_WIDE 173210.0 8794 .. 8806 : 2 ^ 8810 .. 8816 : 2 ^ 9596 .. 9626 : 2 ^ 9682 .. 9690 : 2 ^ 9694 .. 9790 : 2 ^ 13272 .. 13288 : 2 ^ 13304 ^ 13306 ^ 13340 ^ 13342 ^ 13356 .. 13452 : 2 ^ 15078 ^ 15080 ^ 15084 .. 15182 : 2 ^ 17654 .. 17686 : 2 ^ 17690 .. 17726 : 2 ^ 17730 .. 17754 : 2 ^ 17758 .. 17850 : 2 ^ 17854 .. 17894 : 2 ^ 23108 .. 23260 : 2 ^ 27088 .. 27160 : 2 ^ 28432 ^ 28438 ^ 28440 ^ 31262 ^ 31264 .. 31270 : 2 ^ 31274 .. 31282 : 4 ^ 31284 .. 31350 : 2 ^ 31370 .. 31448 : 2 ^ 31468 .. 31530 : 2 ^ 33762 .. 33776 : 2 ^ 33780 .. 33846 : 2 ^ 33850 .. 33868 : 2 ^ 33872 .. 33952 : 2 ^ 36040 .. 36058 : 2 ^ 36062 .. 36108 : 2 ^ 36324 .. 36340 : 2 ^ 36344 .. 36372 : 2 ^ 36376 ^ 36378 ^ 36382 .. 36398 : 2 ^ 36402 .. 36508 : 2 ^ 38876 .. 39028 : 2 ^ 44444 .. 44512 : 2 ^ 44516 .. 44536 : 2 ^ 44542 ^ 44560 ^ 44590 ^ 44614 ^ 44616 ^ 44620 ^ 44622 .. 44686 : 2 ^ 44690 .. 44772 : 2 ^ 47042 ^ 47056 ^ 47060 .. 47072 : 4 ^ 47074 ^ 47076 ^ 47080 ^ 47092 ^ 47094 ^ 47098 ^ 47112 ^ 47116 ^ 47118 .. 47126 : 2 ^ 47130 ^ 47134 ^ 47136 ^ 47140 ^ 47142 .. 47148 : 2 ^ 47154
            hchiang2 Hsin-Fang Chiang made changes -
            Labels datasets
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-9683 [ DM-9683 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues may you please lock down the folders (/datasets/hsc/raw/ssp_extra, /datasets/hsc/repo/SSP_WIDE/, /datasets/hsc/repo/SSP_UDEEP_COSMOS, /datasets/hsc/repo/registry.sqlite3)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues may you please lock down the folders (/datasets/hsc/raw/ssp_extra, /datasets/hsc/repo/SSP_WIDE/, /datasets/hsc/repo/SSP_UDEEP_COSMOS, /datasets/hsc/repo/registry.sqlite3)
            Hide
            gdaues Greg Daues added a comment -

            These directories and files are now locked down / immutable.

            Show
            gdaues Greg Daues added a comment - These directories and files are now locked down / immutable.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you Greg!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you Greg!
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Remote Link This issue links to "Page (Confluence)" [ 15000 ]
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Greg Daues, Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel