Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10128

Obtain missing public HSC RC data and add them to /datasets/hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Some of the HSC RC dataset were not included in HSC-PDR1/DM-9683//datasets/hsc/raw/ssp_pdr1 although some of them are publicly available.

      Obtain them from NAOJ if possible, so to allow processing the exactly same RC dataset and facilitate QA.

        Attachments

          Issue Links

            Activity

            Hide
            gdaues Greg Daues added a comment -

            These should now have immutable flag turned off, so I think you can proceed.

            Show
            gdaues Greg Daues added a comment - These should now have immutable flag turned off, so I think you can proceed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            These 6944 files have been copied to /datasets/hsc/raw/ssp_extra, and then ingested into the butler repo using stack w_2017_15.

            $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_UDEEP_COSMOS' -s
            filter  field                expTime visit
            HSC-G   SSP_UDEEP_COSMOS      5460.0 1886^11688^11690..11712:2^29324^29326^29336^29340^29350^29352
            HSC-I   SSP_UDEEP_COSMOS      9960.0 1226..1232:2^1236..1248:2^19656..19662:2^19680..19684:2^19694..19698:2^19708..19712:2^30482..30504:2
            HSC-R   SSP_UDEEP_COSMOS      5490.0 1200..1222:2^23690..23694:2^23704^23706^23716^23718
            HSC-Y   SSP_UDEEP_COSMOS     23550.0 274..302:2^306..334:2^340..370:2^1856..1862:2^1868..1882:2^11716..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2
            HSC-Z   SSP_UDEEP_COSMOS     16410.0 1164..1194:2^17898..17908:2^17926..17934:2^17944..17952:2^17962^28354^28356..28402:2
            NB0921  SSP_UDEEP_COSMOS     25230.0 23036..23056:2^23594..23606:2^24298..24310:2^25810..25816:2
             
            $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_WIDE' -s
            filter  field                expTime visit
            HSC-G   SSP_WIDE             115530.0 6292..6350:2^9838..9872:2^9880..9908:2^9912..9918:2^11562..11624:2^11630..11640:2^11646..11656:2^11664..11684:2^15186..15196:2^15210..15214:2^15222..15226:2^15234..15238:2^15264..15268:2^15272..15278:2^15282..15318:2^15324..15400:2^25824..25882:2^25886..26014:2^26018..26110:2^29362..29480:2^29484..29502:2^29508^29510^29514^29518^29520..29526:2^29588..29614:2^29618^29620^29624..29630:2^34176..34204:2^34260..34328:2^34332..34344:2^34348..34356:4^34358..34486:2^38206..38266:2^38274..38300:2^38304..38334:2^38338..38354:2^38358^38362^38364^38368^38370..38406:2^38414..38440:2^38444..38470:2^42142..42176:2^42180..42190:2^42194..42342:2^42382..42560:2^46872^46876^46878..46948:2
            HSC-I   SSP_WIDE             161100.0 1252..1318:2^1322..1346:2^7212..7238:2^7286..7314:2^7318..7322:2^7338..7360:2^7364..7374:2^7378..7404:2^7408..7416:2^7424^14078^14080..14104:2^14116..14150:2^14160..14182:2^14192..14214:2^16220..16272:2^16284^16292^16294..16380:2^19378^19382^19384..19432:2^19444..19486:2^19490..19622:2^19628..19654:2^24360..24426:2^24430..24434:2^24438..24494:2^24504..24556:2^24938..25080:2^25084^25088^25090..25150:2^30508..30528:2^30534..30550:2^30554..30594:2^30672..30726:2^35832..35918:2^35922..35954:2^35958..35974:2^36112..36122:2^36126..36266:2^37976..38072:2^45848..45854:2^45858..45876:2^45890..45916:2^45920..45932:2^45936..46008:2^46016..46024:2
            HSC-R   SSP_WIDE             102300.0 7110..7154:2^7158^7164^7166^7168^7204..7208:2^11382..11400:2^11404..11450:2^11458..11480:2^11486..11508:2^11514..11556:2^14836..14848:2^14868..14912:2^14920..14928:2^14932..14990:2^16154..16168:2^16176..16182:2^16190..16196:2^16204..16210:2^23610..23618:2^23622..23670:2^23728..23930:2^28902..28906:2^28910..28930:2^28940..28996:2^29058^29060^29064..29108:2^34030..34098:2^34582..34630:2^34634..34664:2^34668..34774:2^34808..34816:2^40752..40844:2^40848..40972:2^41010..41178:2^45158..45206:2^45210..45230:2^45234^45238^45240^45242^45246..45252:2^45256..45284:2^45288..45320:2^45324^45340^45346..45374:2
            HSC-Y   SSP_WIDE             161800.0 374..432:2^436..480:2^6370..6410:2^6460..6500:2^6512^6522^6524..6546:2^6556..6568:2^13130..13156:2^13166..13172:2^13178..13198:2^13208..13226:2^13236..13254:2^14294..14374:2^14380..14408:2^16054..16062:4^16064^16066^16082..16110:2^16118..16132:2^22438..22540:2^22544..22550:2^22554..22566:2^22572^22584^22586..22598:2^26884..26986:2^26994..27002:2^27006..27084:2^29898..29916:2^29928..29974:2^29978^29998^30000^30004^30006^30028^30036^30038^30050^30074^30076..30080:2^30084^30086^30090..30122:2^30128..30138:2^30142..30148:2^30152..30174:2^30178^30184^30186..30192:2^30204^32258^32260..32360:2^32364..32416:2^32468..32506:2^34870..34916:2^34920..34948:2^34954^36600^36604^36608^36610..36628:2^36632..36720:2^36724..36838:2^36874..36880:2^37520..37574:2^37610..37628:2^39312..39368:2^44008..44036:2^44044..44060:2^44064^44066^44074..44106:2^44110..44166:2^44176..44208:2
            HSC-Z   SSP_WIDE             173210.0 8794..8806:2^8810..8816:2^9596..9626:2^9682..9690:2^9694..9790:2^13272..13288:2^13304^13306^13340^13342^13356..13452:2^15078^15080^15084..15182:2^17654..17686:2^17690..17726:2^17730..17754:2^17758..17850:2^17854..17894:2^23108..23260:2^27088..27160:2^28432^28438^28440^31262^31264..31270:2^31274..31282:4^31284..31350:2^31370..31448:2^31468..31530:2^33762..33776:2^33780..33846:2^33850..33868:2^33872..33952:2^36040..36058:2^36062..36108:2^36324..36340:2^36344..36372:2^36376^36378^36382..36398:2^36402..36508:2^38876..39028:2^44444..44512:2^44516..44536:2^44542^44560^44590^44614^44616^44620^44622..44686:2^44690..44772:2^47042^47056^47060..47072:4^47074^47076^47080^47092^47094^47098^47112^47116^47118..47126:2^47130^47134^47136^47140^47142..47148:2^47154
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - These 6944 files have been copied to /datasets/hsc/raw/ssp_extra , and then ingested into the butler repo using stack w_2017_15 . $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_UDEEP_COSMOS' -s filter field expTime visit HSC-G SSP_UDEEP_COSMOS 5460.0 1886 ^ 11688 ^ 11690 .. 11712 : 2 ^ 29324 ^ 29326 ^ 29336 ^ 29340 ^ 29350 ^ 29352 HSC-I SSP_UDEEP_COSMOS 9960.0 1226 .. 1232 : 2 ^ 1236 .. 1248 : 2 ^ 19656 .. 19662 : 2 ^ 19680 .. 19684 : 2 ^ 19694 .. 19698 : 2 ^ 19708 .. 19712 : 2 ^ 30482 .. 30504 : 2 HSC-R SSP_UDEEP_COSMOS 5490.0 1200 .. 1222 : 2 ^ 23690 .. 23694 : 2 ^ 23704 ^ 23706 ^ 23716 ^ 23718 HSC-Y SSP_UDEEP_COSMOS 23550.0 274 .. 302 : 2 ^ 306 .. 334 : 2 ^ 340 .. 370 : 2 ^ 1856 .. 1862 : 2 ^ 1868 .. 1882 : 2 ^ 11716 .. 11742 : 2 ^ 22602 .. 22608 : 2 ^ 22626 .. 22632 : 2 ^ 22642 .. 22648 : 2 ^ 22658 .. 22664 : 2 HSC-Z SSP_UDEEP_COSMOS 16410.0 1164 .. 1194 : 2 ^ 17898 .. 17908 : 2 ^ 17926 .. 17934 : 2 ^ 17944 .. 17952 : 2 ^ 17962 ^ 28354 ^ 28356 .. 28402 : 2 NB0921 SSP_UDEEP_COSMOS 25230.0 23036 .. 23056 : 2 ^ 23594 .. 23606 : 2 ^ 24298 .. 24310 : 2 ^ 25810 .. 25816 : 2   $ registryInfo.py /datasets/hsc/repo/ --field 'SSP_WIDE' -s filter field expTime visit HSC-G SSP_WIDE 115530.0 6292 .. 6350 : 2 ^ 9838 .. 9872 : 2 ^ 9880 .. 9908 : 2 ^ 9912 .. 9918 : 2 ^ 11562 .. 11624 : 2 ^ 11630 .. 11640 : 2 ^ 11646 .. 11656 : 2 ^ 11664 .. 11684 : 2 ^ 15186 .. 15196 : 2 ^ 15210 .. 15214 : 2 ^ 15222 .. 15226 : 2 ^ 15234 .. 15238 : 2 ^ 15264 .. 15268 : 2 ^ 15272 .. 15278 : 2 ^ 15282 .. 15318 : 2 ^ 15324 .. 15400 : 2 ^ 25824 .. 25882 : 2 ^ 25886 .. 26014 : 2 ^ 26018 .. 26110 : 2 ^ 29362 .. 29480 : 2 ^ 29484 .. 29502 : 2 ^ 29508 ^ 29510 ^ 29514 ^ 29518 ^ 29520 .. 29526 : 2 ^ 29588 .. 29614 : 2 ^ 29618 ^ 29620 ^ 29624 .. 29630 : 2 ^ 34176 .. 34204 : 2 ^ 34260 .. 34328 : 2 ^ 34332 .. 34344 : 2 ^ 34348 .. 34356 : 4 ^ 34358 .. 34486 : 2 ^ 38206 .. 38266 : 2 ^ 38274 .. 38300 : 2 ^ 38304 .. 38334 : 2 ^ 38338 .. 38354 : 2 ^ 38358 ^ 38362 ^ 38364 ^ 38368 ^ 38370 .. 38406 : 2 ^ 38414 .. 38440 : 2 ^ 38444 .. 38470 : 2 ^ 42142 .. 42176 : 2 ^ 42180 .. 42190 : 2 ^ 42194 .. 42342 : 2 ^ 42382 .. 42560 : 2 ^ 46872 ^ 46876 ^ 46878 .. 46948 : 2 HSC-I SSP_WIDE 161100.0 1252 .. 1318 : 2 ^ 1322 .. 1346 : 2 ^ 7212 .. 7238 : 2 ^ 7286 .. 7314 : 2 ^ 7318 .. 7322 : 2 ^ 7338 .. 7360 : 2 ^ 7364 .. 7374 : 2 ^ 7378 .. 7404 : 2 ^ 7408 .. 7416 : 2 ^ 7424 ^ 14078 ^ 14080 .. 14104 : 2 ^ 14116 .. 14150 : 2 ^ 14160 .. 14182 : 2 ^ 14192 .. 14214 : 2 ^ 16220 .. 16272 : 2 ^ 16284 ^ 16292 ^ 16294 .. 16380 : 2 ^ 19378 ^ 19382 ^ 19384 .. 19432 : 2 ^ 19444 .. 19486 : 2 ^ 19490 .. 19622 : 2 ^ 19628 .. 19654 : 2 ^ 24360 .. 24426 : 2 ^ 24430 .. 24434 : 2 ^ 24438 .. 24494 : 2 ^ 24504 .. 24556 : 2 ^ 24938 .. 25080 : 2 ^ 25084 ^ 25088 ^ 25090 .. 25150 : 2 ^ 30508 .. 30528 : 2 ^ 30534 .. 30550 : 2 ^ 30554 .. 30594 : 2 ^ 30672 .. 30726 : 2 ^ 35832 .. 35918 : 2 ^ 35922 .. 35954 : 2 ^ 35958 .. 35974 : 2 ^ 36112 .. 36122 : 2 ^ 36126 .. 36266 : 2 ^ 37976 .. 38072 : 2 ^ 45848 .. 45854 : 2 ^ 45858 .. 45876 : 2 ^ 45890 .. 45916 : 2 ^ 45920 .. 45932 : 2 ^ 45936 .. 46008 : 2 ^ 46016 .. 46024 : 2 HSC-R SSP_WIDE 102300.0 7110 .. 7154 : 2 ^ 7158 ^ 7164 ^ 7166 ^ 7168 ^ 7204 .. 7208 : 2 ^ 11382 .. 11400 : 2 ^ 11404 .. 11450 : 2 ^ 11458 .. 11480 : 2 ^ 11486 .. 11508 : 2 ^ 11514 .. 11556 : 2 ^ 14836 .. 14848 : 2 ^ 14868 .. 14912 : 2 ^ 14920 .. 14928 : 2 ^ 14932 .. 14990 : 2 ^ 16154 .. 16168 : 2 ^ 16176 .. 16182 : 2 ^ 16190 .. 16196 : 2 ^ 16204 .. 16210 : 2 ^ 23610 .. 23618 : 2 ^ 23622 .. 23670 : 2 ^ 23728 .. 23930 : 2 ^ 28902 .. 28906 : 2 ^ 28910 .. 28930 : 2 ^ 28940 .. 28996 : 2 ^ 29058 ^ 29060 ^ 29064 .. 29108 : 2 ^ 34030 .. 34098 : 2 ^ 34582 .. 34630 : 2 ^ 34634 .. 34664 : 2 ^ 34668 .. 34774 : 2 ^ 34808 .. 34816 : 2 ^ 40752 .. 40844 : 2 ^ 40848 .. 40972 : 2 ^ 41010 .. 41178 : 2 ^ 45158 .. 45206 : 2 ^ 45210 .. 45230 : 2 ^ 45234 ^ 45238 ^ 45240 ^ 45242 ^ 45246 .. 45252 : 2 ^ 45256 .. 45284 : 2 ^ 45288 .. 45320 : 2 ^ 45324 ^ 45340 ^ 45346 .. 45374 : 2 HSC-Y SSP_WIDE 161800.0 374 .. 432 : 2 ^ 436 .. 480 : 2 ^ 6370 .. 6410 : 2 ^ 6460 .. 6500 : 2 ^ 6512 ^ 6522 ^ 6524 .. 6546 : 2 ^ 6556 .. 6568 : 2 ^ 13130 .. 13156 : 2 ^ 13166 .. 13172 : 2 ^ 13178 .. 13198 : 2 ^ 13208 .. 13226 : 2 ^ 13236 .. 13254 : 2 ^ 14294 .. 14374 : 2 ^ 14380 .. 14408 : 2 ^ 16054 .. 16062 : 4 ^ 16064 ^ 16066 ^ 16082 .. 16110 : 2 ^ 16118 .. 16132 : 2 ^ 22438 .. 22540 : 2 ^ 22544 .. 22550 : 2 ^ 22554 .. 22566 : 2 ^ 22572 ^ 22584 ^ 22586 .. 22598 : 2 ^ 26884 .. 26986 : 2 ^ 26994 .. 27002 : 2 ^ 27006 .. 27084 : 2 ^ 29898 .. 29916 : 2 ^ 29928 .. 29974 : 2 ^ 29978 ^ 29998 ^ 30000 ^ 30004 ^ 30006 ^ 30028 ^ 30036 ^ 30038 ^ 30050 ^ 30074 ^ 30076 .. 30080 : 2 ^ 30084 ^ 30086 ^ 30090 .. 30122 : 2 ^ 30128 .. 30138 : 2 ^ 30142 .. 30148 : 2 ^ 30152 .. 30174 : 2 ^ 30178 ^ 30184 ^ 30186 .. 30192 : 2 ^ 30204 ^ 32258 ^ 32260 .. 32360 : 2 ^ 32364 .. 32416 : 2 ^ 32468 .. 32506 : 2 ^ 34870 .. 34916 : 2 ^ 34920 .. 34948 : 2 ^ 34954 ^ 36600 ^ 36604 ^ 36608 ^ 36610 .. 36628 : 2 ^ 36632 .. 36720 : 2 ^ 36724 .. 36838 : 2 ^ 36874 .. 36880 : 2 ^ 37520 .. 37574 : 2 ^ 37610 .. 37628 : 2 ^ 39312 .. 39368 : 2 ^ 44008 .. 44036 : 2 ^ 44044 .. 44060 : 2 ^ 44064 ^ 44066 ^ 44074 .. 44106 : 2 ^ 44110 .. 44166 : 2 ^ 44176 .. 44208 : 2 HSC-Z SSP_WIDE 173210.0 8794 .. 8806 : 2 ^ 8810 .. 8816 : 2 ^ 9596 .. 9626 : 2 ^ 9682 .. 9690 : 2 ^ 9694 .. 9790 : 2 ^ 13272 .. 13288 : 2 ^ 13304 ^ 13306 ^ 13340 ^ 13342 ^ 13356 .. 13452 : 2 ^ 15078 ^ 15080 ^ 15084 .. 15182 : 2 ^ 17654 .. 17686 : 2 ^ 17690 .. 17726 : 2 ^ 17730 .. 17754 : 2 ^ 17758 .. 17850 : 2 ^ 17854 .. 17894 : 2 ^ 23108 .. 23260 : 2 ^ 27088 .. 27160 : 2 ^ 28432 ^ 28438 ^ 28440 ^ 31262 ^ 31264 .. 31270 : 2 ^ 31274 .. 31282 : 4 ^ 31284 .. 31350 : 2 ^ 31370 .. 31448 : 2 ^ 31468 .. 31530 : 2 ^ 33762 .. 33776 : 2 ^ 33780 .. 33846 : 2 ^ 33850 .. 33868 : 2 ^ 33872 .. 33952 : 2 ^ 36040 .. 36058 : 2 ^ 36062 .. 36108 : 2 ^ 36324 .. 36340 : 2 ^ 36344 .. 36372 : 2 ^ 36376 ^ 36378 ^ 36382 .. 36398 : 2 ^ 36402 .. 36508 : 2 ^ 38876 .. 39028 : 2 ^ 44444 .. 44512 : 2 ^ 44516 .. 44536 : 2 ^ 44542 ^ 44560 ^ 44590 ^ 44614 ^ 44616 ^ 44620 ^ 44622 .. 44686 : 2 ^ 44690 .. 44772 : 2 ^ 47042 ^ 47056 ^ 47060 .. 47072 : 4 ^ 47074 ^ 47076 ^ 47080 ^ 47092 ^ 47094 ^ 47098 ^ 47112 ^ 47116 ^ 47118 .. 47126 : 2 ^ 47130 ^ 47134 ^ 47136 ^ 47140 ^ 47142 .. 47148 : 2 ^ 47154
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues may you please lock down the folders (/datasets/hsc/raw/ssp_extra, /datasets/hsc/repo/SSP_WIDE/, /datasets/hsc/repo/SSP_UDEEP_COSMOS, /datasets/hsc/repo/registry.sqlite3)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues may you please lock down the folders (/datasets/hsc/raw/ssp_extra, /datasets/hsc/repo/SSP_WIDE/, /datasets/hsc/repo/SSP_UDEEP_COSMOS, /datasets/hsc/repo/registry.sqlite3)
            Hide
            gdaues Greg Daues added a comment -

            These directories and files are now locked down / immutable.

            Show
            gdaues Greg Daues added a comment - These directories and files are now locked down / immutable.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you Greg!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you Greg!

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Greg Daues, Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel