Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10436

Add HSC Bright Object Masks (version "Arcturus") to /datasets/hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Store a set of HSC Bright Object Masks of version "Arcturus" to the /datasets/hsc butler root repo. These masks should become available as Butler type brightObjectMask

      Also adding links for NB0816 and NB0921 filters for all tracts/patches.

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggered by RFC-337 [ RFC-337 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Labels datasets
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues, may I have a new folder /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ and permission to add new data into it?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues , may I have a new folder /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ and permission to add new data into it?
            Hide
            gdaues Greg Daues added a comment -

            Hsin-Fang Chiang. I have created such a directory; can you check if access is suitable ?

            Show
            gdaues Greg Daues added a comment - Hsin-Fang Chiang . I have created such a directory; can you check if access is suitable ?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues: Yes. Copying data over now. Thank you!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues : Yes. Copying data over now. Thank you!
            hchiang2 Hsin-Fang Chiang made changes -
            Remote Link This issue links to "Page (Confluence)" [ 15037 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues may you please make /datasets/hsc/repo/deepCoadd/ immutable?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues may you please make /datasets/hsc/repo/deepCoadd/ immutable?
            hchiang2 Hsin-Fang Chiang made changes -
            Reviewers Greg Daues [ gdaues ]
            Status To Do [ 10001 ] In Review [ 10004 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            brightObjectMask can now be accessed from the HSC butler repo. For example,

            butler = lsst.daf.persistence.Butler("/datasets/hsc/repo/")
            butler.get("brightObjectMask", tract=8284, patch='5,2', filter='HSC-I') 
            

            Note: not every patch/tract has brightObjectMask available.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - brightObjectMask can now be accessed from the HSC butler repo. For example, butler = lsst.daf.persistence.Butler( "/datasets/hsc/repo/" ) butler.get( "brightObjectMask" , tract= 8284 , patch= '5,2' , filter= 'HSC-I' ) Note: not every patch/tract has brightObjectMask available.
            Hide
            gdaues Greg Daues added a comment -

            The directory /datasets/hsc/repo/deepCoadd/ and the subdirectories/contents should now be immutable, group lsst_users, readable by all, i.e., ready for use.

            Show
            gdaues Greg Daues added a comment - The directory /datasets/hsc/repo/deepCoadd/ and the subdirectories/contents should now be immutable, group lsst_users , readable by all, i.e., ready for use.
            Hide
            gdaues Greg Daues added a comment -

            I attempted the lines above for v13_0 of the stack and obtain many WARN messages; it does not look like it really worked. Is that to be expected?
            Sample:

            ObjectMaskCatalog WARN: Unexpected line "box(37.039951,-6.990420,0.034446d,0.003445d,0) # ID: 5178395927953628288, mag: 11.389953" at /datasets/hsc/repo/deepCoadd/BrightObjectMasks/8284/BrightObjectMask-8284-5,2-HSC-I.reg:135
            

            Show
            gdaues Greg Daues added a comment - I attempted the lines above for v13_0 of the stack and obtain many WARN messages; it does not look like it really worked. Is that to be expected? Sample: ObjectMaskCatalog WARN: Unexpected line "box(37.039951,-6.990420,0.034446d,0.003445d,0) # ID: 5178395927953628288, mag: 11.389953" at /datasets/hsc/repo/deepCoadd/BrightObjectMasks/8284/BrightObjectMask-8284-5,2-HSC-I.reg:135
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I see those WARN messages using the v13_0 stack to retrieve them too, but a w_2017_17 stack or newer can read it fine. I guess it's because of this commit from DM-10308. Older stacks were not compatible with these object masks.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I see those WARN messages using the v13_0 stack to retrieve them too, but a w_2017_17 stack or newer can read it fine. I guess it's because of this commit from DM-10308 . Older stacks were not compatible with these object masks.
            Hide
            gdaues Greg Daues added a comment -

            In my tests I had trouble with the v13_0 stack, and also with a build b2913 yesterday, but now I see indeed that it is working with the w_2017_17 stack, and today b2914 shows success. So looks good.

            Show
            gdaues Greg Daues added a comment - In my tests I had trouble with the v13_0 stack, and also with a build b2913 yesterday, but now I see indeed that it is working with the w_2017_17 stack, and today b2914 shows success. So looks good.
            gdaues Greg Daues made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues Sorry, I need the permission of /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ again. I will make many new links (all within the same folder), but won't add new files.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues Sorry, I need the permission of /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ again. I will make many new links (all within the same folder), but won't add new files.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            (I won't modify or delete any existing files/links either. So what's already immutable can stay immutable)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited (I won't modify or delete any existing files/links either. So what's already immutable can stay immutable)
            Hide
            gdaues Greg Daues added a comment -

            Hsin-Fang Chiang The directory /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ has been reopened.

            Show
            gdaues Greg Daues added a comment - Hsin-Fang Chiang The directory /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ has been reopened.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues, the folder can be locked down again. Thank you!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues , the folder can be locked down again. Thank you!
            hchiang2 Hsin-Fang Chiang made changes -
            Description Store a set of HSC Bright Object Masks of version "Arcturus" to the {{/datasets/hsc}} butler root repo. These masks should become available as Butler type {{brightObjectMask}} Store a set of HSC Bright Object Masks of version "Arcturus" to the {{/datasets/hsc}} butler root repo. These masks should become available as Butler type {{brightObjectMask}}

            Also adding links for {{NB0816}} and {{NB0921}} filters for all tracts/patches.
            Hide
            gdaues Greg Daues added a comment -

            Hsin-Fang Chiang The directory /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ and its contents are once again immutable.

            Show
            gdaues Greg Daues added a comment - Hsin-Fang Chiang The directory /datasets/hsc/repo/deepCoadd/BrightObjectMasks/ and its contents are once again immutable.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you!

            Done.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you! Done.
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Greg Daues
                Watchers:
                Greg Daues, Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel