Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34689

Write afterburner task to create HIPS directory tree with coadds

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      After DM-34687 we need to write an afterburner task to take in the HIPS-compatible HPX maps and make a HIPS directory tree with the correct format following https://www.ivoa.net/documents/HiPS/

        Attachments

          Issue Links

            Activity

            No builds found.
            erykoff Eli Rykoff created issue -
            gpdf Gregory Dubois-Felsmann made changes -
            Field Original Value New Value
            Link This issue FF-depends on DM-34772 [ DM-34772 ]
            Hide
            tjenness Tim Jenness added a comment -

            Are we writing our own tooling for this or downloading all the images from the butler repo into a directory (with `butler retrieve-artifacts`) and then running hipsgen?

            Show
            tjenness Tim Jenness added a comment - Are we writing our own tooling for this or downloading all the images from the butler repo into a directory (with `butler retrieve-artifacts`) and then running hipsgen?
            Hide
            erykoff Eli Rykoff added a comment -

            Writing our own tooling is the plan.

            Show
            erykoff Eli Rykoff added a comment - Writing our own tooling is the plan.
            Hide
            tjenness Tim Jenness added a comment -

            and that includes making the lower order images?

            Show
            tjenness Tim Jenness added a comment - and that includes making the lower order images?
            Hide
            erykoff Eli Rykoff added a comment -

            Ah, good question. That may be farmed out to hipsgen, but I will see if it's hard to do as part of the original plan for this ticket.

            Show
            erykoff Eli Rykoff added a comment - Ah, good question. That may be farmed out to hipsgen, but I will see if it's hard to do as part of the original plan for this ticket.
            erykoff Eli Rykoff made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            tjenness Tim Jenness added a comment -

            I think if you are using hipsgen to generate the lower orders then it will want to put the files into the hips directory structure itself so I don't think anything is gained from us creating a hips directory tree with just the high res in it. Maybe I'm wrong.

            Show
            tjenness Tim Jenness added a comment - I think if you are using hipsgen to generate the lower orders then it will want to put the files into the hips directory structure itself so I don't think anything is gained from us creating a hips directory tree with just the high res in it. Maybe I'm wrong.
            Hide
            erykoff Eli Rykoff added a comment -

            I believe I have figured out how to write out all the lower orders as well in an efficient manner.

            Show
            erykoff Eli Rykoff added a comment - I believe I have figured out how to write out all the lower orders as well in an efficient manner.
            Hide
            krughoff Simon Krughoff added a comment -

            We will also need the tooling to create the ancillary files. In particular the properties file required for the standard. It would also be nice to have the other recommended, but not specifically required files as well.

            Show
            krughoff Simon Krughoff added a comment - We will also need the tooling to create the ancillary files. In particular the properties file required for the standard. It would also be nice to have the other recommended, but not specifically required files as well.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            For a limited-area map like DC2, the MOC will be particularly useful, I think.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - For a limited-area map like DC2, the MOC will be particularly useful, I think.
            Hide
            erykoff Eli Rykoff added a comment -

            It's coming along, I have it building the full HiPS tree and I know how to generate the properties and MOC files which I will add next week. I am still waiting on DM-34772 to know what values should be put in the property files.

            One thing I am completely unsure of is how to label different base directories based on band. As I understand it, each HiPS tree using FITS files is generated from a single band. And then Aladdin or suchlike will do color compositing of the single bands. But I don't know what names these should be given. Any guidance is appreciated!

            Show
            erykoff Eli Rykoff added a comment - It's coming along, I have it building the full HiPS tree and I know how to generate the properties and MOC files which I will add next week. I am still waiting on DM-34772 to know what values should be put in the property files. One thing I am completely unsure of is how to label different base directories based on band. As I understand it, each HiPS tree using FITS files is generated from a single band. And then Aladdin or suchlike will do color compositing of the single bands. But I don't know what names these should be given. Any guidance is appreciated!
            Hide
            krughoff Simon Krughoff added a comment -

            As you say, each band will needs its own HiPS tree including properties, MOC, etc. files. There is a special file called hipslist (I think) that exposes all the HiPS resources available on a particular server. That file is essentially a concatenation of all properties files and needs to be served from a distinct location.

            As far as the name of the root of the various HiPS trees, you can decide as long as it is somewhat informative and consistent. The important information should be discoverable from the properties file.

            Show
            krughoff Simon Krughoff added a comment - As you say, each band will needs its own HiPS tree including properties, MOC, etc. files. There is a special file called hipslist (I think) that exposes all the HiPS resources available on a particular server. That file is essentially a concatenation of all properties files and needs to be served from a distinct location. As far as the name of the root of the various HiPS trees, you can decide as long as it is somewhat informative and consistent. The important information should be discoverable from the properties file.
            Hide
            krughoff Simon Krughoff added a comment -

            Note that if you do include a colorized version of the files, there are a few extra fields necessary in the properties file.

            Show
            krughoff Simon Krughoff added a comment - Note that if you do include a colorized version of the files, there are a few extra fields necessary in the properties file.
            Hide
            tjenness Tim Jenness added a comment -

            I don't think the top-level names are special are they? Aren't they human readable? The metadata files say what they contain and the hipslist lists everything (Russ's tech note talks about the hipslist handling).

            Show
            tjenness Tim Jenness added a comment - I don't think the top-level names are special are they? Aren't they human readable? The metadata files say what they contain and the hipslist lists everything (Russ's tech note talks about the hipslist handling).
            Hide
            krughoff Simon Krughoff added a comment -

            What Tim Jenness said is what I was trying to say, so we are in agreement

            Show
            krughoff Simon Krughoff added a comment - What Tim Jenness said is what I was trying to say, so we are in agreement
            Hide
            erykoff Eli Rykoff added a comment -

            That makes sense, but I was wondering if there was some sort of convention.

            Also, looking at the documentation for the properties file I don't know where the band should be recorded. https://www.ivoa.net/documents/HiPS/20170519/REC-HIPS-1.0-20170519.pdf. I guess maybe in the obs_title, but this is just free-form and not machine readable.

            I'll put it in the name of the directory of course, and then it can be put into the hipslist. Is this something that my tool should be making as well?

            Show
            erykoff Eli Rykoff added a comment - That makes sense, but I was wondering if there was some sort of convention. Also, looking at the documentation for the properties file I don't know where the band should be recorded. https://www.ivoa.net/documents/HiPS/20170519/REC-HIPS-1.0-20170519.pdf . I guess maybe in the obs_title , but this is just free-form and not machine readable. I'll put it in the name of the directory of course, and then it can be put into the hipslist . Is this something that my tool should be making as well?
            Hide
            tjenness Tim Jenness added a comment -

            You can look at lots of examples here: https://aladin.u-strasbg.fr/hips/list

            Show
            tjenness Tim Jenness added a comment - You can look at lots of examples here: https://aladin.u-strasbg.fr/hips/list
            Hide
            erykoff Eli Rykoff added a comment -

            Gregory Dubois-Felsmann I have what should be functional HiPS trees for ci_hsc_gen3 on lsst-devl here: /project/erykoff/ci_hsc_hips_testing/ci_hsc_gen3_run1/hips. There's one directory

            {r}

            and one

            {i}

            , each with a properties file and a Moc.fits. I don't have a good value for creator_did, I'm waiting for DM-34772.

            I'm going to be writing what tests I can, but confirming actual functionality of the HiPS tree is something that I was hoping you could check (or suggest somebody else who could check) and report back if there are any problems.

            Thanks!

            Show
            erykoff Eli Rykoff added a comment - Gregory Dubois-Felsmann I have what should be functional HiPS trees for ci_hsc_gen3 on lsst-devl here: /project/erykoff/ci_hsc_hips_testing/ci_hsc_gen3_run1/hips . There's one directory {r} and one {i} , each with a properties file and a Moc.fits . I don't have a good value for creator_did , I'm waiting for DM-34772 . I'm going to be writing what tests I can, but confirming actual functionality of the HiPS tree is something that I was hoping you could check (or suggest somebody else who could check) and report back if there are any problems. Thanks!
            Hide
            tjenness Tim Jenness added a comment -

            Eli Rykoff can you move the files to the public_html tree at NCSA for web access and see if Aladin will work with them?

            Show
            tjenness Tim Jenness added a comment - Eli Rykoff can you move the files to the public_html tree at NCSA for web access and see if Aladin will work with them?
            Hide
            krughoff Simon Krughoff added a comment -

            Tim Jenness I've copied them here.

            Show
            krughoff Simon Krughoff added a comment - Tim Jenness I've copied them here .
            Hide
            tjenness Tim Jenness added a comment -

            Thanks. I'm having trouble finding a macOS tool that will work unless I install Java and use Aladin Desktop (Aladin Lite does not work with FITS files).

            Show
            tjenness Tim Jenness added a comment - Thanks. I'm having trouble finding a macOS tool that will work unless I install Java and use Aladin Desktop (Aladin Lite does not work with FITS files).
            erykoff Eli Rykoff made changes -
            erykoff Eli Rykoff made changes -
            erykoff Eli Rykoff made changes -
            Hide
            erykoff Eli Rykoff added a comment -

            I'm still working on cleaning up the metadata/linting. But I got the Aladin java thing working on my linux workstation (no luck on lsst-devl03, it froze and needed a kill -9).

            Here is what ds9 shows for the patch in ci_hsc:

            And here is what the initial view of java -jar AladinBeta.jar hips/i looks like:

            And zooming out, it looks like the patch (with the asinh scaling, I didn't work hard on this):

            Show
            erykoff Eli Rykoff added a comment - I'm still working on cleaning up the metadata/linting. But I got the Aladin java thing working on my linux workstation (no luck on lsst-devl03, it froze and needed a kill -9). Here is what ds9 shows for the patch in ci_hsc : And here is what the initial view of java -jar AladinBeta.jar hips/i looks like: And zooming out, it looks like the patch (with the asinh scaling, I didn't work hard on this):
            Show
            erykoff Eli Rykoff added a comment - PRs are here: https://github.com/lsst/pipe_tasks/pull/706 https://github.com/lsst/ci_imsim/pull/28 https://github.com/lsst/ci_hsc_gen3/pull/83
            erykoff Eli Rykoff made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-34494 [ 1449397 ] DM-35761 [ 1880231 ]
            Hide
            tjenness Tim Jenness added a comment -

            This looks good to me. I have some suggestions for removing code reuse and adding some comments. My main concern is that there are no tests to see if anything happened. The ci_hsc_gen3 tests don't see if some files turned up in relevant Norder directories and nothing checks that a property file contains the expected content.

            Show
            tjenness Tim Jenness added a comment - This looks good to me. I have some suggestions for removing code reuse and adding some comments. My main concern is that there are no tests to see if anything happened. The ci_hsc_gen3 tests don't see if some files turned up in relevant Norder directories and nothing checks that a property file contains the expected content.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            erykoff Eli Rykoff made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            erykoff Eli Rykoff made changes -
            Story Points 8

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Tim Jenness
              Watchers:
              Colin Slater, Eli Rykoff, Gregory Dubois-Felsmann, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.