XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
8
• Team:
Data Release Production
• Urgent?:
No

#### Description

After DM-34687 we need to write an afterburner task to take in the HIPS-compatible HPX maps and make a HIPS directory tree with the correct format following https://www.ivoa.net/documents/HiPS/

#### Activity

Hide
Tim Jenness added a comment -

Are we writing our own tooling for this or downloading all the images from the butler repo into a directory (with butler retrieve-artifacts) and then running hipsgen?

Show
Tim Jenness added a comment - Are we writing our own tooling for this or downloading all the images from the butler repo into a directory (with butler retrieve-artifacts) and then running hipsgen?
Hide
Eli Rykoff added a comment -

Writing our own tooling is the plan.

Show
Eli Rykoff added a comment - Writing our own tooling is the plan.
Hide
Tim Jenness added a comment -

and that includes making the lower order images?

Show
Tim Jenness added a comment - and that includes making the lower order images?
Hide
Eli Rykoff added a comment -

Ah, good question. That may be farmed out to hipsgen, but I will see if it's hard to do as part of the original plan for this ticket.

Show
Eli Rykoff added a comment - Ah, good question. That may be farmed out to hipsgen, but I will see if it's hard to do as part of the original plan for this ticket.
Hide
Tim Jenness added a comment -

I think if you are using hipsgen to generate the lower orders then it will want to put the files into the hips directory structure itself so I don't think anything is gained from us creating a hips directory tree with just the high res in it. Maybe I'm wrong.

Show
Tim Jenness added a comment - I think if you are using hipsgen to generate the lower orders then it will want to put the files into the hips directory structure itself so I don't think anything is gained from us creating a hips directory tree with just the high res in it. Maybe I'm wrong.
Hide
Eli Rykoff added a comment -

I believe I have figured out how to write out all the lower orders as well in an efficient manner.

Show
Eli Rykoff added a comment - I believe I have figured out how to write out all the lower orders as well in an efficient manner.
Hide
Simon Krughoff added a comment -

We will also need the tooling to create the ancillary files. In particular the properties file required for the standard. It would also be nice to have the other recommended, but not specifically required files as well.

Show
Simon Krughoff added a comment - We will also need the tooling to create the ancillary files. In particular the properties file required for the standard. It would also be nice to have the other recommended, but not specifically required files as well.
Hide
Gregory Dubois-Felsmann added a comment -

For a limited-area map like DC2, the MOC will be particularly useful, I think.

Show
Gregory Dubois-Felsmann added a comment - For a limited-area map like DC2, the MOC will be particularly useful, I think.
Hide
Eli Rykoff added a comment -

It's coming along, I have it building the full HiPS tree and I know how to generate the properties and MOC files which I will add next week. I am still waiting on DM-34772 to know what values should be put in the property files.

One thing I am completely unsure of is how to label different base directories based on band. As I understand it, each HiPS tree using FITS files is generated from a single band. And then Aladdin or suchlike will do color compositing of the single bands. But I don't know what names these should be given. Any guidance is appreciated!

Show
Eli Rykoff added a comment - It's coming along, I have it building the full HiPS tree and I know how to generate the properties and MOC files which I will add next week. I am still waiting on DM-34772 to know what values should be put in the property files. One thing I am completely unsure of is how to label different base directories based on band. As I understand it, each HiPS tree using FITS files is generated from a single band. And then Aladdin or suchlike will do color compositing of the single bands. But I don't know what names these should be given. Any guidance is appreciated!
Hide
Simon Krughoff added a comment -

As you say, each band will needs its own HiPS tree including properties, MOC, etc. files. There is a special file called hipslist (I think) that exposes all the HiPS resources available on a particular server. That file is essentially a concatenation of all properties files and needs to be served from a distinct location.

As far as the name of the root of the various HiPS trees, you can decide as long as it is somewhat informative and consistent. The important information should be discoverable from the properties file.

Show
Simon Krughoff added a comment - As you say, each band will needs its own HiPS tree including properties, MOC, etc. files. There is a special file called hipslist (I think) that exposes all the HiPS resources available on a particular server. That file is essentially a concatenation of all properties files and needs to be served from a distinct location. As far as the name of the root of the various HiPS trees, you can decide as long as it is somewhat informative and consistent. The important information should be discoverable from the properties file.
Hide
Simon Krughoff added a comment -

Note that if you do include a colorized version of the files, there are a few extra fields necessary in the properties file.

Show
Simon Krughoff added a comment - Note that if you do include a colorized version of the files, there are a few extra fields necessary in the properties file.
Hide
Tim Jenness added a comment -

I don't think the top-level names are special are they? Aren't they human readable? The metadata files say what they contain and the hipslist lists everything (Russ's tech note talks about the hipslist handling).

Show
Tim Jenness added a comment - I don't think the top-level names are special are they? Aren't they human readable? The metadata files say what they contain and the hipslist lists everything (Russ's tech note talks about the hipslist handling).
Hide
Simon Krughoff added a comment -

What Tim Jenness said is what I was trying to say, so we are in agreement

Show
Simon Krughoff added a comment - What Tim Jenness said is what I was trying to say, so we are in agreement
Hide
Eli Rykoff added a comment -

That makes sense, but I was wondering if there was some sort of convention.

Also, looking at the documentation for the properties file I don't know where the band should be recorded. https://www.ivoa.net/documents/HiPS/20170519/REC-HIPS-1.0-20170519.pdf. I guess maybe in the obs_title, but this is just free-form and not machine readable.

I'll put it in the name of the directory of course, and then it can be put into the hipslist. Is this something that my tool should be making as well?

Show
Eli Rykoff added a comment - That makes sense, but I was wondering if there was some sort of convention. Also, looking at the documentation for the properties file I don't know where the band should be recorded. https://www.ivoa.net/documents/HiPS/20170519/REC-HIPS-1.0-20170519.pdf . I guess maybe in the obs_title , but this is just free-form and not machine readable. I'll put it in the name of the directory of course, and then it can be put into the hipslist . Is this something that my tool should be making as well?
Hide
Tim Jenness added a comment -

You can look at lots of examples here: https://aladin.u-strasbg.fr/hips/list

Show
Tim Jenness added a comment - You can look at lots of examples here: https://aladin.u-strasbg.fr/hips/list
Hide
Eli Rykoff added a comment -

Gregory Dubois-Felsmann I have what should be functional HiPS trees for ci_hsc_gen3 on lsst-devl here: /project/erykoff/ci_hsc_hips_testing/ci_hsc_gen3_run1/hips. There's one directory

{r}

and one

{i}

, each with a properties file and a Moc.fits. I don't have a good value for creator_did, I'm waiting for DM-34772.

I'm going to be writing what tests I can, but confirming actual functionality of the HiPS tree is something that I was hoping you could check (or suggest somebody else who could check) and report back if there are any problems.

Thanks!

Show
Eli Rykoff added a comment - Gregory Dubois-Felsmann I have what should be functional HiPS trees for ci_hsc_gen3 on lsst-devl here: /project/erykoff/ci_hsc_hips_testing/ci_hsc_gen3_run1/hips . There's one directory {r} and one {i} , each with a properties file and a Moc.fits . I don't have a good value for creator_did , I'm waiting for DM-34772 . I'm going to be writing what tests I can, but confirming actual functionality of the HiPS tree is something that I was hoping you could check (or suggest somebody else who could check) and report back if there are any problems. Thanks!
Hide
Tim Jenness added a comment -

Eli Rykoff can you move the files to the public_html tree at NCSA for web access and see if Aladin will work with them?

Show
Tim Jenness added a comment - Eli Rykoff can you move the files to the public_html tree at NCSA for web access and see if Aladin will work with them?
Hide
Simon Krughoff added a comment -

Tim Jenness I've copied them here.

Show
Simon Krughoff added a comment - Tim Jenness I've copied them here .
Hide
Tim Jenness added a comment -

Thanks. I'm having trouble finding a macOS tool that will work unless I install Java and use Aladin Desktop (Aladin Lite does not work with FITS files).

Show
Tim Jenness added a comment - Thanks. I'm having trouble finding a macOS tool that will work unless I install Java and use Aladin Desktop (Aladin Lite does not work with FITS files).
Hide
Eli Rykoff added a comment -

I'm still working on cleaning up the metadata/linting. But I got the Aladin java thing working on my linux workstation (no luck on lsst-devl03, it froze and needed a kill -9).

Here is what ds9 shows for the patch in ci_hsc:

And here is what the initial view of java -jar AladinBeta.jar hips/i looks like:

And zooming out, it looks like the patch (with the asinh scaling, I didn't work hard on this):

Show
Eli Rykoff added a comment - I'm still working on cleaning up the metadata/linting. But I got the Aladin java thing working on my linux workstation (no luck on lsst-devl03, it froze and needed a kill -9). Here is what ds9 shows for the patch in ci_hsc : And here is what the initial view of java -jar AladinBeta.jar hips/i looks like: And zooming out, it looks like the patch (with the asinh scaling, I didn't work hard on this):
Hide
Eli Rykoff added a comment -
Show
Eli Rykoff added a comment - PRs are here: https://github.com/lsst/pipe_tasks/pull/706 https://github.com/lsst/ci_imsim/pull/28 https://github.com/lsst/ci_hsc_gen3/pull/83
Hide
Tim Jenness added a comment -

This looks good to me. I have some suggestions for removing code reuse and adding some comments. My main concern is that there are no tests to see if anything happened. The ci_hsc_gen3 tests don't see if some files turned up in relevant Norder directories and nothing checks that a property file contains the expected content.

Show
Tim Jenness added a comment - This looks good to me. I have some suggestions for removing code reuse and adding some comments. My main concern is that there are no tests to see if anything happened. The ci_hsc_gen3 tests don't see if some files turned up in relevant Norder directories and nothing checks that a property file contains the expected content.

#### People

Assignee:
Eli Rykoff
Reporter:
Eli Rykoff
Reviewers:
Tim Jenness
Watchers:
Colin Slater, Eli Rykoff, Gregory Dubois-Felsmann, Simon Krughoff, Tim Jenness