Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14325

deepDiff datasets not supported by HSC

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_subaru
    • Labels:
      None

      Description

      Trying to run imageDifference.py on an HSC dataset crashes with an error saying that the deepDiff_diaSrc dataset type does not have a template. Manual inspection of obs_subaru/policy/HscMapper.yaml confirms that there are no diaSrc or deepDiff datasets mentioned.

      Please add the appropriate datasets so that we can do image differencing on HSC data.

        Attachments

          Issue Links

            Activity

            No builds found.
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Link This issue relates to RFC-369 [ RFC-369 ]
            krzys Krzysztof Findeisen made changes -
            Risk Score 0
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-12315 [ DM-12315 ]
            krzys Krzysztof Findeisen made changes -
            Story Points 2 1
            Issue Type Story [ 10001 ] Bug [ 1 ]
            swinbank John Swinbank made changes -
            Epic Link DM-12728 [ 36327 ]
            swinbank John Swinbank made changes -
            Team Alert Production [ 10300 ]
            swinbank John Swinbank made changes -
            Epic Link DM-12728 [ 36327 ] DM-14447 [ 80385 ]
            Hide
            price Paul Price added a comment -

            Shouldn't these datasets be defined in obs_base rather than obs_subaru, so we don't have to manually add them to every obs package?

            Show
            price Paul Price added a comment - Shouldn't these datasets be defined in obs_base rather than obs_subaru, so we don't have to manually add them to every obs package?
            price Paul Price made changes -
            Component/s obs_base [ 10719 ]
            Component/s obs_subaru [ 10747 ]
            Hide
            krzys Krzysztof Findeisen added a comment -

            They already are defined in obs_base. But, as usual, the templates are instrument-specific.

            Show
            krzys Krzysztof Findeisen added a comment - They already are defined in obs_base . But, as usual, the templates are instrument-specific .
            Hide
            price Paul Price added a comment -

            Ah, the templates are instrument-specific because they depend on the visit. I was thinking they're like coadds, but they're not.

            Show
            price Paul Price added a comment - Ah, the templates are instrument-specific because they depend on the visit. I was thinking they're like coadds, but they're not.
            price Paul Price made changes -
            Component/s obs_subaru [ 10747 ]
            Component/s obs_base [ 10719 ]
            krzys Krzysztof Findeisen made changes -
            Assignee Krzysztof Findeisen [ krzys ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Sprint AP F18-2 [ 747 ]
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Paul Price, since you seem to be responsible for a lot of the other entries in the mapper file, could you take a look and see if my additions are reasonable? Thanks!

            Show
            krzys Krzysztof Findeisen added a comment - Hi Paul Price , since you seem to be responsible for a lot of the other entries in the mapper file, could you take a look and see if my additions are reasonable? Thanks!
            krzys Krzysztof Findeisen made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            price Paul Price added a comment -

            The existing templates (e.g., for deepCoadd_tempExp_diffsrc) suggest that the first element of the path should be deepCoadd-diff instead of deepDiff.
            Why isn't the tract part of the path? Aren't you using coadds as templates, in which case you need to disambiguate between different (but overlapping) templates?
            deepDiff_metadata needs a leading element in the filename, because we should never rely on merely the directory structure to identify a file. Along the same lines, deepDiff_matchedExp's leading filename element matches one that already exists.

            Show
            price Paul Price added a comment - The existing templates (e.g., for deepCoadd_tempExp_diffsrc ) suggest that the first element of the path should be deepCoadd-diff instead of deepDiff . Why isn't the tract part of the path? Aren't you using coadds as templates, in which case you need to disambiguate between different (but overlapping) templates? deepDiff_metadata needs a leading element in the filename, because we should never rely on merely the directory structure to identify a file. Along the same lines, deepDiff_matchedExp 's leading filename element matches one that already exists.
            price Paul Price made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            krzys Krzysztof Findeisen added a comment -

            These are (AFAIK) difference images, not coadds. So tract would not make sense (and is not used in e.g. the corresponding DECam templates). I don't know if that affects your answer for deepCoadd-diff, but I was under the impression that they were unrelated datasets.

            Can you explain what you mean by leading element? For example, there's:

            processCcd_metadata:
                template: '%(pointing)05d/%(filter)s/processCcd_metadata/%(visit)07d-%(ccd)03d.boost'
            isr_metadata:
                template: '%(pointing)05d/%(filter)s/isr_metadata/%(visit)07d-%(ccd)03d.boost'
            

            Show
            krzys Krzysztof Findeisen added a comment - These are (AFAIK) difference images, not coadds. So tract would not make sense (and is not used in e.g. the corresponding DECam templates). I don't know if that affects your answer for deepCoadd-diff , but I was under the impression that they were unrelated datasets. Can you explain what you mean by leading element? For example, there's: processCcd_metadata: template: '%(pointing)05d/%(filter)s/processCcd_metadata/%(visit)07d-%(ccd)03d.boost' isr_metadata: template: '%(pointing)05d/%(filter)s/isr_metadata/%(visit)07d-%(ccd)03d.boost'
            Hide
            price Paul Price added a comment -

            What is the template used for the difference images? If it's a coadd, then tract needs to be in the template for the difference image.

            By "leading element of the filename", I mean the part after the directory structure, but before the standard %(visit)07d-%(ccd)03d. The two examples you give above do not have any leading element, which I don't think is good (but they are existing and therefore harder to change than something being added now).

            Show
            price Paul Price added a comment - What is the template used for the difference images? If it's a coadd, then tract needs to be in the template for the difference image. By "leading element of the filename", I mean the part after the directory structure, but before the standard %(visit)07d-%(ccd)03d . The two examples you give above do not have any leading element, which I don't think is good (but they are existing and therefore harder to change than something being added now).
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            What is the template used for the difference images? If it's a coadd, then tract needs to be in the template for the difference image.

            At the very least, it can be either a coadd or a calexp. There may be other modes as well.

            Show
            krzys Krzysztof Findeisen added a comment - - edited What is the template used for the difference images? If it's a coadd, then tract needs to be in the template for the difference image. At the very least, it can be either a coadd or a calexp. There may be other modes as well.
            Hide
            price Paul Price added a comment -

            Then that would need to have multiple datasets, because those have different requirements on the filename template.

            Show
            price Paul Price added a comment - Then that would need to have multiple datasets, because those have different requirements on the filename template.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I am not aware of any requirements (or lack thereof) on the output of ImageDifferenceTask, but reworking it seems a bit out of scope for this ticket. What do you suggest for the time being?

            Show
            krzys Krzysztof Findeisen added a comment - I am not aware of any requirements (or lack thereof) on the output of ImageDifferenceTask , but reworking it seems a bit out of scope for this ticket. What do you suggest for the time being?
            krzys Krzysztof Findeisen made changes -
            Comment [ I am not aware of any requirements (or lack thereof) on the output of {{ImageDifferenceTask}}, but reworking it seems a bit out of scope for this ticket. What do you suggest for the time being? ]
            Hide
            price Paul Price added a comment -

            Well, I guess we just follow the example of obs_decam and proceed with the broken option of not including the tract, and hope this gets fixed.

            Show
            price Paul Price added a comment - Well, I guess we just follow the example of obs_decam and proceed with the broken option of not including the tract , and hope this gets fixed.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I've added or updated the leading elements as requested, but not changed the name of the top-level directory. (I still don't know what a deepCoadd_*diffsrc is, but it seems to be HSC-specific and therefore not related to a generic task like ImageDifferenceTask.)

            Thanks for the review!

            Show
            krzys Krzysztof Findeisen added a comment - I've added or updated the leading elements as requested, but not changed the name of the top-level directory. (I still don't know what a deepCoadd_*diffsrc is, but it seems to be HSC-specific and therefore not related to a generic task like ImageDifferenceTask .) Thanks for the review!
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            krzys Krzysztof Findeisen made changes -
            Risk Score 0 1
            wmwood-vasey Michael Wood-Vasey made changes -
            Link This issue relates to DM-20524 [ DM-20524 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Paul Price
              Watchers:
              Eric Bellm, Krzysztof Findeisen, Meredith Rawls, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.