Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17018

deprecate and disable ScaleZeroPointTask

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:

      Description

      Implementation ticket for RFC-545.

      Once PhotoCalib replaces Calib (DM-10153), we will set a config parameter in AssembleCoaddTask to allow disabling ScaleZeroPointTask and mark it deprecated. We can then test whether we think we still need it by disabling it and processing some HSC data.

      Now that PhotoCalib is defined in nanojansky, the resulting warps will have "human understandable" calibrated fluxes, without having the mean calibration divided out.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Gabor Kovacs [X] and I pair coded on this today, so that he can get coadds that have nJy units to make difference imaging all have the same units throughout.

            Jenkins run, to see what it breaks: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30609/pipeline

            Show
            Parejkoj John Parejko added a comment - Gabor Kovacs [X] and I pair coded on this today, so that he can get coadds that have nJy units to make difference imaging all have the same units throughout. Jenkins run, to see what it breaks: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30609/pipeline
            Hide
            Parejkoj John Parejko added a comment - - edited

            Lauren MacArthur: can you please review this implementation of RFC-545? It's a relatively small change, but will allow you to still use ScaleZeroPointTask in HSC if you explicitly enable it.

            Jenkins passed with ci_hsc: are there other tests you think should be done?

            Show
            Parejkoj John Parejko added a comment - - edited Lauren MacArthur : can you please review this implementation of RFC-545 ? It's a relatively small change, but will allow you to still use ScaleZeroPointTask in HSC if you explicitly enable it. Jenkins passed with ci_hsc: are there other tests you think should be done?
            Hide
            Parejkoj John Parejko added a comment -

            Oh, and here's the PR since Jira's not seeing it: https://github.com/lsst/pipe_tasks/pull/321

            Show
            Parejkoj John Parejko added a comment - Oh, and here's the PR since Jira's not seeing it: https://github.com/lsst/pipe_tasks/pull/321
            Hide
            lauren Lauren MacArthur added a comment -

            Sorry for the delay on this.  It has taken me a while to digest the comments and resolution on the RFC. I have to admit that I'm not entirely convinced the conclusion was to deprecate ScaleZeroPointTask, but rather adding a (configurable) option to allow it to be disabled.  In particular, Yusra AlSayyad's last comment ends with a dangling sentence where there is a suggestion of re-working a "new" ScaleZeroPointTask. This ticket does not seem to make any official move towards deprecation of the task itself, but rather the deprecation of having the option of using the task in the context of assembleCoadd. Some clarification of the fate of the task itself vs. simply disabling the possibility of using it in assembleCoadd (and perhaps being a bit more distinctive in the use of the terms "deprecate" vs. "disable") – both here an on the RFC – would be appreciated.

            Regardless, as to the question:

            are there other tests you think should be done?

            Uh...yup Particularly considering and in the context of one of Jim Bosch's comments on the RFC:

            pipeline-output changes tend to be disruptive in unexpected ways

            First, simply creating a few coadds using this branch to make sure they look reasonable (simply running ci_hsc and having it pass does not accomplish this). How different is the relative flux scaling of the final coadds (in RE many of Paul's comments)? (As another visual, creating 3-color images using the same scaling for both should result in similar looking images.) Further, I think processing all the way through multiband to make sure we get consistent output catalogs with the current behavior is an important sanity check.

            I also wonder if we may see any adverse/unexpected effects in the final scaling of the variance of the warps (I appreciate this algorithm shouldn't be sensitive to any overall scaling, but I again refer back to Jim's point above!)

            I would also like to see a coadd created with this branch with the config doScaleZeroPoint set to True to make sure we get bitwise-identical coadds between it and running on current master (or the latest RC2 processing run). Also, along those lines, it seems to me that HSC would still like to continue with the current behavior, in which case this ticket merge should be accompanied by a config override in obs_subaru to set doScaleZeroPoint = True. I'm not 100% confident on this, so perhaps Paul Price/Yusra AlSayyad/Jim Bosch should weigh in here.

            Show
            lauren Lauren MacArthur added a comment - Sorry for the delay on this.  It has taken me a while to digest the comments and resolution on the RFC. I have to admit that I'm not entirely convinced the conclusion was to deprecate ScaleZeroPointTask , but rather adding a (configurable) option to allow it to be disabled.  In particular, Yusra AlSayyad 's last comment ends with a dangling sentence where there is a suggestion of re-working a "new"  ScaleZeroPointTask . This ticket does not seem to make any official move towards deprecation of the task itself, but rather the deprecation of having the option of using the task in the context of assembleCoadd . Some clarification of the fate of the task itself vs. simply disabling the possibility of using it in assembleCoadd (and perhaps being a bit more distinctive in the use of the terms "deprecate" vs. "disable") – both here an on the RFC – would be appreciated. Regardless, as to the question: are there other tests you think should be done? Uh...yup Particularly considering and in the context of one of Jim Bosch 's comments on the RFC: pipeline-output changes tend to be disruptive in unexpected ways First, simply creating a few coadds using this branch to make sure they look reasonable (simply running ci_hsc  and having it pass does not accomplish this). How different is the relative flux scaling of the final coadds (in RE many of Paul's comments)? (As another visual, creating 3-color images using the same scaling for both should result in similar looking images.) Further, I think processing all the way through multiband to make sure we get consistent output catalogs with the current behavior is an important sanity check. I also wonder if we may see any adverse/unexpected effects in the final scaling of the variance of the warps (I appreciate this algorithm shouldn't be sensitive to any overall scaling, but I again refer back to Jim's point above!) I would also like to see a coadd created with this branch with the config doScaleZeroPoint set to True to make sure we get bitwise-identical coadds between it and running on current master (or the latest RC2 processing run). Also, along those lines, it seems to me that HSC would still like to continue with the current behavior, in which case this ticket merge should be accompanied by a config override in obs_subaru to set doScaleZeroPoint = True . I'm not 100% confident on this, so perhaps Paul Price / Yusra AlSayyad / Jim Bosch should weigh in here.
            Hide
            Parejkoj John Parejko added a comment -

            Gabor Kovacs [X]: I'm going to hand this off to you, since you wanted it for building coadds with useful units.

            Show
            Parejkoj John Parejko added a comment - Gabor Kovacs [X] : I'm going to hand this off to you, since you wanted it for building coadds with useful units.
            Hide
            gkovacs Gabor Kovacs [X] (Inactive) added a comment -

            DM-22432 is created to follow this up. I leave this ticket blocked until we figure out whether we want to merge these changes in their present form.

            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - DM-22432 is created to follow this up. I leave this ticket blocked until we figure out whether we want to merge these changes in their present form.
            Hide
            kannawad Arun Kannawadi added a comment -

            Is this a "Won't Fix" ticket? As Fred Moolekamp and I are building and assembling new kinds of coadds, it'd be good to know if this is relevant or not.

            Show
            kannawad Arun Kannawadi added a comment - Is this a "Won't Fix" ticket? As Fred Moolekamp and I are building and assembling new kinds of coadds, it'd be good to know if this is relevant or not.
            Hide
            jbosch Jim Bosch added a comment -

            I think it's preferable for new coadd code to not use this task, and to convert inputs to nJy as early as possible instead.  But removing it from old coaddition code is not a priority for me (in large part because I hope it'll just be superseded by new coadd code eventually), and I if using it is the best way to get new coadd code working (e.g. using existing warps, even if we want to switch to different new warps later), I think that's fine too.

             

            Show
            jbosch Jim Bosch added a comment - I think it's preferable for new coadd code to not use this task, and to convert inputs to nJy as early as possible instead.  But removing it from old coaddition code is not a priority for me (in large part because I hope it'll just be superseded by new coadd code eventually), and I if using it is the best way to get new coadd code working (e.g. using existing warps, even if we want to switch to different new warps later), I think that's fine too.  

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Lauren MacArthur
              Watchers:
              Arun Kannawadi, Hsin-Fang Chiang, Ian Sullivan, Jim Bosch, John Parejko, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.