Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6965

Record variance rescaling in coaddition

    Details

    • Team:
      Data Release Production

      Description

      The variance rescaling we're doing on the coadds (to try to account for missing covariance) is currently causing the HSC weak lensing team a lot of problems, because they affect every uncertainty value we produce and they're hard to include in simulations that otherwise look a lot like the data. I think we need to disable this - and probably take it out of the codebase entirely, just to be safe - before the next major HSC release, even if we don't have anything to replace it. I think it's doing more harm than good. That will require changing coadd detection thresholds to compensate, I think.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          I don't think we need to add it to ExposureInfo; I'd be content with just putting it in the flexible metadata.

          The issue was that the WL team was trying to make simulations with the same noise properties as our coadds, and the scaled variance made that a bit harder to do, both in terms of making the simulations and providing a similarly-scaled variance plane to the measurement algorithms.

          Show
          jbosch Jim Bosch added a comment - I don't think we need to add it to ExposureInfo; I'd be content with just putting it in the flexible metadata. The issue was that the WL team was trying to make simulations with the same noise properties as our coadds, and the scaled variance made that a bit harder to do, both in terms of making the simulations and providing a similarly-scaled variance plane to the measurement algorithms.
          Hide
          price Paul Price added a comment - - edited

          Jim Bosch, would you mind reviewing this, since you know what you want and we're in a hurry to get things in?

          price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-6965=) $ git sub
          commit dcf5c4df8baec049568958594d851e44cd367a82 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Nov 1 19:01:32 2017 +1000
           
              DetectCoaddSourcesTask: record variance scaling factor
              
              Without this value, it's difficult to make simulations with the same noise
              properties as our coadds.
           
           python/lsst/pipe/tasks/multiBand.py | 3 ++-
           1 file changed, 2 insertions(+), 1 deletion(-)
           
           
          price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-6965=) $ git sub
          commit da951ac2ae268fafe339f1c01b285be34aa1acf8 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Nov 1 18:39:12 2017 +1000
           
              multibandDriver: write metadata from detection
              
              We now put some useful information in the detection metadata, so we need
              to make sure it's clean and write it out.
              
              Not using MultiBandDriverTask.writeMetadata because the 'dataRef' that's
              provided there is actually a list of data references, and it's not clear
              we can identify which entry in the metadata belongs to which dataRef.
           
           python/lsst/pipe/drivers/multiBandDriver.py | 2 ++
           1 file changed, 2 insertions(+)
           
           
          price@pap-laptop:~/LSST/ci_hsc (tickets/DM-6965=) $ git sub
          commit 195b6038d5ef28d55be71e9c07d50f5bcaf7fcfa
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Nov 1 19:15:54 2017 +1000
           
              DetectionValidation: check that variance scale is recorded
           
           python/lsst/ci/hsc/validate.py | 8 ++++++++
           1 file changed, 8 insertions(+)
           
          commit e4080a681ada2b22dbe712fc09164595c1ff1225 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Nov 1 20:15:18 2017 +1000
           
              validate: fix printing of assertion values
              
              Can't assume the values are integers.
           
           python/lsst/ci/hsc/validate.py | 8 ++++----
           1 file changed, 4 insertions(+), 4 deletions(-)
          

          Show
          price Paul Price added a comment - - edited Jim Bosch , would you mind reviewing this, since you know what you want and we're in a hurry to get things in? price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-6965=) $ git sub commit dcf5c4df8baec049568958594d851e44cd367a82 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 19:01:32 2017 +1000   DetectCoaddSourcesTask: record variance scaling factor Without this value, it's difficult to make simulations with the same noise properties as our coadds.   python/lsst/pipe/tasks/multiBand.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)     price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-6965=) $ git sub commit da951ac2ae268fafe339f1c01b285be34aa1acf8 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 18:39:12 2017 +1000   multibandDriver: write metadata from detection We now put some useful information in the detection metadata, so we need to make sure it's clean and write it out. Not using MultiBandDriverTask.writeMetadata because the 'dataRef' that's provided there is actually a list of data references, and it's not clear we can identify which entry in the metadata belongs to which dataRef.   python/lsst/pipe/drivers/multiBandDriver.py | 2 ++ 1 file changed, 2 insertions(+)     price@pap-laptop:~/LSST/ci_hsc (tickets/DM-6965=) $ git sub commit 195b6038d5ef28d55be71e9c07d50f5bcaf7fcfa Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 19:15:54 2017 +1000   DetectionValidation: check that variance scale is recorded   python/lsst/ci/hsc/validate.py | 8 ++++++++ 1 file changed, 8 insertions(+)   commit e4080a681ada2b22dbe712fc09164595c1ff1225 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 20:15:18 2017 +1000   validate: fix printing of assertion values Can't assume the values are integers.   python/lsst/ci/hsc/validate.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
          Hide
          jbosch Jim Bosch added a comment -

          Looks good!

          I left some questions on the PRs just to make sure I understand some things, but I have no concerns that need addressing.

          Show
          jbosch Jim Bosch added a comment - Looks good! I left some questions on the PRs just to make sure I understand some things, but I have no concerns that need addressing.
          Hide
          price Paul Price added a comment -

          Jenkins passed with some changes (from branch u/price/20171102 in obs_subaru) to fix DM-10443.

          Show
          price Paul Price added a comment - Jenkins passed with some changes (from branch u/price/20171102 in obs_subaru) to fix DM-10443 .
          Hide
          price Paul Price added a comment -

          Merged to master.

          Show
          price Paul Price added a comment - Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, John Swinbank, Paul Price, Robert Lupton
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel