XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Release Production

Description

The variance rescaling we're doing on the coadds (to try to account for missing covariance) is currently causing the HSC weak lensing team a lot of problems, because they affect every uncertainty value we produce and they're hard to include in simulations that otherwise look a lot like the data. I think we need to disable this - and probably take it out of the codebase entirely, just to be safe - before the next major HSC release, even if we don't have anything to replace it. I think it's doing more harm than good. That will require changing coadd detection thresholds to compensate, I think.

Activity

Hide
Jim Bosch added a comment -

I don't think we need to add it to ExposureInfo; I'd be content with just putting it in the flexible metadata.

The issue was that the WL team was trying to make simulations with the same noise properties as our coadds, and the scaled variance made that a bit harder to do, both in terms of making the simulations and providing a similarly-scaled variance plane to the measurement algorithms.

Show
Jim Bosch added a comment - I don't think we need to add it to ExposureInfo; I'd be content with just putting it in the flexible metadata. The issue was that the WL team was trying to make simulations with the same noise properties as our coadds, and the scaled variance made that a bit harder to do, both in terms of making the simulations and providing a similarly-scaled variance plane to the measurement algorithms.
Hide
Paul Price added a comment - - edited

Jim Bosch, would you mind reviewing this, since you know what you want and we're in a hurry to get things in?

 price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-6965=) $git sub commit dcf5c4df8baec049568958594d851e44cd367a82 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price  Date: Wed Nov 1 19:01:32 2017 +1000    DetectCoaddSourcesTask: record variance scaling factor    Without this value, it's difficult to make simulations with the same noise  properties as our coadds.    python/lsst/pipe/tasks/multiBand.py | 3 ++-  1 file changed, 2 insertions(+), 1 deletion(-)     price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-6965=)$ git sub commit da951ac2ae268fafe339f1c01b285be34aa1acf8 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price  Date: Wed Nov 1 18:39:12 2017 +1000    multibandDriver: write metadata from detection    We now put some useful information in the detection metadata, so we need  to make sure it's clean and write it out.    Not using MultiBandDriverTask.writeMetadata because the 'dataRef' that's  provided there is actually a list of data references, and it's not clear  we can identify which entry in the metadata belongs to which dataRef.    python/lsst/pipe/drivers/multiBandDriver.py | 2 ++  1 file changed, 2 insertions(+)     price@pap-laptop:~/LSST/ci_hsc (tickets/DM-6965=) $git sub commit 195b6038d5ef28d55be71e9c07d50f5bcaf7fcfa Author: Paul Price  Date: Wed Nov 1 19:15:54 2017 +1000    DetectionValidation: check that variance scale is recorded    python/lsst/ci/hsc/validate.py | 8 ++++++++  1 file changed, 8 insertions(+)   commit e4080a681ada2b22dbe712fc09164595c1ff1225 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price  Date: Wed Nov 1 20:15:18 2017 +1000    validate: fix printing of assertion values    Can't assume the values are integers.    python/lsst/ci/hsc/validate.py | 8 ++++----  1 file changed, 4 insertions(+), 4 deletions(-)  Show Paul Price added a comment - - edited Jim Bosch , would you mind reviewing this, since you know what you want and we're in a hurry to get things in? price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-6965=)$ git sub commit dcf5c4df8baec049568958594d851e44cd367a82 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 19:01:32 2017 +1000   DetectCoaddSourcesTask: record variance scaling factor Without this value, it's difficult to make simulations with the same noise properties as our coadds.   python/lsst/pipe/tasks/multiBand.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)     price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-6965=) $git sub commit da951ac2ae268fafe339f1c01b285be34aa1acf8 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 18:39:12 2017 +1000 multibandDriver: write metadata from detection We now put some useful information in the detection metadata, so we need to make sure it's clean and write it out. Not using MultiBandDriverTask.writeMetadata because the 'dataRef' that's provided there is actually a list of data references, and it's not clear we can identify which entry in the metadata belongs to which dataRef. python/lsst/pipe/drivers/multiBandDriver.py | 2 ++ 1 file changed, 2 insertions(+) price@pap-laptop:~/LSST/ci_hsc (tickets/DM-6965=)$ git sub commit 195b6038d5ef28d55be71e9c07d50f5bcaf7fcfa Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 19:15:54 2017 +1000   DetectionValidation: check that variance scale is recorded   python/lsst/ci/hsc/validate.py | 8 ++++++++ 1 file changed, 8 insertions(+)   commit e4080a681ada2b22dbe712fc09164595c1ff1225 (HEAD -> tickets/DM-6965, origin/tickets/DM-6965) Author: Paul Price <price@astro.princeton.edu> Date: Wed Nov 1 20:15:18 2017 +1000   validate: fix printing of assertion values Can't assume the values are integers.   python/lsst/ci/hsc/validate.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Hide
Jim Bosch added a comment -

Looks good!

I left some questions on the PRs just to make sure I understand some things, but I have no concerns that need addressing.

Show
Jim Bosch added a comment - Looks good! I left some questions on the PRs just to make sure I understand some things, but I have no concerns that need addressing.
Hide
Paul Price added a comment -

Jenkins passed with some changes (from branch u/price/20171102 in obs_subaru) to fix DM-10443.

Show
Paul Price added a comment - Jenkins passed with some changes (from branch u/price/20171102 in obs_subaru) to fix DM-10443 .
Hide
Paul Price added a comment -

Merged to master.

Show
Paul Price added a comment - Merged to master.

People

• Assignee:
Paul Price
Reporter:
Jim Bosch
Reviewers:
Jim Bosch
Watchers:
Jim Bosch, John Swinbank, Paul Price, Robert Lupton