# CModel priors are weighted incorrectly relative to likelihood

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Sprint:
DRP S17-4
• Team:
Data Release Production

#### Description

When per-pixel variances are turned off, CModel likelihoods are computed without using the variance at all. This would not matter in a pure likelihood fit, but it means the prior and likelihood are not given the appropriate relative weights - and the relative weighting is not even consistent; it depends on the noise level of the image.

Since the typical effect of this is to make the prior much more informative, there is some danger that fixing this bug will cause other problems due to poorly-constrained fits. To avoid this, I'll add a configuration option to tune the relative weighting of the prior via a constant (which we could set to the typical variance level of the images to get behavior like what we have now without the inconsistency).

#### Attachments

1. meanVariance.py
3 kB
2. psf_vs_cmodel.png
409 kB

#### Activity

No builds found.
Jim Bosch created issue -
Field Original Value New Value
Link This issue blocks DM-9870 [ DM-9870 ]
Hide
Jim Bosch added a comment -

Has been in progress since last week.

Show
Jim Bosch added a comment - Has been in progress since last week.
 Status To Do [ 10001 ] In Progress [ 3 ]
 Sprint DRP S17-4 [ 363 ]
Hide
Paul Price added a comment -

You asked for the mean variance by filter for the Wide survey. I used the attached script (meanVariance.py) to collect the median variance for each patch, and then took the median of those. The results:

• g: 0.001076
• r: 0.002587
• i: 0.003834
• z: 0.014951
• y: 0.060202

Example command-line:

 pprice@tiger-sumire:~ $python -c 'from meanVariance import MeanVarianceTask; MeanVarianceTask.parseAndSubmit()' /tigress/HSC/HSC --rerun production-20160523 --id tract=10045^15827^8279^8766^9075^9128^9318^9371^9561^9614^9802^10046^15828^8284^8767^9076^9132^9319^9375^9562^9617^9803^10047^15829^8285^8768^9077^9133^9320^9376^9563^9618^9804^10048^15830^8520^9004^9078^9134^9321^9377^9564^9619^9805^10049^15831^8521^9005^9103^9135^9345^9378^9588^9620^9806^10287^16005^8526^9006^9104^9313^9346^9556^9589^9621^9807^10288^16006^8762^9007^9106^9314^9349^9557^9592^9798^10289^16007^8763^9008^9125^9315^9368^9558^9611^9799^10290^16008^8764^9009^9126^9316^9369^9559^9612^9800^10291^16009^8765^9010^9127^9317^9370^9560^9613^9801 filter=HSC-I --cores 14 --no-versions  It needs a tweaked obs_base with:  pprice@tiger-sumire:~/LSST/obs/base (master *=)$ git diff diff --git a/policy/exposures.yaml b/policy/exposures.yaml index 2e4e85c..ad0d906 100644 --- a/policy/exposures.yaml +++ b/policy/exposures.yaml @@ -8,7 +8,7 @@ deepCoadd_calexp:  persistable: ExposureF  storage: FitsStorage  python: lsst.afw.image.ExposureF - template: deepCoadd-results/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits + template: deepCoadd/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits  level: None  deepDiff_differenceExp:  template: '' 

Show
Paul Price added a comment - You asked for the mean variance by filter for the Wide survey. I used the attached script (meanVariance.py) to collect the median variance for each patch, and then took the median of those. The results: g: 0.001076 r: 0.002587 i: 0.003834 z: 0.014951 y: 0.060202 Example command-line: pprice@tiger-sumire:~ $python -c 'from meanVariance import MeanVarianceTask; MeanVarianceTask.parseAndSubmit()' /tigress/HSC/HSC --rerun production-20160523 --id tract=10045^15827^8279^8766^9075^9128^9318^9371^9561^9614^9802^10046^15828^8284^8767^9076^9132^9319^9375^9562^9617^9803^10047^15829^8285^8768^9077^9133^9320^9376^9563^9618^9804^10048^15830^8520^9004^9078^9134^9321^9377^9564^9619^9805^10049^15831^8521^9005^9103^9135^9345^9378^9588^9620^9806^10287^16005^8526^9006^9104^9313^9346^9556^9589^9621^9807^10288^16006^8762^9007^9106^9314^9349^9557^9592^9798^10289^16007^8763^9008^9125^9315^9368^9558^9611^9799^10290^16008^8764^9009^9126^9316^9369^9559^9612^9800^10291^16009^8765^9010^9127^9317^9370^9560^9613^9801 filter=HSC-I --cores 14 --no-versions It needs a tweaked obs_base with: pprice@tiger-sumire:~/LSST/obs/base (master *=)$ git diff diff --git a/policy/exposures.yaml b/policy/exposures.yaml index 2e4e85c..ad0d906 100644 --- a/policy/exposures.yaml +++ b/policy/exposures.yaml @@ -8,7 +8,7 @@ deepCoadd_calexp: persistable: ExposureF storage: FitsStorage python: lsst.afw.image.ExposureF - template: deepCoadd-results/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits + template: deepCoadd/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits level: None deepDiff_differenceExp: template: ''
 Attachment meanVariance.py [ 29329 ]
 Link This issue blocks DM-9907 [ DM-9907 ]
Hide
Jim Bosch added a comment -

Preliminary tests on UDeep COSMOS runs look good (see attached figure):

• The run with the bug fixed but no scaling factor (middle) preserves extendedness removes the artificial pileup around PSF - CModel = 0.6.
• The run with the bug fixed and a scaling factor derived from the Wide survey i-band coadd variance restores that pileup (and the rest of the original behavior). That's despite the fact that this is UDeep data, so it doesn't look like the results are super sensitive to that factor. This is i-band data, so it's where we'd expect this scaling to do best at reproducing the old behavior, but since it's usually the reference band it's also the band where that matters the most.

So I think this is working well enough that we can do the rest of the testing as part of the usual RC validation.

Show
Jim Bosch added a comment - Preliminary tests on UDeep COSMOS runs look good (see attached figure): The run with the bug fixed but no scaling factor (middle) preserves extendedness removes the artificial pileup around PSF - CModel = 0.6. The run with the bug fixed and a scaling factor derived from the Wide survey i-band coadd variance restores that pileup (and the rest of the original behavior). That's despite the fact that this is UDeep data, so it doesn't look like the results are super sensitive to that factor. This is i-band data, so it's where we'd expect this scaling to do best at reproducing the old behavior, but since it's usually the reference band it's also the band where that matters the most. So I think this is working well enough that we can do the rest of the testing as part of the usual RC validation.
 Attachment psf_vs_cmodel.png [ 29346 ]
Hide
Jim Bosch added a comment - - edited

When we run the HSC RC data, we'll want to revert the configuration change in obs_subaru for the alternate run. With the configuration change in obs_subaru here we should get approximately the same behavior as we had before, but more consistent.

Show
Jim Bosch added a comment - - edited Finally ready for review. When we run the HSC RC data, we'll want to revert the configuration change in obs_subaru for the alternate run. With the configuration change in obs_subaru here we should get approximately the same behavior as we had before, but more consistent.
 Reviewers Paul Price [ price ] Status In Progress [ 3 ] In Review [ 10004 ]
Hide
Paul Price added a comment -

Love the plot!

Some trivial comments on the GitHub PRs.

Show
Paul Price added a comment - Love the plot! Some trivial comments on the GitHub PRs.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
 Epic Link DM-1111 [ 13918 ]
Hide
Jim Bosch added a comment -

So, it turns out this change breaks ci_hsc: there's a check there that greater than 95% of the sources we marked as calib_psfUsed must have base_ClassificationExtendedness_value == 0 on the coadd, and with this change that number is only 94.3% (3 out of 530 sources were classified differently). I think I'm going to call that unnecessary strictness in ci_hsc rather than change the extendedness threshold, but we should consider revisiting the extendedness threshold in the RC processing.

Show
Jim Bosch added a comment - So, it turns out this change breaks ci_hsc: there's a check there that greater than 95% of the sources we marked as calib_psfUsed must have base_ClassificationExtendedness_value == 0 on the coadd, and with this change that number is only 94.3% (3 out of 530 sources were classified differently). I think I'm going to call that unnecessary strictness in ci_hsc rather than change the extendedness threshold, but we should consider revisiting the extendedness threshold in the RC processing.
Hide
Jim Bosch added a comment -

The modified ci_hsc is now running successfully through jenkins, so I'm ready to merge. I'll leave it open a bit longer in case Paul Price wants to comment on the ci_hsc changes.

Show
Jim Bosch added a comment - The modified ci_hsc is now running successfully through jenkins, so I'm ready to merge. I'll leave it open a bit longer in case Paul Price wants to comment on the ci_hsc changes.
Hide
Paul Price added a comment -

No, merge away.

Show
Paul Price added a comment - No, merge away.
Hide
Jim Bosch added a comment -

Merged to master.

Show
Jim Bosch added a comment - Merged to master.
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Component/s ci_hsc [ 13140 ] Component/s obs_subaru [ 10747 ]

#### People

Assignee:
Jim Bosch
Reporter:
Jim Bosch
Reviewers:
Paul Price
Watchers:
Jim Bosch, Paul Price