Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9795

CModel priors are weighted incorrectly relative to likelihood

    XMLWordPrintable

Details

    • 2
    • DRP S17-4
    • Data Release Production

    Description

      When per-pixel variances are turned off, CModel likelihoods are computed without using the variance at all. This would not matter in a pure likelihood fit, but it means the prior and likelihood are not given the appropriate relative weights - and the relative weighting is not even consistent; it depends on the noise level of the image.

      Since the typical effect of this is to make the prior much more informative, there is some danger that fixing this bug will cause other problems due to poorly-constrained fits. To avoid this, I'll add a configuration option to tune the relative weighting of the prior via a constant (which we could set to the typical variance level of the images to get behavior like what we have now without the inconsistency).

      Attachments

        Issue Links

          Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-9870 [ DM-9870 ]
            jbosch Jim Bosch added a comment -

            Has been in progress since last week.

            jbosch Jim Bosch added a comment - Has been in progress since last week.
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Sprint DRP S17-4 [ 363 ]
            price Paul Price added a comment -

            You asked for the mean variance by filter for the Wide survey. I used the attached script (meanVariance.py) to collect the median variance for each patch, and then took the median of those. The results:

            • g: 0.001076
            • r: 0.002587
            • i: 0.003834
            • z: 0.014951
            • y: 0.060202

            Example command-line:

            pprice@tiger-sumire:~ $ python -c 'from meanVariance import MeanVarianceTask; MeanVarianceTask.parseAndSubmit()' /tigress/HSC/HSC --rerun production-20160523 --id tract=10045^15827^8279^8766^9075^9128^9318^9371^9561^9614^9802^10046^15828^8284^8767^9076^9132^9319^9375^9562^9617^9803^10047^15829^8285^8768^9077^9133^9320^9376^9563^9618^9804^10048^15830^8520^9004^9078^9134^9321^9377^9564^9619^9805^10049^15831^8521^9005^9103^9135^9345^9378^9588^9620^9806^10287^16005^8526^9006^9104^9313^9346^9556^9589^9621^9807^10288^16006^8762^9007^9106^9314^9349^9557^9592^9798^10289^16007^8763^9008^9125^9315^9368^9558^9611^9799^10290^16008^8764^9009^9126^9316^9369^9559^9612^9800^10291^16009^8765^9010^9127^9317^9370^9560^9613^9801 filter=HSC-I --cores 14 --no-versions
            

            It needs a tweaked obs_base with:

            pprice@tiger-sumire:~/LSST/obs/base (master *=) $ git diff
            diff --git a/policy/exposures.yaml b/policy/exposures.yaml
            index 2e4e85c..ad0d906 100644
            --- a/policy/exposures.yaml
            +++ b/policy/exposures.yaml
            @@ -8,7 +8,7 @@ deepCoadd_calexp:
                 persistable: ExposureF
                 storage: FitsStorage
                 python: lsst.afw.image.ExposureF
            -    template: deepCoadd-results/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits
            +    template: deepCoadd/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits
                 level: None
             deepDiff_differenceExp:
                 template:      ''
            

            price Paul Price added a comment - You asked for the mean variance by filter for the Wide survey. I used the attached script (meanVariance.py) to collect the median variance for each patch, and then took the median of those. The results: g: 0.001076 r: 0.002587 i: 0.003834 z: 0.014951 y: 0.060202 Example command-line: pprice@tiger-sumire:~ $ python -c 'from meanVariance import MeanVarianceTask; MeanVarianceTask.parseAndSubmit()' /tigress/HSC/HSC --rerun production-20160523 --id tract=10045^15827^8279^8766^9075^9128^9318^9371^9561^9614^9802^10046^15828^8284^8767^9076^9132^9319^9375^9562^9617^9803^10047^15829^8285^8768^9077^9133^9320^9376^9563^9618^9804^10048^15830^8520^9004^9078^9134^9321^9377^9564^9619^9805^10049^15831^8521^9005^9103^9135^9345^9378^9588^9620^9806^10287^16005^8526^9006^9104^9313^9346^9556^9589^9621^9807^10288^16006^8762^9007^9106^9314^9349^9557^9592^9798^10289^16007^8763^9008^9125^9315^9368^9558^9611^9799^10290^16008^8764^9009^9126^9316^9369^9559^9612^9800^10291^16009^8765^9010^9127^9317^9370^9560^9613^9801 filter=HSC-I --cores 14 --no-versions It needs a tweaked obs_base with: pprice@tiger-sumire:~/LSST/obs/base (master *=) $ git diff diff --git a/policy/exposures.yaml b/policy/exposures.yaml index 2e4e85c..ad0d906 100644 --- a/policy/exposures.yaml +++ b/policy/exposures.yaml @@ -8,7 +8,7 @@ deepCoadd_calexp: persistable: ExposureF storage: FitsStorage python: lsst.afw.image.ExposureF - template: deepCoadd-results/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits + template: deepCoadd/%(filter)s/%(tract)d/%(patch)s/calexp-%(filter)s-%(tract)d-%(patch)s.fits level: None deepDiff_differenceExp: template: ''
            price Paul Price made changes -
            Attachment meanVariance.py [ 29329 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-9907 [ DM-9907 ]
            jbosch Jim Bosch added a comment -

            Preliminary tests on UDeep COSMOS runs look good (see attached figure):

            • The run with the bug fixed but no scaling factor (middle) preserves extendedness removes the artificial pileup around PSF - CModel = 0.6.
            • The run with the bug fixed and a scaling factor derived from the Wide survey i-band coadd variance restores that pileup (and the rest of the original behavior). That's despite the fact that this is UDeep data, so it doesn't look like the results are super sensitive to that factor. This is i-band data, so it's where we'd expect this scaling to do best at reproducing the old behavior, but since it's usually the reference band it's also the band where that matters the most.

            So I think this is working well enough that we can do the rest of the testing as part of the usual RC validation.

            jbosch Jim Bosch added a comment - Preliminary tests on UDeep COSMOS runs look good (see attached figure): The run with the bug fixed but no scaling factor (middle) preserves extendedness removes the artificial pileup around PSF - CModel = 0.6. The run with the bug fixed and a scaling factor derived from the Wide survey i-band coadd variance restores that pileup (and the rest of the original behavior). That's despite the fact that this is UDeep data, so it doesn't look like the results are super sensitive to that factor. This is i-band data, so it's where we'd expect this scaling to do best at reproducing the old behavior, but since it's usually the reference band it's also the band where that matters the most. So I think this is working well enough that we can do the rest of the testing as part of the usual RC validation.
            jbosch Jim Bosch made changes -
            Attachment psf_vs_cmodel.png [ 29346 ]
            jbosch Jim Bosch added a comment - - edited

            Finally ready for review.

            When we run the HSC RC data, we'll want to revert the configuration change in obs_subaru for the alternate run. With the configuration change in obs_subaru here we should get approximately the same behavior as we had before, but more consistent.

            jbosch Jim Bosch added a comment - - edited Finally ready for review. When we run the HSC RC data, we'll want to revert the configuration change in obs_subaru for the alternate run. With the configuration change in obs_subaru here we should get approximately the same behavior as we had before, but more consistent.
            jbosch Jim Bosch made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            price Paul Price added a comment -

            Love the plot!

            Some trivial comments on the GitHub PRs.

            price Paul Price added a comment - Love the plot! Some trivial comments on the GitHub PRs.
            price Paul Price made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            swinbank John Swinbank made changes -
            Epic Link DM-1111 [ 13918 ]
            jbosch Jim Bosch added a comment -

            So, it turns out this change breaks ci_hsc: there's a check there that greater than 95% of the sources we marked as calib_psfUsed must have base_ClassificationExtendedness_value == 0 on the coadd, and with this change that number is only 94.3% (3 out of 530 sources were classified differently). I think I'm going to call that unnecessary strictness in ci_hsc rather than change the extendedness threshold, but we should consider revisiting the extendedness threshold in the RC processing.

            jbosch Jim Bosch added a comment - So, it turns out this change breaks ci_hsc: there's a check there that greater than 95% of the sources we marked as calib_psfUsed must have base_ClassificationExtendedness_value == 0 on the coadd, and with this change that number is only 94.3% (3 out of 530 sources were classified differently). I think I'm going to call that unnecessary strictness in ci_hsc rather than change the extendedness threshold, but we should consider revisiting the extendedness threshold in the RC processing.
            jbosch Jim Bosch added a comment -

            The modified ci_hsc is now running successfully through jenkins, so I'm ready to merge. I'll leave it open a bit longer in case price wants to comment on the ci_hsc changes.

            jbosch Jim Bosch added a comment - The modified ci_hsc is now running successfully through jenkins, so I'm ready to merge. I'll leave it open a bit longer in case price wants to comment on the ci_hsc changes.
            price Paul Price added a comment -

            No, merge away.

            price Paul Price added a comment - No, merge away.
            jbosch Jim Bosch added a comment -

            Merged to master.

            jbosch Jim Bosch added a comment - Merged to master.
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            price Paul Price made changes -
            Component/s ci_hsc [ 13140 ]
            Component/s obs_subaru [ 10747 ]

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Paul Price
              Jim Bosch, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.