Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16785

Investigate configurations for fitting DC2 u-band PSFs

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP S19-1
    • Team:
      Data Release Production

      Description

      Background: https://github.com/LSSTDESC/DC2-production/issues/307

      Quick how to reproduce from Hsin-Fang Chiang:

      processCcd.py /datasets/DC2/repo --rerun private/rhl --id visit=1237924 raftName=R43 detectorName=S22
      

      See if we can come up with an appropriate configuration for selecting and fitting calibration stars with low source density

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Going to lower order in the spatial variation is a sufficiently important way to ensure robustness when there aren't any PSF stars that I think we need to fix this, though I'm also worried that if it's deep in the bowels of PSFEx we need Robert Lupton time (and possibly a non-trivial amount of it) to debug what's going on.  Lauren MacArthur, could you create a ticket for that with a how-to-reproduce?

          To continue our search for a possibly more expedient solution, I wonder if it's time to bring PcaPsf out of cold storage and throw it at this problem.

          Show
          jbosch Jim Bosch added a comment - Going to lower order in the spatial variation is a sufficiently important way to ensure robustness when there aren't any PSF stars that I think we need to fix this, though I'm also worried that if it's deep in the bowels of PSFEx we need Robert Lupton time (and possibly a non-trivial amount of it) to debug what's going on.  Lauren MacArthur , could you create a ticket for that with a how-to-reproduce? To continue our search for a possibly more expedient solution, I wonder if it's time to bring PcaPsf out of cold storage and throw it at this problem.
          Hide
          lauren Lauren MacArthur added a comment -

          Ha... I was just starting to wonder if it would be worth giving PcaPsf a try! I will get to all of the above next week. Thanks both for your comments, suggestions, and insight

          Show
          lauren Lauren MacArthur added a comment - Ha... I was just starting to wonder if it would be worth giving PcaPsf a try! I will get to all of the above next week. Thanks both for your comments, suggestions, and insight
          Hide
          lauren Lauren MacArthur added a comment -

          Ok, I've tried every permutation and combination of SDSS vs. HSM shape and pca vs. psfex and it seems the latter two perform the best in terms of number of sfm failures, so I don't see any reason to change any of those configs for future processing. I also tried changing

          charImage.measurePsf.psfDeterminer["psfex"].samplingSize = 0.5
          

          psfex kicks into the sampling mode set by this parameter when the PSF FWHM drops below 3 pixels, which is always true for the sigma ~ 0.8 pixel visits we are looking at here. Changing this (in either direction, I tried setting it to 1 and 0.25) just ended up in more failures.

          I do strongly recommend changing the fluxMin config:

          charImage.measureApCorr.sourceSelector["objectSize"].fluxMin
          

          to something much smaller than the default of 12500 (HSC overrides this to 4000, which roughly corresponds to a S/N of 20). Ideally, this would actually be a S/N cut which you will be able to set via config once DM-17043 lands (and see that ticket for some discussion and example flux vs. S/N histograms). Using a min S/N of 20, I recover on order 50 additional CCDs in both the u-band 738955 and g-band 254378 visits considered here.

          Another parameter I noticed HSC overrides is

          processCcd.charImage.measurePsf.starSelector['objectSize'].widthMin=0.9
          

          Clearly, the value 0.9 can't be used if the PSF sigma is actually 0.8 pixels, but I tried with 0.78 and recovered 4 additional CCDs in both the u-band visit 738955 and the g-band 254378 (but I haven't really explored the consequences of this cut beyond this.)

          Finally, I note that in looking at a well-sampled r-band visit (440940), I noticed that there does not seem to be a brighter-fatter correction being applied in the processing (see this discussion on Slack, #desc-dm-dc2 on Jan 7, 2019 ~3pm EST) . As such, I explored using a maximum S/N threshold cut (reasoning that the stars affected by BF are not representative of the underlying PSF, so may be hindering the modeling). This did result in a significant recovery of CCDs in the u and g band visits considered here (with no difference for the r band visit, for which all CCDs pass). A sweet-spot in terms of number of CCDs passing seemed to be at S/N~200-250, but this does seem quite a low number, so further investigation may be warranted before adopting any global config change.

          So, the bottom line is that the CCD failures now seem largely down to the failure of the CModel measurement in SFM (needed to compute aperture correction maps for use on the coadds), and that these trace to the flag for the double shapely psf approximation failure:

          modelfit_DoubleShapeletPsfApprox_flag_invalidMoments True
          

          Whether this is simply a matter of hitting a regime of under-sampling that just can't be managed, or if the PSFs in the DC2 simulations are funky in a unique way that exacerbates the effects of under-sampling, or if we could make algorithmic/config changes that would allow for CModel to pass is a matter for future investigation.

          Show
          lauren Lauren MacArthur added a comment - Ok, I've tried every permutation and combination of SDSS vs. HSM shape and pca vs. psfex and it seems the latter two perform the best in terms of number of sfm failures, so I don't see any reason to change any of those configs for future processing. I also tried changing charImage.measurePsf.psfDeterminer[ "psfex" ].samplingSize = 0.5 psfex kicks into the sampling mode set by this parameter when the PSF FWHM drops below 3 pixels, which is always true for the sigma ~ 0.8 pixel visits we are looking at here. Changing this (in either direction, I tried setting it to 1 and 0.25) just ended up in more failures. I do strongly recommend changing the fluxMin config: charImage.measureApCorr.sourceSelector[ "objectSize" ].fluxMin to something much smaller than the default of 12500 (HSC overrides this to 4000, which roughly corresponds to a S/N of 20). Ideally, this would actually be a S/N cut which you will be able to set via config once DM-17043 lands (and see that ticket for some discussion and example flux vs. S/N histograms). Using a min S/N of 20, I recover on order 50 additional CCDs in both the u-band 738955 and g-band 254378 visits considered here. Another parameter I noticed HSC overrides is processCcd.charImage.measurePsf.starSelector[ 'objectSize' ].widthMin = 0.9 Clearly, the value 0.9 can't be used if the PSF sigma is actually 0.8 pixels, but I tried with 0.78 and recovered 4 additional CCDs in both the u-band visit 738955 and the g-band 254378 (but I haven't really explored the consequences of this cut beyond this.) Finally, I note that in looking at a well-sampled r-band visit (440940), I noticed that there does not seem to be a brighter-fatter correction being applied in the processing (see this discussion on Slack , #desc-dm-dc2 on Jan 7, 2019 ~3pm EST) . As such, I explored using a maximum S/N threshold cut (reasoning that the stars affected by BF are not representative of the underlying PSF, so may be hindering the modeling). This did result in a significant recovery of CCDs in the u and g band visits considered here (with no difference for the r band visit, for which all CCDs pass). A sweet-spot in terms of number of CCDs passing seemed to be at S/N~200-250, but this does seem quite a low number, so further investigation may be warranted before adopting any global config change. So, the bottom line is that the CCD failures now seem largely down to the failure of the CModel measurement in SFM (needed to compute aperture correction maps for use on the coadds), and that these trace to the flag for the double shapely psf approximation failure: modelfit_DoubleShapeletPsfApprox_flag_invalidMoments True Whether this is simply a matter of hitting a regime of under-sampling that just can't be managed, or if the PSFs in the DC2 simulations are funky in a unique way that exacerbates the effects of under-sampling, or if we could make algorithmic/config changes that would allow for CModel to pass is a matter for future investigation.
          Hide
          lauren Lauren MacArthur added a comment -

          Would you mind giving this a look and judging whether the investigations done/conclusions & recommendations provided do indeed cover the scope of this ticket? Feel free to push back (or assign further tickets to me!) if you think there is more to be done (@rhl may also want to weigh in here).

          Show
          lauren Lauren MacArthur added a comment - Would you mind giving this a look and judging whether the investigations done/conclusions & recommendations provided do indeed cover the scope of this ticket? Feel free to push back (or assign further tickets to me!) if you think there is more to be done (@rhl may also want to weigh in here).
          Hide
          jbosch Jim Bosch added a comment -

          Thanks for the very thorough investigation. The only thing missing was an action for me, and I've now done that: you were (of course) absolutely right - DoubleShapeletPsfApprox is not relying on the PSF shape slot; it's actually doing its own unweighted moments calculation internally.

          I don't recall exactly what my thinking was, but I think this algorithm probably predates the PSF shape slot, and expecting unweighted moments to work on a PSF model image (which should be nonnegative and essentially noiseless) is a reasonable assumption to make.

          I do still think those ought to be reasonable assumptions, to the extent that I'd be wary of fixing this by making DoubleShapeletPsfApprox more forgiving, because I think that PSF model is pretty clearly unphysical and that could lead to subtler problems downstream.

          Show
          jbosch Jim Bosch added a comment - Thanks for the very thorough investigation. The only thing missing was an action for me, and I've now done that: you were (of course) absolutely right - DoubleShapeletPsfApprox is not relying on the PSF shape slot; it's actually doing its own unweighted moments calculation internally. I don't recall exactly what my thinking was, but I think this algorithm probably predates the PSF shape slot, and expecting unweighted moments to work on a PSF model image (which should be nonnegative and essentially noiseless) is a reasonable assumption to make. I do still think those ought to be reasonable assumptions, to the extent that I'd be wary of fixing this by making DoubleShapeletPsfApprox more forgiving, because I think that PSF model is pretty clearly unphysical and that could lead to subtler problems downstream.

            People

            • Assignee:
              lauren Lauren MacArthur
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Hsin-Fang Chiang, James Chiang, Jim Bosch, Joshua Meyers, Lauren MacArthur, Robert Lupton, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel