Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30359

Large ellipticity residuals in deepCoadd_forced_src in some bands in HSC RC2

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 3
    • Data Release Production
    • No

    Description

      In the process of compiling the Characterization Metric Report for v22.0.0 of the Science Pipelines, jcarlin noticed that the residual ellipticity correlation function metric values (TEx) computed with faro increased by two or more orders of magnitude in some bands relative to previous reports (see Section 5 of the report). There have been recent updates in the way that faro computes the TEx metrics. Previously, the reported values came from matched source catalogs (i.e., average of single-epoch measurements) whereas now the calculation is done on the coadd (deepCoadd_forced_src). Also, the scalar metric value reported by faro is now the average of absolute values of the correlation function values over a range of angular scales, rather than the absolute value of the average of the correlation function values over a range of angular scales.

      Upon further investigation, it appears that the i-band performance is roughly as expected, but the r and z bands exhibit large systematic offsets between measured and PSF model ellipticities for selected stars. The effect is persistent with different quality flag selections (e.g., detect_isPrimary and detect_isIsolated) and varying SNR thresholds. The differences are observed in the moments as well.

      The effect was first noticed for collection HSC/runs/RC2/w_2021_18/DM-29973 tract 9615, and is also observed in collection HSC/runs/RC2/w_2021_14/DM-29528 tract 9615.

      Attachments

        Issue Links

          Activity

            Colin S. pulled up the ellipticity residual plots (like those Lauren shared above) for w_18 and w_14, and the residuals are noticeably larger for w_18. Not sure if that is a concern.

            Also, I'm curious about those figures – it says "cat forced scarlet" at the bottom. R. Lupton recently explained that once sources have been deblended by scarlet, their PSFs are not preserved, and they should thus not be used for ellipticity/shape assessment. To ensure non-deblended sources, I think instead of deblend_nChild==0, one has to use deblend_isPrimary and deblend_isIsolated. Can you guys confirm the correct source selection to use?

            jcarlin Jeffrey Carlin added a comment - Colin S. pulled up the ellipticity residual plots (like those Lauren shared above) for w_18 and w_14, and the residuals are noticeably larger for w_18. Not sure if that is a concern. Also, I'm curious about those figures – it says "cat forced scarlet" at the bottom. R. Lupton recently explained that once sources have been deblended by scarlet, their PSFs are not preserved, and they should thus not be used for ellipticity/shape assessment. To ensure non-deblended sources, I think instead of deblend_nChild==0, one has to use deblend_isPrimary and deblend_isIsolated. Can you guys confirm the correct source selection to use?
            ctslater Colin Slater added a comment -

            Hopefully before anyone goes searching in w_14, what I saw was that e1 residuals increased sometime after v21 (https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-psfMagHist.png). It sounds like that's not the main issue here, but might come up again in the characterization report once the column issue is fixed.

            ctslater Colin Slater added a comment - Hopefully before anyone goes searching in w_14, what I saw was that e1 residuals increased sometime after v21 ( https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-psfMagHist.png ). It sounds like that's not the main issue here, but might come up again in the characterization report once the column issue is fixed.

            Jeff, indeed, the label in the plot is telling you that these data come from the forced_src catalog and that the deblender in play in this processing was scarlet. As for the selection of objects, I am basically using detect_isPrimary but without the detect_isTractInner condition. This gives me the non-scarlet-model entry for isolated objects, but I am not selecting only isolated objects, so indeed many of the objects in these plots will have been subject to any PSF-destruction from the deblending process. In this sense, I totally agree that these should not be considered an official "metric" for PSF assessment (and, if anything, should be expected to show larger discrepancies). I do sub-select on the calib_psf_used flag, so you are only seeing stars that were deemed suitable for PSF modeling in the SFM stage, so this is a fairly pristine stellar sample...but now that we're looking at coadd depths, many will suffer from blending not seen at the visit-level (and, as has been much discussed, this does lead to deblending "issues" that now land these objects classified as galaxies [sic] on the coadds as evidenced by the red points in the plot). Since the "official" KPM-type defined metrics are computed elsewhere, this plot does not attempt to mimic it in any way (and all we have here are the src - psfModel (at position of src) residuals, the numbers here do not reflect the TEx metrics which refer to the correlations of these residuals on various angular ranges. So, I guess all my plots are demonstrating is whether this general distribution (mean and spread) is not changing run to run. Since it is entirely possible for the general distribution of the numbers going in to the TEx measurements could look similar in a simple scatter plot, but in reality have entirely different correlation patterns on the sky, the TEx values could indeed vary significantly. The best clue my plots can provide here is the visual provided by the "sky" and quiver plots. E.g. you could blink these to see if any pattern differences jump out (but I suspect that the metric may be more sensitive than the eye in this case):
            https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_14_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-sky-stars.png
            vs.
            https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_18_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-sky-stars.png

            or

            https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_14_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-ellipResids-quiver.png
            vs.
            https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_18_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-ellipResids-quiver.png

            lauren Lauren MacArthur added a comment - Jeff, indeed, the label in the plot is telling you that these data come from the forced_src catalog and that the deblender in play in this processing was scarlet . As for the selection of objects, I am basically using detect_isPrimary but without the detect_isTractInner condition. This gives me the non-scarlet-model entry for isolated objects, but I am not selecting only isolated objects, so indeed many of the objects in these plots will have been subject to any PSF-destruction from the deblending process. In this sense, I totally agree that these should not be considered an official "metric" for PSF assessment (and, if anything, should be expected to show larger discrepancies). I do sub-select on the calib_psf_used flag, so you are only seeing stars that were deemed suitable for PSF modeling in the SFM stage, so this is a fairly pristine stellar sample...but now that we're looking at coadd depths, many will suffer from blending not seen at the visit-level (and, as has been much discussed, this does lead to deblending "issues" that now land these objects classified as galaxies [sic] on the coadds as evidenced by the red points in the plot). Since the "official" KPM-type defined metrics are computed elsewhere, this plot does not attempt to mimic it in any way (and all we have here are the src - psfModel (at position of src) residuals, the numbers here do not reflect the TEx metrics which refer to the correlations of these residuals on various angular ranges. So, I guess all my plots are demonstrating is whether this general distribution (mean and spread) is not changing run to run. Since it is entirely possible for the general distribution of the numbers going in to the TEx measurements could look similar in a simple scatter plot, but in reality have entirely different correlation patterns on the sky, the TEx values could indeed vary significantly. The best clue my plots can provide here is the visual provided by the "sky" and quiver plots. E.g. you could blink these to see if any pattern differences jump out (but I suspect that the metric may be more sensitive than the eye in this case): https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_14_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-sky-stars.png vs. https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_18_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-e1Resids-sky-stars.png or https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_14_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-ellipResids-quiver.png vs. https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_18_qaplots/HSC-R/tract-9615/plot-t9615-HSC-R-ellipResids-quiver.png

            We've found (and fixed) the bug causing the huge differences in ellipticity residuals, but it appears like there are still some subtle differences for reasons probably not linked to `faro` at all. So can we consider this ticket Done?

            kannawad Arun Kannawadi added a comment - We've found (and fixed) the bug causing the huge differences in ellipticity residuals, but it appears like there are still some subtle differences for reasons probably not linked to `faro` at all. So can we consider this ticket Done?

            I think we sufficiently explored this, and I'm happy to call the ticket Done. Thanks!

            jcarlin Jeffrey Carlin added a comment - I think we sufficiently explored this, and I'm happy to call the ticket Done. Thanks!

            People

              kannawad Arun Kannawadi
              kbechtol Keith Bechtol
              Arun Kannawadi, Colin Slater, Jeffrey Carlin, Keith Bechtol, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours
                  3h

                  Jenkins

                    No builds found.