Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6490

Investigate calibration zeropoint offset between HSC vs. LSST processCcd.py runs

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      As reported in DM-4730, while the scatter between single frame processing measurements of the same dataset on the HSC vs. LSST stacks is quite good (rms = 0.009 mag between Gaussian fluxes, for example, in the figure shown on that ticket), there is a clear offset (0.0166 mag in the figure shown) in the zeropoint between the two stacks (it is systematic, i.e. no trend with magnitude). The cause may well be due to slight differences in the reference stars selected for calibration. We also speculated about differences in slot definitions used in the calibrations steps (e.g. for aperture corrections, psfex, etc...), so I have rerun visit 1322 through both stacks having forced all apertures used in calibration to be the same, namely a circular aperture of 12 pixels measured using the sinc algorithm (as opposed to "naive"). I have attached the processCcd.py config files for the two runs so my settings can be reproduced.

      Also of note, I am using a meas_algorithms branch on the HSC stack with the following commit:

      commit 173ad0b32ed4f4ab074f1a942d2d3f758e189917
      Author: Lauren MacArthur <lauren@astro.princeton.edu>
      Date:   Wed Jan 13 16:35:59 2016 -0500
       
          Hack to allow flux.aperture to be used in apCorr
          
          Since it does not seem possible to access the nth element of a
          schema element that is an array in the context of setting a config
          override, this allows for flux.aperture to be set as
          calibrate.measureApCorr.reference and it sets it to index 4 (which
          corresponds to a radius of 12 pixels) in the __init__.  This was
          selected to match the current LSST default.
       
      diff --git a/python/lsst/meas/algorithms/measureApCorr.py b/python/lsst/meas/algorithms/measureApCorr.py
      index 9f6c599..f1fa99d 100644
      --- a/python/lsst/meas/algorithms/measureApCorr.py
      +++ b/python/lsst/meas/algorithms/measureApCorr.py
      @@ -81,6 +81,9 @@ class MeasureApCorrTask(lsst.pipe.base.Task):
           def __init__(self, schema, **kwds):
               lsst.pipe.base.Task.__init__(self, **kwds)
               self.reference = KeyTuple(self.config.reference, schema)
      +        if self.config.reference == 'flux.aperture':
      +            print "NOTE: setting aperture correction flux to flux.aperture[4] ==> radius = 12 pixels"
      +            self.reference.flux = self.reference.flux[4]
               self.toCorrect = {}
      

      I attach some of the figures comparing the PSF fluxes from these runs which compare the output of the two stacks having matched the two src catalogs. There are two sets: 1) having adjusted the flux for each source to the zeropoint calculated in the calibration and stored as FLUXMAG0 2) having adjusted the flux for all sources to a common zeropoint (zp=33.0, chosen to roughly match the calibrated zp). Note that my figures do include aperture corrections (in DM-5301, many of the plots show fluxes pre-aperture correction). I have also included plots that directly compare the aperture corrections applied (difference in mag units). Finally, I also include plots comparing the 12 pixel circular aperture mags (i.e. to which no apCorr is added).

      Clearly, the zeropoint determined in the calibration of the two stacks differs between the two stacks and, in particular, there seem to be some very problematic CCDs where the differences are particularly significant (~0.05 mag, and not always in the same direction). Please investigate the source of this discrepancy.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            It appears the CCD variation noted in this issue is an artifact of using the SDSS reference catalogs (sdss-dr9-fink-v5b) vs PanSTARRS (ps1_pv2_20150302 on tiger at Princeton). Below is a plot comparing the processCcd.py output from the HSC 4.0.3 stack versus the LSST (weekly w_2016_32), with no config overrides, using the SDSS reference catalog:

            versus the same run but using the PanSTARRS reference catalog:

            The wild CCD-to-CCD variations are now gone. The remaining differences and offset seen here:

            seem consistent with the conclusions in DM-6491, i.e. they are consistent with the difference being caused by differences in the calibration frames being used in the different stacks.

            Show
            lauren Lauren MacArthur added a comment - It appears the CCD variation noted in this issue is an artifact of using the SDSS reference catalogs (sdss-dr9-fink-v5b) vs PanSTARRS (ps1_pv2_20150302 on tiger at Princeton). Below is a plot comparing the processCcd.py output from the HSC 4.0.3 stack versus the LSST (weekly w_2016_32), with no config overrides, using the SDSS reference catalog: versus the same run but using the PanSTARRS reference catalog: The wild CCD-to-CCD variations are now gone. The remaining differences and offset seen here: seem consistent with the conclusions in DM-6491 , i.e. they are consistent with the difference being caused by differences in the calibration frames being used in the different stacks.
            Hide
            lauren Lauren MacArthur added a comment -

            On further digging, I noted a difference in the weighting schemes for the reference stars in the zeropoint calculation between the two stacks. Specifically, on HSC, the reference star fluxes are being weighted by dmagErr instead of the inverse variance and the sigma was being computed using np.std(dmag) instead of computing it from the weighted sum. This was reported and fixed on LSST with DM-2423 (with this commit). The fix never made it back to the HSC stack, hence the differences in the zeropoints noted in this issue. The likely explanation as to why the SDSS reference catalog revealed this issue while PanSTARRS did not is a much wider spread in the errors in the former.

            To demonstrate, I have re-run the same visit with the HSC stack having corrected the weighting. With this fix, the difference between the processCcd.py runs of both stacks (using all the config overrides that attempt to match the configs) looks like:

            Using sdss-dr9-fink-v5b

            For the most part, the agreement is now good (within the context of the differences in the calibration frames being used in the two stacks) with mean=0.0026, stdev = 0.0083 for matched non-extended sources with i_mag<21. There are still seemingly a few problematic CCDs, namely 38, 39, 46, and 47. A quick look at the images reveals that these are fairly low density fields and the number of stars being used in the photometric fit differs between the stacks:

            CCD 38

            LSST: Magnitude zero point: 32.815483 +/- 0.000705 from 24 stars
            HSC:  Magnitude zero point: 32.947309 +/- 0.000172 from 35 stars
            

            CCD 47

            LSST: Magnitude zero point: 32.807384 +/- 0.000479 from 29 stars
            HSC:  Magnitude zero point: 32.780480 +/- 0.000178 from 35 stars
            

            Compared with a CCD with consistent fits:

            CCD 49

            LSST: Magnitude zero point: 32.777060 +/- 0.000445 from 42 stars
            HSC:  Magnitude zero point: 32.775965 +/- 0.000172 from 46 stars
            

            Now using ps1_pv2_20150302:

            which has fewer (but different!) problematic CCDs and the overall agreement is very good (mean=0.0004, stdev = 0.0071 for matched non-extended sources with i_mag<21).

            Also of potential interest is any differences in the ZP calculation between the two reference catalogs but using the same stack. The following compares two identical LSST runs except on used the sdss-dr9-fink-v5b reference catalog and the other used ps1_pv2_20150302 (on tiger at Princeton):

            Some of the same problematic CCDs noted above reappear here. This simply demonstrates that photometric ZPs can be sensitive to the catalog used (not surprisingly!).

            Having discussed the above with Robert Lupton and John Swinbank, we are satisfied that we have gotten to the root of this issue and that it is now safe to close.

            Show
            lauren Lauren MacArthur added a comment - On further digging, I noted a difference in the weighting schemes for the reference stars in the zeropoint calculation between the two stacks. Specifically, on HSC, the reference star fluxes are being weighted by dmagErr instead of the inverse variance and the sigma was being computed using np.std(dmag) instead of computing it from the weighted sum. This was reported and fixed on LSST with DM-2423 (with this commit). The fix never made it back to the HSC stack, hence the differences in the zeropoints noted in this issue. The likely explanation as to why the SDSS reference catalog revealed this issue while PanSTARRS did not is a much wider spread in the errors in the former. To demonstrate, I have re-run the same visit with the HSC stack having corrected the weighting. With this fix, the difference between the processCcd.py runs of both stacks (using all the config overrides that attempt to match the configs) looks like: Using sdss-dr9-fink-v5b For the most part, the agreement is now good (within the context of the differences in the calibration frames being used in the two stacks) with mean=0.0026, stdev = 0.0083 for matched non-extended sources with i_mag<21. There are still seemingly a few problematic CCDs, namely 38, 39, 46, and 47. A quick look at the images reveals that these are fairly low density fields and the number of stars being used in the photometric fit differs between the stacks: CCD 38 LSST: Magnitude zero point: 32.815483 +/- 0.000705 from 24 stars HSC: Magnitude zero point: 32.947309 +/- 0.000172 from 35 stars CCD 47 LSST: Magnitude zero point: 32.807384 +/- 0.000479 from 29 stars HSC: Magnitude zero point: 32.780480 +/- 0.000178 from 35 stars Compared with a CCD with consistent fits: CCD 49 LSST: Magnitude zero point: 32.777060 +/- 0.000445 from 42 stars HSC: Magnitude zero point: 32.775965 +/- 0.000172 from 46 stars Now using ps1_pv2_20150302 : which has fewer (but different!) problematic CCDs and the overall agreement is very good ( mean=0.0004, stdev = 0.0071 for matched non-extended sources with i_mag<21). Also of potential interest is any differences in the ZP calculation between the two reference catalogs but using the same stack. The following compares two identical LSST runs except on used the sdss-dr9-fink-v5b reference catalog and the other used ps1_pv2_20150302 (on tiger at Princeton): Some of the same problematic CCDs noted above reappear here. This simply demonstrates that photometric ZPs can be sensitive to the catalog used (not surprisingly!). Having discussed the above with Robert Lupton and John Swinbank , we are satisfied that we have gotten to the root of this issue and that it is now safe to close.
            Hide
            lauren Lauren MacArthur added a comment -

            John Swinbank, would you mind giving this a quick look before I mark it as done?

            Show
            lauren Lauren MacArthur added a comment - John Swinbank , would you mind giving this a quick look before I mark it as done?
            Hide
            swinbank John Swinbank added a comment -

            Thanks for this nice summary — looks great!

            Show
            swinbank John Swinbank added a comment - Thanks for this nice summary — looks great!

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                John Swinbank
                Watchers:
                Bob Armstrong, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel