Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29695

Examine brighter-fatter correction impact on variance planes

    XMLWordPrintable

Details

    • Improvement
    • Status: In Progress
    • Resolution: Unresolved
    • None
    • ip_isr
    • None

    Description

      A couple months ago it was noticed that the variance planes in calexps were 10-20% lower than empirical for HSC, and 10% higher than empirical for DECam. Brighter fatter is the most likely source, so the suggestion is to run ScaleVarianceTask on the images right before BF and right after OR on calexps with isr.BF turned on or off to confirm this is the cause of the differences.

      The prior analysis can be found in https://github.com/lsst-dm/diffimTests/tree/master/tickets/DM-22396_ScaleVariance_debug

      Attachments

        1. HSC_EXPAPPROXIMATION_PTC_det73.pdf
          39 kB
        2. HSC_PTC_det73.pdf
          37 kB
        3. image-2021-05-01-13-54-07-685.png
          image-2021-05-01-13-54-07-685.png
          199 kB
        4. image-2021-05-01-13-54-18-630.png
          image-2021-05-01-13-54-18-630.png
          156 kB
        5. image-2021-05-01-13-54-29-623.png
          image-2021-05-01-13-54-29-623.png
          167 kB
        6. PTC_detS29.pdf
          64 kB
        7. screenshot-1.png
          screenshot-1.png
          134 kB
        8. screenshot-2.png
          screenshot-2.png
          276 kB
        9. screenshot-3.png
          screenshot-3.png
          622 kB
        10. screenshot-4.png
          screenshot-4.png
          75 kB
        11. screenshot-5.png
          screenshot-5.png
          286 kB

        Issue Links

          Activity

            Aside: the work on making the PTC task function with DEcam data and HSC data that was developed in this ticket was moved to another ticket, DM-30130, to separate things.

            plazas Andrés Alejandro Plazas Malagón added a comment - Aside: the work on making the PTC task function with DEcam data and HSC data that was developed in this ticket was moved to another ticket, DM-30130 , to separate things.

            Chris checked the math used in Bernstein+17 ("Instrumental response model and detrending for the Dark Energy Camera") and confirmed that their formula for the variance and what's in our ISR code coincide. Therefore, we seem to agree on the methods.

            We discussed with Chris checking the values after the background subtraction that is performed by a context manager in the scale variance factor task before calculating the pixel and image scale factors. This distribution should be centered around zero with a standard deviation close to 1 if we understand the noise distribution. Chris calculated this distribution for DEcam images, and we found that it is indeed centered around zero but with a sigma of ~0.82, which means that the variance plane will be higher by a factor of ~ 0.82^2 = 0.679, which is consistent with the scale factor that we have been finding for DECam images.

            plazas Andrés Alejandro Plazas Malagón added a comment - Chris checked the math used in Bernstein+17 ("Instrumental response model and detrending for the Dark Energy Camera") and confirmed that their formula for the variance and what's in our ISR code coincide. Therefore, we seem to agree on the methods. We discussed with Chris checking the values after the background subtraction that is performed by a context manager in the scale variance factor task before calculating the pixel and image scale factors. This distribution should be centered around zero with a standard deviation close to 1 if we understand the noise distribution. Chris calculated this distribution for DEcam images, and we found that it is indeed centered around zero but with a sigma of ~0.82, which means that the variance plane will be higher by a factor of ~ 0.82^2 = 0.679, which is consistent with the scale factor that we have been finding for DECam images.

            Another check, suggested by Chris, is to look at the scale factor reported by the task after making the variance plane, before making the flat, and after making the flat, and do this for the whole focal plane for a range of DECam exposures. If all the values at each step are the same for a detector, then it's not a feature dependent on the sky, sources, etc.

            Command:

            nohup pipetask run -j 9 -d "detector IN (1..62) AND exposure IN (288976,289016,289409,289493,289614,289820,289871) AND instrument='DECam'" -b /project/mrawls/hits2014-3/butler.yaml -i DECam/raw/all,DECam/calib,DECam/calib/DM-26971 -p /home/plazas/lsst_devel/LSST/obs_decam/pipelines/DRP.yaml#isr -c isr:doBrighterFatter=False -c isr:doLinearize=False -c isr:connections.bias='bias' -c isr:biasDataProductName='bias' -c isr:connections.flat='flat' -c isr:flatDataProductName='flat' -o u/andres/DM-29695/postISR.decam.2021MAY13.6 --register-dataset-types 2>&1 | tee log.DM-29695.2021MAY13.6
            

            Full output in /home/plazas/lsst_devel/LSST/ip_isr/log.DM-29695.2021MAY13.6

            For one detector, the scale factors seem to be consistent across different exposures:

            isr INFO: DM-29695: before var map scaleFactors 57.96983820109894, 58.452972001601424 28987161 N30 61 A
            isr INFO: DM-29695: after var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 A
            isr INFO: DM-29695: before var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 B
            isr INFO: DM-29695: after var map scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61 B
            isr INFO: DM-29695: before flat scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61
            isr INFO: DM-29695: after flat scaleFactors 0.681931749483376, 0.6841051717491675 28987161 N30 61)
            

            Picking one exposure, 289614, the results seem to also be consistent across different detectors. For example,

            isr INFO: DM-29695: before var map scaleFactors 50.29182536205966, 50.7197337712512 28961411 S23 11 A
            isr INFO: DM-29695: after var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 A
            isr INFO: DM-29695: before var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 B
            isr INFO: DM-29695: after var map scaleFactors 0.6629060554064795, 0.6629273259263699 28961411 S23 11 B
            isr INFO: DM-29695: before flat scaleFactors 0.6626422372744426, 0.6626858302814864 28961411 S23 11
            isr INFO: DM-29695: after flat scaleFactors 0.6357515313246155, 0.6352227458010834 28961411 S23 11
            

            isr INFO: DM-29695: before var map scaleFactors 54.73647988872145, 55.15204801863784 28961452 N21 52 A
            isr INFO: DM-29695: after var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 A
            isr INFO: DM-29695: before var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 B
            isr INFO: DM-29695: after var map scaleFactors 0.6934252927812252, 0.6933824287273015 28961452 N21 52 B
            isr INFO: DM-29695: before flat scaleFactors 0.6931716272864122, 0.6931485679424054 28961452 N21 52
            isr INFO: DM-29695: after flat scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52
            isr INFO: DM-29695: scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52
            

            isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A
            isr INFO: DM-29695: after var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A
            isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 B
            isr INFO: DM-29695: after var map scaleFactors 0.4203686394123188, 0.4204171251468196 28961431 S7 31 B
            isr INFO: DM-29695: before flat scaleFactors 0.4203600888247675, 0.4204082965977526 28961431 S7 31
            isr INFO: DM-29695: after flat scaleFactors 0.40519949367064567, 0.4051799370697797 28961431 S7 31
            

            plazas Andrés Alejandro Plazas Malagón added a comment - Another check, suggested by Chris, is to look at the scale factor reported by the task after making the variance plane, before making the flat, and after making the flat, and do this for the whole focal plane for a range of DECam exposures. If all the values at each step are the same for a detector, then it's not a feature dependent on the sky, sources, etc. Command: nohup pipetask run -j 9 -d "detector IN (1..62) AND exposure IN (288976,289016,289409,289493,289614,289820,289871) AND instrument='DECam'" -b /project/mrawls/hits2014-3/butler.yaml -i DECam/raw/all,DECam/calib,DECam/calib/DM-26971 -p /home/plazas/lsst_devel/LSST/obs_decam/pipelines/DRP.yaml#isr -c isr:doBrighterFatter=False -c isr:doLinearize=False -c isr:connections.bias='bias' -c isr:biasDataProductName='bias' -c isr:connections.flat='flat' -c isr:flatDataProductName='flat' -o u/andres/DM-29695/postISR.decam.2021MAY13.6 --register-dataset-types 2>&1 | tee log.DM-29695.2021MAY13.6 Full output in /home/plazas/lsst_devel/LSST/ip_isr/log. DM-29695 .2021MAY13.6 For one detector, the scale factors seem to be consistent across different exposures: isr INFO: DM-29695: before var map scaleFactors 57.96983820109894, 58.452972001601424 28987161 N30 61 A isr INFO: DM-29695: after var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 A isr INFO: DM-29695: before var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 B isr INFO: DM-29695: after var map scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61 B isr INFO: DM-29695: before flat scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61 isr INFO: DM-29695: after flat scaleFactors 0.681931749483376, 0.6841051717491675 28987161 N30 61) Picking one exposure, 289614, the results seem to also be consistent across different detectors. For example, isr INFO: DM-29695: before var map scaleFactors 50.29182536205966, 50.7197337712512 28961411 S23 11 A isr INFO: DM-29695: after var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 A isr INFO: DM-29695: before var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 B isr INFO: DM-29695: after var map scaleFactors 0.6629060554064795, 0.6629273259263699 28961411 S23 11 B isr INFO: DM-29695: before flat scaleFactors 0.6626422372744426, 0.6626858302814864 28961411 S23 11 isr INFO: DM-29695: after flat scaleFactors 0.6357515313246155, 0.6352227458010834 28961411 S23 11 isr INFO: DM-29695: before var map scaleFactors 54.73647988872145, 55.15204801863784 28961452 N21 52 A isr INFO: DM-29695: after var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 A isr INFO: DM-29695: before var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 B isr INFO: DM-29695: after var map scaleFactors 0.6934252927812252, 0.6933824287273015 28961452 N21 52 B isr INFO: DM-29695: before flat scaleFactors 0.6931716272864122, 0.6931485679424054 28961452 N21 52 isr INFO: DM-29695: after flat scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52 isr INFO: DM-29695: scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52 isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A isr INFO: DM-29695: after var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 B isr INFO: DM-29695: after var map scaleFactors 0.4203686394123188, 0.4204171251468196 28961431 S7 31 B isr INFO: DM-29695: before flat scaleFactors 0.4203600888247675, 0.4204082965977526 28961431 S7 31 isr INFO: DM-29695: after flat scaleFactors 0.40519949367064567, 0.4051799370697797 28961431 S7 31

            It seems that:

            • the impact of BFE correction was minimal.
            • the math in the code at the time is consistent with the literature (Bernstein+17)
            • the gains might be a partial culprit.
            • HSc and DEcam provide similar conclusions
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited It seems that: the impact of BFE correction was minimal. the math in the code at the time is consistent with the literature (Bernstein+17) the gains might be a partial culprit. HSc and DEcam provide similar conclusions

            DECam gains with the most recent code seem consistent with the previous finding above (~ 4 e/ADU):

            Forming a new repo at USDF using the data from the old repo at NCSA (same as above):

            butler create DECamGen3Test3-2023NOV09
            butler register-instrument ./DECamGen3Test3-2023NOV09 lsst.obs.decam.DarkEnergyCamera
            butler write-curated-calibrations ./DECamGen3Test3-2023NOV09 DECam
            butler ingest-raws ./DECamGen3Test3-2023NOV09 ./DECamGen3Test2/DECam/raw/all/raw/20121119/ct4m20121119t*/*.fz 
            

            pipetask run -j 8 -d "detector IN (5, 10, 15, 20) AND instrument='DECam' AND exposure IN (153088,153089,153090,153091,153092,153095,153096,153097,153098,153099,153100,153101,153102,153103,153104,153105,153106,153107,153108,153109,153110,153111,153112,153115,153116,153117,153118,153119,153120,153121,153122,153123,153124,153125,153126,153127,153128,153129,153130,153131,153132,153133,153134,153135,153136,153030,153035,153039,153040,153079,153080,153081,153082,153085,153086,153087)" -b ./DECamGen3Test3-2023NOV09 -i DECam/raw/all,DECam/calib  -p ${CP_PIPE_DIR}/pipelines/DarkEnergyCamera/cpPtc.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c ptcIsr:doLinearize=False -c ptcIsr:doCrosstalk=False -c ptcIsr:doDefect=False -c ptcIsr:doBias=False -c ptcIsr:doDark=False -c ptcIsr:doFlat=False -c ptcSolve:doLegacyTurnoffSelection=True -c ptcSolve:sigmaCutPtcOutliers=6 -o DM-29695-ptc_2023NOV09.13 --register-dataset-types
            

            Detector 5:

            butler = dB.Butler("./DECamGen3Test3-2023NOV09", collections=["DM-29695-ptc_2023NOV09.13"])
             
            detector=5
            plot_names = ['ptcVarMean', 'ptcVarMeanLog', 'ptcNormalizedVar', 'ptcCov01Mean', 'ptcCov10Mean', 'ptcVarResiduals',
                                   'ptcNormalizedCov01', 'ptcNormalizedCov10', 'ptcAandBMatrices', 'ptcAandBDistance', ' ptcACumulativeSum', 'ptcARelativeBias']
            for plot_name in plot_names:
             ref = butler.registry.findDataset(plot_name, detector=detector)
             print ("Plot number", plot_name)
             uri = butler.getURI(ref)
             display(Image(data=uri.read()))
            

            plazas Andrés Alejandro Plazas Malagón added a comment - - edited DECam gains with the most recent code seem consistent with the previous finding above (~ 4 e/ADU): Forming a new repo at USDF using the data from the old repo at NCSA (same as above): butler create DECamGen3Test3-2023NOV09 butler register-instrument ./DECamGen3Test3-2023NOV09 lsst.obs.decam.DarkEnergyCamera butler write-curated-calibrations ./DECamGen3Test3-2023NOV09 DECam butler ingest-raws ./DECamGen3Test3-2023NOV09 ./DECamGen3Test2/DECam/raw/all/raw/20121119/ct4m20121119t*/*.fz pipetask run -j 8 -d "detector IN (5, 10, 15, 20) AND instrument='DECam' AND exposure IN (153088,153089,153090,153091,153092,153095,153096,153097,153098,153099,153100,153101,153102,153103,153104,153105,153106,153107,153108,153109,153110,153111,153112,153115,153116,153117,153118,153119,153120,153121,153122,153123,153124,153125,153126,153127,153128,153129,153130,153131,153132,153133,153134,153135,153136,153030,153035,153039,153040,153079,153080,153081,153082,153085,153086,153087)" -b ./DECamGen3Test3-2023NOV09 -i DECam/raw/all,DECam/calib -p ${CP_PIPE_DIR}/pipelines/DarkEnergyCamera/cpPtc.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c ptcIsr:doLinearize=False -c ptcIsr:doCrosstalk=False -c ptcIsr:doDefect=False -c ptcIsr:doBias=False -c ptcIsr:doDark=False -c ptcIsr:doFlat=False -c ptcSolve:doLegacyTurnoffSelection=True -c ptcSolve:sigmaCutPtcOutliers=6 -o DM-29695-ptc_2023NOV09.13 --register-dataset-types Detector 5: butler = dB.Butler( "./DECamGen3Test3-2023NOV09" , collections = [ "DM-29695-ptc_2023NOV09.13" ]) detector = 5 plot_names = [ 'ptcVarMean' , 'ptcVarMeanLog' , 'ptcNormalizedVar' , 'ptcCov01Mean' , 'ptcCov10Mean' , 'ptcVarResiduals' , 'ptcNormalizedCov01' , 'ptcNormalizedCov10' , 'ptcAandBMatrices' , 'ptcAandBDistance' , ' ptcACumulativeSum' , 'ptcARelativeBias' ] for plot_name in plot_names: ref = butler.registry.findDataset(plot_name, detector = detector) print ( "Plot number" , plot_name) uri = butler.getURI(ref) display(Image(data = uri.read()))

            People

              plazas Andrés Alejandro Plazas Malagón
              czw Christopher Waters
              Andrés Alejandro Plazas Malagón, Arun Kannawadi, Christopher Waters, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.