Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26428

Bin images after difference image of flat pairs has been created (not before) in PTC task.

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 3
    • Data Release Production
    • No

    Description

      We would like to bin the images of the flats that go into the PTC calculation to mitigate the BFE; however, this should be done after the difference image has been constructed, and currently, the code does it before.

      Attachments

        1. binning_before_diff.png
          binning_before_diff.png
          787 kB
        2. PTC_detS00_ComCam_baseline.pdf
          83 kB
        3. PTC_detS00_ComCam_bin16_before_diff.pdf
          90 kB
        4. PTC_detS00_ComCam_bin16.pdf
          90 kB
        5. screenshot-1.png
          screenshot-1.png
          435 kB
        6. screenshot-2.png
          screenshot-2.png
          220 kB
        7. screenshot-3.png
          screenshot-3.png
          587 kB
        8. screenshot-4.png
          screenshot-4.png
          761 kB
        9. screenshot-5.png
          screenshot-5.png
          180 kB
        10. screenshot-6.png
          screenshot-6.png
          1.01 MB

        Activity

          Binning after taking the difference of a pair of flats ( L466: https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/ptc/cpExtractPtcTask.py#L466) does not seem to change the results by much:

          Before:

          VAR  with bin size 16:  13.98844325783439
          Shapes:  (125, 31)
          

          After

          VAR with bin size 16:  13.986454854439325
          Shape:  (125, 31)
          

          Compared to the unbinned case, the calculated variance of the difference seems to be going down by 1/N^2, not by 1/N:

          BIN | VAR
          1  | 3393.97
          8  | 55
          16 | 13.9
          

          As demonstrated by Craig Lage (https://lsstc.slack.com/archives/CQ4SMS3QQ/p1598038376016200), the binning code seems to work but giving a noisy measurement in the estimated gains (I was able to find similar results):

          (gain needs to be divide by bin_size^2)

          plazas Andrés Alejandro Plazas Malagón added a comment - - edited Binning after taking the difference of a pair of flats ( L466: https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/ptc/cpExtractPtcTask.py#L466 ) does not seem to change the results by much: Before: VAR with bin size 16: 13.98844325783439 Shapes: (125, 31) After VAR with bin size 16: 13.986454854439325 Shape: (125, 31) Compared to the unbinned case, the calculated variance of the difference seems to be going down by 1/N^2, not by 1/N: BIN | VAR 1 | 3393.97 8 | 55 16 | 13.9 As demonstrated by Craig Lage ( https://lsstc.slack.com/archives/CQ4SMS3QQ/p1598038376016200 ), the binning code seems to work but giving a noisy measurement in the estimated gains (I was able to find similar results): (gain needs to be divide by bin_size^2 )

          Unbinned PTC (from PTC_detS00_ComCam_baseline.pdf ):

          Binned in patches of 16 pix by 16 pix ( PTC_detS00_ComCam_bin16.pdf ):

          Linearity improves and the BF parameter a00 is reduced, as expected. Gains:

          {'C00': 1.4409759403933793,
           'C01': 1.4241357701873651,
           'C02': 1.4058270768734702,
           'C03': 1.4485359400251967,
           'C04': 1.4378064066300194,
           'C05': 1.451370757055121,
           'C06': 1.441141507216955,
           'C07': 1.4515121129272341,
           'C10': 1.3499575856241333,
           'C11': 1.3572503462324907,
           'C12': 1.3569591047121927,
           'C13': 1.371317340302421,
           'C14': 1.3926476291346988,
           'C15': 1.3512727991392848,
           'C16': 1.362150291802783,
           'C17': 1.3295018571987036}
          

          Binned (divided by 256):

          1.3157244188326096
          1.3839957648222407
          1.4106076147647717
          1.4242643729191227
          1.5029549468242906
          1.469101959008747
          1.4407158444586372
          1.248342453768119
          1.2501857930271565
          1.319580893669786
          1.2754729357451928
          1.3750674170105273
          1.3680232662032672
          1.3710250817858658
          1.3568149159652212
          1.3034134146236087
          

          The curves show some scatter, and we think it's expected due to the downsampling (fewer pixels to measure the variance). Chris pointed out that if we bin the image by convolving with a kernel that retains the number of pixels, the point-to-point scatter should get reduced, which is indeed seen in the following plot by Chris:

          Script to get the PTC data:

           
          REPO=/repo/main
           
          EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082, 2021093000083, 2021093000084, 2021093000085, 2021093000086, 2021093000087, 2021093000088, 2021093000089, 2021093000090, 2021093000091, 2021093000092, 2021093000093, 2021093000094, 2021093000095, 2021093000096, 2021093000097, 2021093000098, 2021093000099, 2021093000100, 2021093000101, 2021093000102, 2021093000103, 2021093000104, 2021093000105, 2021093000106, 2021093000107, 2021093000108, 2021093000109, 2021093000110, 2021093000111, 2021093000112, 2021093000113, 2021093000114, 2021093000115, 2021093000116, 2021093000117, 2021093000118'
           
          #EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082'
           
          DETECTORS='0' #,1,2,3,4,5,6,7,8'
          SUFFIX=2021OCT07.test2.binSize_1_after_diff  #"2021OCT05.1.binSize_16_before_diff"   # ptc_2021OCT05.1.baseline/
          #CAL_COL="LSSTComCam/calib/u/plazas/2021SEP16.1" #"LSSTComCam/calib/u/plazas/2021APR29.5" #"LSSTComCam/calib/2021MAY07" #"u/czw/DM-28920/calib.20210720" #"LSSTComCam/calib/u/plazas/2021JUN09.1"
          #inputCollections = f"LSSTComCam/calib/,{calDir}" #LSSTComCam/raw/all is added by default, it seems.
          PTC_COL=u/plazas/ptc_${SUFFIX}
           
          INPUT_CAL_COL="LSSTComCam/calib/u/cslage/20210930"
           
          pipetask --long-log run -j 9 -d "detector IN (${DETECTORS}) AND instrument='LSSTComCam' AND exposure IN (${EXPOSURES_FLAT}) " \
          -b ${REPO} \
          -i LSSTComCam/raw/all,LSSTComCam/calib,${INPUT_CAL_COL} \
          -o ${PTC_COL} \
          -p ${CP_PIPE_DIR}/pipelines/measurePhotonTransferCurve.yaml \
          -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c ptcExtract:binSize=1 -c ptcSolve:binSize=1 -c ptcSolve:minVarPivotSearch=1 -c isr:doCrosstalk=False -c isr:doFlat=False --register-dataset-types
          

          plazas Andrés Alejandro Plazas Malagón added a comment - Unbinned PTC (from PTC_detS00_ComCam_baseline.pdf ): Binned in patches of 16 pix by 16 pix ( PTC_detS00_ComCam_bin16.pdf ): Linearity improves and the BF parameter a00 is reduced, as expected. Gains: {'C00': 1.4409759403933793, 'C01': 1.4241357701873651, 'C02': 1.4058270768734702, 'C03': 1.4485359400251967, 'C04': 1.4378064066300194, 'C05': 1.451370757055121, 'C06': 1.441141507216955, 'C07': 1.4515121129272341, 'C10': 1.3499575856241333, 'C11': 1.3572503462324907, 'C12': 1.3569591047121927, 'C13': 1.371317340302421, 'C14': 1.3926476291346988, 'C15': 1.3512727991392848, 'C16': 1.362150291802783, 'C17': 1.3295018571987036} Binned (divided by 256): 1.3157244188326096 1.3839957648222407 1.4106076147647717 1.4242643729191227 1.5029549468242906 1.469101959008747 1.4407158444586372 1.248342453768119 1.2501857930271565 1.319580893669786 1.2754729357451928 1.3750674170105273 1.3680232662032672 1.3710250817858658 1.3568149159652212 1.3034134146236087 The curves show some scatter, and we think it's expected due to the downsampling (fewer pixels to measure the variance). Chris pointed out that if we bin the image by convolving with a kernel that retains the number of pixels, the point-to-point scatter should get reduced, which is indeed seen in the following plot by Chris: Script to get the PTC data:   REPO=/repo/main   EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082, 2021093000083, 2021093000084, 2021093000085, 2021093000086, 2021093000087, 2021093000088, 2021093000089, 2021093000090, 2021093000091, 2021093000092, 2021093000093, 2021093000094, 2021093000095, 2021093000096, 2021093000097, 2021093000098, 2021093000099, 2021093000100, 2021093000101, 2021093000102, 2021093000103, 2021093000104, 2021093000105, 2021093000106, 2021093000107, 2021093000108, 2021093000109, 2021093000110, 2021093000111, 2021093000112, 2021093000113, 2021093000114, 2021093000115, 2021093000116, 2021093000117, 2021093000118'   #EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082'   DETECTORS='0' #,1,2,3,4,5,6,7,8' SUFFIX=2021OCT07.test2.binSize_1_after_diff #"2021OCT05.1.binSize_16_before_diff" # ptc_2021OCT05.1.baseline/ #CAL_COL="LSSTComCam/calib/u/plazas/2021SEP16.1" #"LSSTComCam/calib/u/plazas/2021APR29.5" #"LSSTComCam/calib/2021MAY07" #"u/czw/DM-28920/calib.20210720" #"LSSTComCam/calib/u/plazas/2021JUN09.1" #inputCollections = f"LSSTComCam/calib/,{calDir}" #LSSTComCam/raw/all is added by default, it seems. PTC_COL=u/plazas/ptc_${SUFFIX}   INPUT_CAL_COL="LSSTComCam/calib/u/cslage/20210930"   pipetask --long-log run -j 9 -d "detector IN (${DETECTORS}) AND instrument='LSSTComCam' AND exposure IN (${EXPOSURES_FLAT}) " \ -b ${REPO} \ -i LSSTComCam/raw/all,LSSTComCam/calib,${INPUT_CAL_COL} \ -o ${PTC_COL} \ -p ${CP_PIPE_DIR}/pipelines/measurePhotonTransferCurve.yaml \ -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c ptcExtract:binSize=1 -c ptcSolve:binSize=1 -c ptcSolve:minVarPivotSearch=1 -c isr:doCrosstalk=False -c isr:doFlat=False --register-dataset-types

          Do you have the PTC curve plots from the previous version of the code?  Do they also have the lower-flux cutoff points and the upwards curve?  This seems to be over-correcting a lot of the points, effectively throwing away all the points that deviate from the linear trend.

          I'm a bit curious as to the repeatability.  Have you tried more than one PTC set to see if the gains are consistent for either method?

          czw Christopher Waters added a comment - Do you have the PTC curve plots from the previous version of the code?  Do they also have the lower-flux cutoff points and the upwards curve?  This seems to be over-correcting a lot of the points, effectively throwing away all the points that deviate from the linear trend. I'm a bit curious as to the repeatability.  Have you tried more than one PTC set to see if the gains are consistent for either method?

          Binning the two individual images before taking the difference of the images (which is what the current code does), gives similar results to binning after taking the difference ( PTC_detS00_ComCam_bin16_before_diff.pdf ):

          plazas Andrés Alejandro Plazas Malagón added a comment - Binning the two individual images before taking the difference of the images (which is what the current code does), gives similar results to binning after taking the difference ( PTC_detS00_ComCam_bin16_before_diff.pdf ):

          REPO=/repo/main
            
          EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082, 2021093000083, 2021093000084, 2021093000085, 2021093000086, 2021093000087, 2021093000088, 2021093000089, 2021093000090, 2021093000091, 2021093000092, 2021093000093, 2021093000094, 2021093000095, 2021093000096, 2021093000097, 2021093000098, 2021093000099, 2021093000100, 2021093000101, 2021093000102, 2021093000103, 2021093000104, 2021093000105, 2021093000106, 2021093000107, 2021093000108, 2021093000109, 2021093000110, 2021093000111, 2021093000112, 2021093000113, 2021093000114, 2021093000115, 2021093000116, 2021093000117, 2021093000118'
           
          #EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082'
           
          DETECTORS='0' #,1,2,3,4,5,6,7,8'
          SUFFIX=2023FEB13.test3.binSize_16_after_diff  #"2021OCT05.1.binSize_16_before_diff"   # ptc_2021OCT05.1.baseline/
          #CAL_COL="LSSTComCam/calib/u/plazas/2021SEP16.1" #"LSSTComCam/calib/u/plazas/2021APR29.5" #"LSSTComCam/calib/2021MAY07" #"u/czw/DM-28920/calib.20210720" #"LSSTComCam/calib/u/plazas/2021JUN09.1"
          #inputCollections = f"LSSTComCam/calib/,{calDir}" #LSSTComCam/raw/all is added by default, it seems.
          PTC_COL=u/plazas/ptc_${SUFFIX}
           
          INPUT_CAL_COL="LSSTComCam/calib/u/cslage/20210930"
           
          pipetask --long-log run -j 32 -d "detector IN (${DETECTORS}) AND instrument='LSSTComCam' AND exposure IN (${EXPOSURES_FLAT}) " \
          -b ${REPO} \
          -i LSSTComCam/raw/all,LSSTComCam/calib,${INPUT_CAL_COL} \
          -o ${PTC_COL} \
          -p ${CP_PIPE_DIR}/pipelines/cpPtc.yaml \
          -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c ptcExtract:binSize=16 -c ptcSolve:binSize=16 -c ptcSolve:minVarPivotSearch=1 -c isr:doCrosstalk=False -c isr:doFlat=False --register-dataset-types
          

          pipetask --long-log run -p ${CP_PIPE_DIR}/pipelines/cpPlotPtc.yaml -b /sdf/group/rubin/repo/main -o u/plazas/test-DM-26428-plot.1 -i LSSTComCam/raw/all -j 8 -i u/plazas/ptc_2023FEB13.test3.binSize_16_after_diff/20230213T191141Z --register-dataset-types -d "instrument='LSSTComCam' AND detector IN (0) AND exposure=2021093000079" --register-dataset-types
          

          plazas Andrés Alejandro Plazas Malagón added a comment - REPO=/repo/main EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082, 2021093000083, 2021093000084, 2021093000085, 2021093000086, 2021093000087, 2021093000088, 2021093000089, 2021093000090, 2021093000091, 2021093000092, 2021093000093, 2021093000094, 2021093000095, 2021093000096, 2021093000097, 2021093000098, 2021093000099, 2021093000100, 2021093000101, 2021093000102, 2021093000103, 2021093000104, 2021093000105, 2021093000106, 2021093000107, 2021093000108, 2021093000109, 2021093000110, 2021093000111, 2021093000112, 2021093000113, 2021093000114, 2021093000115, 2021093000116, 2021093000117, 2021093000118'   #EXPOSURES_FLAT='2021093000079, 2021093000080, 2021093000081, 2021093000082'   DETECTORS='0' #,1,2,3,4,5,6,7,8' SUFFIX=2023FEB13.test3.binSize_16_after_diff #"2021OCT05.1.binSize_16_before_diff" # ptc_2021OCT05.1.baseline/ #CAL_COL="LSSTComCam/calib/u/plazas/2021SEP16.1" #"LSSTComCam/calib/u/plazas/2021APR29.5" #"LSSTComCam/calib/2021MAY07" #"u/czw/DM-28920/calib.20210720" #"LSSTComCam/calib/u/plazas/2021JUN09.1" #inputCollections = f"LSSTComCam/calib/,{calDir}" #LSSTComCam/raw/all is added by default, it seems. PTC_COL=u/plazas/ptc_${SUFFIX}   INPUT_CAL_COL="LSSTComCam/calib/u/cslage/20210930"   pipetask --long-log run -j 32 -d "detector IN (${DETECTORS}) AND instrument='LSSTComCam' AND exposure IN (${EXPOSURES_FLAT}) " \ -b ${REPO} \ -i LSSTComCam/raw/all,LSSTComCam/calib,${INPUT_CAL_COL} \ -o ${PTC_COL} \ -p ${CP_PIPE_DIR}/pipelines/cpPtc.yaml \ -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c ptcExtract:binSize=16 -c ptcSolve:binSize=16 -c ptcSolve:minVarPivotSearch=1 -c isr:doCrosstalk=False -c isr:doFlat=False --register-dataset-types pipetask --long-log run -p ${CP_PIPE_DIR}/pipelines/cpPlotPtc.yaml -b /sdf/group/rubin/repo/main -o u/plazas/test-DM-26428-plot.1 -i LSSTComCam/raw/all -j 8 -i u/plazas/ptc_2023FEB13.test3.binSize_16_after_diff/20230213T191141Z --register-dataset-types -d "instrument='LSSTComCam' AND detector IN (0) AND exposure=2021093000079" --register-dataset-types

          Sorry this was dropped for so long.  Looks good.

          czw Christopher Waters added a comment - Sorry this was dropped for so long.  Looks good.
          plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38144/pipeline

          People

            plazas Andrés Alejandro Plazas Malagón
            plazas Andrés Alejandro Plazas Malagón
            Christopher Waters
            Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine, Robert Lupton
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.