Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24531

Validate auxTel crosstalk matrix from DM-18050 and DM-23623

    Details

    • Type: Story
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: cp_pipe, ip_isr, obs_lsst
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Apply crosstalk correction to auxTel data with visible crosstalk, and see if it disappears.

        Attachments

          Issue Links

            Activity

            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Just making you jpegs of all the images with amp overlays so you can pick

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Just making you jpegs of all the images with amp overlays so you can pick
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            For the moment, the SLAC matrix from above (including the coefficient [10, 11] = 15.8821e-4) is in

            /project/shared/auxTel/calibrations/crosstalk/crosstalk-det000.yaml
            

            and it can be retrieved via

            import lsst.daf.persistence as dafPersist
            butler = dafPersist.Butler("/project/shared/auxTel/")
             xtalkDictionary = butler.get("crosstalk", detector=0)
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - For the moment, the SLAC matrix from above (including the coefficient [10, 11] = 15.8821e-4) is in /project/shared/auxTel/calibrations/crosstalk/crosstalk-det000.yaml and it can be retrieved via import lsst.daf.persistence as dafPersist butler = dafPersist.Butler( "/project/shared/auxTel/" ) xtalkDictionary = butler.get( "crosstalk" , detector = 0 )
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            Adding more data (using the movies made by Merlin: https://lsst-web.ncsa.illinois.edu/~mfl/) and raising the threshold (to 90K) does get more sources and decrease the errors for the matrix derived with measureCrosstalk.py, but the errors are still somewhat comparable to the coefficients, and the crosstalk does not completely goes away:

            Command (w_18)

             measureCrosstalk.py /project/shared/auxTel/ --rerun plazas/crosstalk-auxTel-2020MAY05-T90K --crosstalkName auxTel --outputFileName /project/shared/auxTel/rerun/plazas/crosstalk-auxTel-2020MAY05-T90K/ctValues.yaml --dump-ratios /project/shared/auxTel/rerun/plazas/crosstalk-auxTel-2020MAY05-T90K/ctRatios.pkl -c isr.doBias=True isr.doLinearize=False -c isr.doDefect=True --id detector=0 expId=2020021700240..2020021700261^2020021700274^2020021700275^2020021700279^2020021700286^2020021700332..2020021700339^2020021700361..2020021700377^2020021800027..2020021800087^2020021800090..2020021800104^2020021800145..2020021800156^2020021800179..2020021800230^2020021800252..2020021800270^2020021800282^2020021800330^2020021900135..2020021900140^2020021900153^2020021900180^2020021900186^2020021900211^2020021900223^2020021900229^2020021900274..2020021900304^2020021900315..2020021900329^2020022000141..2020022000148^2020022000152..2020022000158^2020022000161..2020022000167^2020022000307..2020022000320^2020022000341..2020022000420^2020022000473..2020022000478^2020022100729..2020022100734^2020022100747..2020022100755^2020022100774..2020022100778^2020031200154..2020031200164^2020031200189^2020031300120..2020031300140^2020031300280..2020031300320^2020031400041..2020031400046^2020031400080..2020031400095^2020031400135^2020031400324..2020031400330^2020031500125..2020031500140 -c threshold=90000 --clobber-versions --clobber-config
            

            row 11 of matrix:

            [-2.54445149e-05 4.87099351e-06 -1.63118166e-05 -1.69873056e-05
            -1.23233058e-05 -1.28576867e-05 -1.14204813e-05 -1.04377168e-05
            -6.19106095e-06 4.97129877e-05 6.05482492e-04 0.00000000e+00
            1.74665664e-04 9.71086192e-05 1.67969029e-05 -3.13741621e-05]

            errors for row 11:

            [7.74531072e-05 8.57302366e-05 7.48624734e-05 7.74247528e-05
            7.82358620e-05 7.73914217e-05 7.81350172e-05 7.76592715e-05
            7.89291589e-05 8.03140283e-05 8.96375859e-05 nan
            9.55726136e-05 8.55294129e-05 8.33563608e-05 7.62277195e-05]

            Using Chris's magic number for [10,11] ([10,11] = 15.8821e-4 ):

            Increasing threshold to 110k does not change things much:

            row 11, coeff:

            [-2.35623447e-05 1.06210500e-05 -1.67456965e-05 -1.46037573e-05
            7.61219708e-04 -1.19054484e-05 -8.67544259e-06 -1.41007686e-05
            1.66061018e-05 2.91818928e-04 0.00000000e+00 1.71138858e-03
            5.02814801e-05 3.42305430e-05 -9.75924559e-06 5.82532994e-05]

            row 11, errors:

            [6.97678697e-05 7.81832205e-05 6.60678270e-05 6.85879131e-05
            7.17506555e-05 6.90882371e-05 6.70679365e-05 6.91467430e-05
            7.14215785e-05 7.01673271e-05 7.91788334e-05 nan
            8.44053284e-05 8.31981888e-05 7.99703557e-05 6.83048711e-05]

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited Adding more data (using the movies made by Merlin: https://lsst-web.ncsa.illinois.edu/~mfl/ ) and raising the threshold (to 90K) does get more sources and decrease the errors for the matrix derived with measureCrosstalk.py, but the errors are still somewhat comparable to the coefficients, and the crosstalk does not completely goes away: Command (w_18) measureCrosstalk.py /project/shared/auxTel/ --rerun plazas/crosstalk-auxTel-2020MAY05-T90K --crosstalkName auxTel --outputFileName /project/shared/auxTel/rerun/plazas/crosstalk-auxTel-2020MAY05-T90K/ctValues.yaml --dump-ratios /project/shared/auxTel/rerun/plazas/crosstalk-auxTel-2020MAY05-T90K/ctRatios.pkl -c isr.doBias=True isr.doLinearize=False -c isr.doDefect=True --id detector=0 expId=2020021700240..2020021700261^2020021700274^2020021700275^2020021700279^2020021700286^2020021700332..2020021700339^2020021700361..2020021700377^2020021800027..2020021800087^2020021800090..2020021800104^2020021800145..2020021800156^2020021800179..2020021800230^2020021800252..2020021800270^2020021800282^2020021800330^2020021900135..2020021900140^2020021900153^2020021900180^2020021900186^2020021900211^2020021900223^2020021900229^2020021900274..2020021900304^2020021900315..2020021900329^2020022000141..2020022000148^2020022000152..2020022000158^2020022000161..2020022000167^2020022000307..2020022000320^2020022000341..2020022000420^2020022000473..2020022000478^2020022100729..2020022100734^2020022100747..2020022100755^2020022100774..2020022100778^2020031200154..2020031200164^2020031200189^2020031300120..2020031300140^2020031300280..2020031300320^2020031400041..2020031400046^2020031400080..2020031400095^2020031400135^2020031400324..2020031400330^2020031500125..2020031500140 -c threshold=90000 --clobber-versions --clobber-config row 11 of matrix: [-2.54445149e-05 4.87099351e-06 -1.63118166e-05 -1.69873056e-05 -1.23233058e-05 -1.28576867e-05 -1.14204813e-05 -1.04377168e-05 -6.19106095e-06 4.97129877e-05 6.05482492e-04 0.00000000e+00 1.74665664e-04 9.71086192e-05 1.67969029e-05 -3.13741621e-05] errors for row 11: [7.74531072e-05 8.57302366e-05 7.48624734e-05 7.74247528e-05 7.82358620e-05 7.73914217e-05 7.81350172e-05 7.76592715e-05 7.89291589e-05 8.03140283e-05 8.96375859e-05 nan 9.55726136e-05 8.55294129e-05 8.33563608e-05 7.62277195e-05] Using Chris's magic number for [10,11] ( [10,11] = 15.8821e-4 ): Increasing threshold to 110k does not change things much: row 11, coeff: [-2.35623447e-05 1.06210500e-05 -1.67456965e-05 -1.46037573e-05 7.61219708e-04 -1.19054484e-05 -8.67544259e-06 -1.41007686e-05 1.66061018e-05 2.91818928e-04 0.00000000e+00 1.71138858e-03 5.02814801e-05 3.42305430e-05 -9.75924559e-06 5.82532994e-05] row 11, errors: [6.97678697e-05 7.81832205e-05 6.60678270e-05 6.85879131e-05 7.17506555e-05 6.90882371e-05 6.70679365e-05 6.91467430e-05 7.14215785e-05 7.01673271e-05 7.91788334e-05 nan 8.44053284e-05 8.31981888e-05 7.99703557e-05 6.83048711e-05]
            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - SLAC crosstalk measurements: https://confluence.slac.stanford.edu/pages/viewpage.action?pageId=268380244
            Hide
            cslage Craig Lage added a comment -

            Andrés, nice summary of correcting the LATISS crosstalk.  I just wanted to add that the ribbon cables on the LATISS Dewar that connect the CCD to the WREB board are much longer than the ribbon cables in the production rafts that Adam Snyder measured at SLAC.  So to me it is not surprising that there is more corsstalk in LATISS than in the production rafts.  With the ITL CCD at UC Davis, which has a similar layout to LATISS, we measured crosstalk coefficients to nearest neighbors as high as 1.7E-3, which is close to the value of 1.59E-3 that you found as the "magic number" that best corrected the nearest neighbor crosstalk.

            Show
            cslage Craig Lage added a comment - Andrés, nice summary of correcting the LATISS crosstalk.  I just wanted to add that the ribbon cables on the LATISS Dewar that connect the CCD to the WREB board are much longer than the ribbon cables in the production rafts that Adam Snyder measured at SLAC.  So to me it is not surprising that there is more corsstalk in LATISS than in the production rafts.  With the ITL CCD at UC Davis, which has a similar layout to LATISS, we measured crosstalk coefficients to nearest neighbors as high as 1.7E-3, which is close to the value of 1.59E-3 that you found as the "magic number" that best corrected the nearest neighbor crosstalk.

              People

              • Assignee:
                plazas Andrés Alejandro Plazas Malagón
                Reporter:
                plazas Andrés Alejandro Plazas Malagón
                Watchers:
                Andrés Alejandro Plazas Malagón, Craig Lage, John Swinbank, Merlin Fisher-Levine
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel