Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29272

PTC task: Validate that the variance calculation provided by awf and the C_00 entry of the covariance matrix produced via FFT using Astier's code is the same

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently, both are measured in the extract task. Then, in solve, the aft variances are used to fit to the POLYNOMIAL and EXPAPPROXIMATION models, while the full covariance matrix, C (of size lag by lag), is used when FULLCOVARIANCE is chosen as ptcFitType.

        Attachments

          Issue Links

            Activity

            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Using mock images with low noise and no bad pixels (similar to what's currently in the unit tests), the fractional difference between the two quantities (last column of the following table, multiplied by 100) is of about 0.03%:

            v1 = varAfw, varAstier (Cov_00)
             
            v1,  v2,  v1/v2,  v1 - v2,  v1/v2 - 1.0
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05
            35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05
            53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05
            70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05
            88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05
            106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05
            124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05
            141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05
            159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05
            

            Details are in the following notebook: DM-29272-afwVar-vs-astierCov00-2021MAR31.pdf

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Using mock images with low noise and no bad pixels (similar to what's currently in the unit tests), the fractional difference between the two quantities (last column of the following table, multiplied by 100 ) is of about 0.03% : v1 = varAfw, varAstier (Cov_00)   v1, v2, v1/v2, v1 - v2, v1/v2 - 1.0 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 17761558.190587375 17761122.858278777 1.000024510404667 435.3323085978627 2.451040466699972e-05 35507086.83886415 35506216.56712814 1.0000245104046601 870.2717360109091 2.4510404660116336e-05 53252977.638272606 53251672.41823293 1.0000245104046577 1305.2200396731496 2.4510404657673845e-05 70998792.25315416 70997052.08667703 1.0000245104046714 1740.1664771288633 2.451040467144061e-05 88745107.86803024 88742932.74283701 1.000024510404671 2175.125193223357 2.4510404670996522e-05 106490599.81075263 106487989.74703187 1.000024510404666 2610.0637207627296 2.4510404665889496e-05 124237875.42441216 124234830.37844545 1.0000245104046703 3045.0459667146206 2.4510404670330388e-05 141983829.22090435 141980349.2250911 1.00002451040466 3479.9958132505417 2.451040465989429e-05 159729728.80735695 159725813.863025 1.000024510404657 3914.944331943989 2.451040465700771e-05 Details are in the following notebook: DM-29272-afwVar-vs-astierCov00-2021MAR31.pdf
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Using BOT data, sometimes the agreement is very good (~1e-5%), sometimes of the order of 0.01-0.1%, but sometimes it’s bad (tens of percent).

            Command:

            pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado-DM-29271.11 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types 
            

            3020100800222094 3020100800221094 C10 0
            fractional Diff:  81.53380744280012
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 81.53380744280012
            3020100800222094 3020100800221094 C11 1
            fractional Diff:  60.944722563153505
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 60.944722563153505
            3020100800222094 3020100800221094 C12 2
            fractional Diff:  68.07483301812269
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 68.07483301812269
            3020100800222094 3020100800221094 C13 3
            fractional Diff:  39.10950256450224
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 39.10950256450224
            3020100800222094 3020100800221094 C14 4
            fractional Diff:  88.34500612805367
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 88.34500612805367
            3020100800222094 3020100800221094 C15 5
            fractional Diff:  83.00982514248888
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 83.00982514248888
            3020100800222094 3020100800221094 C16 6
            fractional Diff:  94.48031837820562
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 94.48031837820562
            3020100800222094 3020100800221094 C17 7
            fractional Diff:  47.361144518206245
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 47.361144518206245
            3020100800222094 3020100800221094 C07 8
            fractional Diff:  91.5344493074709
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 91.5344493074709
            3020100800222094 3020100800221094 C06 9
            fractional Diff:  87.0463458878482
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 87.0463458878482
            3020100800222094 3020100800221094 C05 10
            fractional Diff:  75.96993072700133
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 75.96993072700133
            3020100800222094 3020100800221094 C04 11
            fractional Diff:  51.77244552107647
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 51.77244552107647
            3020100800222094 3020100800221094 C03 12
            fractional Diff:  50.7022164019906
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 50.7022164019906
            3020100800222094 3020100800221094 C02 13
            fractional Diff:  53.16009063163637
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 53.16009063163637
            3020100800222094 3020100800221094 C01 14
            fractional Diff:  78.43735052743898
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 78.43735052743898
            3020100800222094 3020100800221094 C00 15
            fractional Diff:  54.44212488640701
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 54.44212488640701
            3020100800155094 3020100800156094 C10 0
            fractional Diff:  9.896972819056771e-05
            3020100800155094 3020100800156094 C11 1
            fractional Diff:  0.0032806681476094113
            3020100800155094 3020100800156094 C12 2
            fractional Diff:  1.7099768043885688
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.7099768043885688
            3020100800155094 3020100800156094 C13 3
            fractional Diff:  6.549413531950377
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 6.549413531950377
            3020100800155094 3020100800156094 C14 4
            fractional Diff:  9.779741174664736e-05
            3020100800155094 3020100800156094 C15 5
            fractional Diff:  9.779597536230256e-05
            3020100800155094 3020100800156094 C16 6
            fractional Diff:  9.779367784457094e-05
            3020100800155094 3020100800156094 C17 7
            fractional Diff:  9.91423244389722e-05
            3020100800155094 3020100800156094 C07 8
            fractional Diff:  9.913603313815855e-05
            3020100800155094 3020100800156094 C06 9
            fractional Diff:  0.0027410153967788453
            3020100800155094 3020100800156094 C05 10
            fractional Diff:  9.780925429581089e-05
            3020100800155094 3020100800156094 C04 11
            fractional Diff:  9.78027520304181e-05
            3020100800155094 3020100800156094 C03 12
            fractional Diff:  9.779788296970793e-05
            3020100800155094 3020100800156094 C02 13
            fractional Diff:  9.7790810604792e-05
            3020100800155094 3020100800156094 C01 14
            fractional Diff:  9.779318450586771e-05
            3020100800155094 3020100800156094 C00 15
            fractional Diff:  9.881285862878286e-05
            3020100800158094 3020100800159094 C10 0
            fractional Diff:  9.896972210654553e-05
            3020100800158094 3020100800159094 C11 1
            fractional Diff:  9.780678542625765e-05
            3020100800158094 3020100800159094 C12 2
            fractional Diff:  3.2773899916427207
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 3.2773899916427207
            3020100800158094 3020100800159094 C13 3
            fractional Diff:  9.780038334739061e-05
            3020100800158094 3020100800159094 C14 4
            fractional Diff:  9.779741241278117e-05
            3020100800158094 3020100800159094 C15 5
            fractional Diff:  9.779597232029147e-05
            3020100800158094 3020100800159094 C16 6
            fractional Diff:  0.0030967523353075954
            3020100800158094 3020100800159094 C17 7
            fractional Diff:  0.7246736611467686
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7246736611467686
            3020100800158094 3020100800159094 C07 8
            fractional Diff:  9.913602951883149e-05
            3020100800158094 3020100800159094 C06 9
            fractional Diff:  9.779521492614407e-05
            3020100800158094 3020100800159094 C05 10
            fractional Diff:  9.780924414837244e-05
            3020100800158094 3020100800159094 C04 11
            fractional Diff:  9.780275644910574e-05
            3020100800158094 3020100800159094 C03 12
            fractional Diff:  9.77978900085219e-05
            3020100800158094 3020100800159094 C02 13
            fractional Diff:  9.779079890304132e-05
            3020100800158094 3020100800159094 C01 14
            fractional Diff:  9.77931900791873e-05
            3020100800158094 3020100800159094 C00 15
            fractional Diff:  9.881286426871583e-05
            3020100800161094 3020100800162094 C10 0
            fractional Diff:  1.5893083029948296
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.5893083029948296
            3020100800161094 3020100800162094 C11 1
            fractional Diff:  1.7771141248741795
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.7771141248741795
            3020100800161094 3020100800162094 C12 2
            fractional Diff:  5.787465570642314
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 5.787465570642314
            3020100800161094 3020100800162094 C13 3
            fractional Diff:  9.780038490170284e-05
            3020100800161094 3020100800162094 C14 4
            fractional Diff:  9.779741629856176e-05
            3020100800161094 3020100800162094 C15 5
            fractional Diff:  0.7519796717185057
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7519796717185057
            3020100800161094 3020100800162094 C16 6
            fractional Diff:  9.779369483098321e-05
            3020100800161094 3020100800162094 C17 7
            fractional Diff:  9.914228029650474e-05
            3020100800161094 3020100800162094 C07 8
            fractional Diff:  37.352008689423364
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 37.352008689423364
            3020100800161094 3020100800162094 C06 9
            fractional Diff:  2.5754245799020303
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.5754245799020303
            3020100800161094 3020100800162094 C05 10
            fractional Diff:  9.780923309055112e-05
            3020100800161094 3020100800162094 C04 11
            fractional Diff:  9.78027785647484e-05
            3020100800161094 3020100800162094 C03 12
            fractional Diff:  9.779790295372237e-05
            3020100800161094 3020100800162094 C02 13
            fractional Diff:  9.779082326133448e-05
            3020100800161094 3020100800162094 C01 14
            fractional Diff:  0.0030752463881733583
            3020100800161094 3020100800162094 C00 15
            fractional Diff:  9.881286879842577e-05
            3020100800164094 3020100800165094 C10 0
            fractional Diff:  9.896971902012552e-05
            3020100800164094 3020100800165094 C11 1
            fractional Diff:  9.780679099957723e-05
            3020100800164094 3020100800165094 C12 2
            fractional Diff:  9.780849061780117e-05
            3020100800164094 3020100800165094 C13 3
            fractional Diff:  0.4199973622268316
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.4199973622268316
            3020100800164094 3020100800165094 C14 4
            fractional Diff:  9.779741108051354e-05
            3020100800164094 3020100800165094 C15 5
            fractional Diff:  9.779595546710595e-05
            3020100800164094 3020100800165094 C16 6
            fractional Diff:  0.6731444474279313
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6731444474279313
            3020100800164094 3020100800165094 C17 7
            fractional Diff:  1.9569846353705067
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.9569846353705067
            3020100800164094 3020100800165094 C07 8
            fractional Diff:  9.913604408495758e-05
            3020100800164094 3020100800165094 C06 9
            fractional Diff:  9.779518337360571e-05
            3020100800164094 3020100800165094 C05 10
            fractional Diff:  9.780928063030103e-05
            3020100800164094 3020100800165094 C04 11
            fractional Diff:  9.78027691722616e-05
            3020100800164094 3020100800165094 C03 12
            fractional Diff:  0.6880864361574202
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6880864361574202
            3020100800164094 3020100800165094 C02 13
            fractional Diff:  9.779081098226783e-05
            3020100800164094 3020100800165094 C01 14
            fractional Diff:  9.779319256608687e-05
            3020100800164094 3020100800165094 C00 15
            fractional Diff:  0.03433559871320879
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.03433559871320879
            3020100800167094 3020100800168094 C10 0
            fractional Diff:  9.89697190867389e-05
            3020100800167094 3020100800168094 C11 1
            fractional Diff:  0.7472444758620522
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7472444758620522
            3020100800167094 3020100800168094 C12 2
            fractional Diff:  0.6746051094063565
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6746051094063565
            3020100800167094 3020100800168094 C13 3
            fractional Diff:  0.32767688881890633
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.32767688881890633
            3020100800167094 3020100800168094 C14 4
            fractional Diff:  9.779741647619744e-05
            3020100800167094 3020100800168094 C15 5
            fractional Diff:  9.779598821868518e-05
            3020100800167094 3020100800168094 C16 6
            fractional Diff:  9.779368017603929e-05
            3020100800167094 3020100800168094 C17 7
            fractional Diff:  0.18652917705404004
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.18652917705404004
            3020100800167094 3020100800168094 C07 8
            fractional Diff:  0.47624519397281295
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.47624519397281295
            3020100800167094 3020100800168094 C06 9
            fractional Diff:  9.779520322439339e-05
            3020100800167094 3020100800168094 C05 10
            fractional Diff:  0.25426251599734995
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.25426251599734995
            3020100800167094 3020100800168094 C04 11
            fractional Diff:  9.780277427928752e-05
            3020100800167094 3020100800168094 C03 12
            fractional Diff:  9.779787049080113e-05
            3020100800167094 3020100800168094 C02 13
            fractional Diff:  9.779084908512203e-05
            3020100800167094 3020100800168094 C01 14
            fractional Diff:  0.7391955620787871
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7391955620787871
            3020100800167094 3020100800168094 C00 15
            fractional Diff:  0.9518967445093174
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.9518967445093174
            3020100800170094 3020100800171094 C10 0
            fractional Diff:  0.47083621342879356
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.47083621342879356
            3020100800170094 3020100800171094 C11 1
            fractional Diff:  2.1533649212643113
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.1533649212643113
            3020100800170094 3020100800171094 C12 2
            fractional Diff:  9.780851515373001e-05
            3020100800170094 3020100800171094 C13 3
            fractional Diff:  0.8539846247759386
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.8539846247759386
            3020100800170094 3020100800171094 C14 4
            fractional Diff:  0.13341624991648482
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.13341624991648482
            3020100800170094 3020100800171094 C15 5
            fractional Diff:  4.070580087847942
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 4.070580087847942
            3020100800170094 3020100800171094 C16 6
            fractional Diff:  9.779366678674961e-05
            3020100800170094 3020100800171094 C17 7
            fractional Diff:  1.1848584723613276
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.1848584723613276
            3020100800170094 3020100800171094 C07 8
            fractional Diff:  0.40123721857229233
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.40123721857229233
            3020100800170094 3020100800171094 C06 9
            fractional Diff:  2.697308097265194
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.697308097265194
            3020100800170094 3020100800171094 C05 10
            fractional Diff:  9.780927903157988e-05
            3020100800170094 3020100800171094 C04 11
            fractional Diff:  0.2961000134249603
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.2961000134249603
            3020100800170094 3020100800171094 C03 12
            fractional Diff:  4.705865491175675
            cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 4.705865491175675
            3020100800170094 3020100800171094 C02 13
            fractional Diff:  9.779081300287373e-05
            3020100800170094 3020100800171094 C01 14
            fractional Diff:  9.779317009517285e-05
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Using BOT data, sometimes the agreement is very good ( ~1e-5% ), sometimes of the order of 0.01-0.1% , but sometimes it’s bad (tens of percent). Command: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado-DM-29271.11 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types 3020100800222094 3020100800221094 C10 0 fractional Diff: 81.53380744280012 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 81.53380744280012 3020100800222094 3020100800221094 C11 1 fractional Diff: 60.944722563153505 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 60.944722563153505 3020100800222094 3020100800221094 C12 2 fractional Diff: 68.07483301812269 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 68.07483301812269 3020100800222094 3020100800221094 C13 3 fractional Diff: 39.10950256450224 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 39.10950256450224 3020100800222094 3020100800221094 C14 4 fractional Diff: 88.34500612805367 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 88.34500612805367 3020100800222094 3020100800221094 C15 5 fractional Diff: 83.00982514248888 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 83.00982514248888 3020100800222094 3020100800221094 C16 6 fractional Diff: 94.48031837820562 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 94.48031837820562 3020100800222094 3020100800221094 C17 7 fractional Diff: 47.361144518206245 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 47.361144518206245 3020100800222094 3020100800221094 C07 8 fractional Diff: 91.5344493074709 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 91.5344493074709 3020100800222094 3020100800221094 C06 9 fractional Diff: 87.0463458878482 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 87.0463458878482 3020100800222094 3020100800221094 C05 10 fractional Diff: 75.96993072700133 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 75.96993072700133 3020100800222094 3020100800221094 C04 11 fractional Diff: 51.77244552107647 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 51.77244552107647 3020100800222094 3020100800221094 C03 12 fractional Diff: 50.7022164019906 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 50.7022164019906 3020100800222094 3020100800221094 C02 13 fractional Diff: 53.16009063163637 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 53.16009063163637 3020100800222094 3020100800221094 C01 14 fractional Diff: 78.43735052743898 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 78.43735052743898 3020100800222094 3020100800221094 C00 15 fractional Diff: 54.44212488640701 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 54.44212488640701 3020100800155094 3020100800156094 C10 0 fractional Diff: 9.896972819056771e-05 3020100800155094 3020100800156094 C11 1 fractional Diff: 0.0032806681476094113 3020100800155094 3020100800156094 C12 2 fractional Diff: 1.7099768043885688 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.7099768043885688 3020100800155094 3020100800156094 C13 3 fractional Diff: 6.549413531950377 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 6.549413531950377 3020100800155094 3020100800156094 C14 4 fractional Diff: 9.779741174664736e-05 3020100800155094 3020100800156094 C15 5 fractional Diff: 9.779597536230256e-05 3020100800155094 3020100800156094 C16 6 fractional Diff: 9.779367784457094e-05 3020100800155094 3020100800156094 C17 7 fractional Diff: 9.91423244389722e-05 3020100800155094 3020100800156094 C07 8 fractional Diff: 9.913603313815855e-05 3020100800155094 3020100800156094 C06 9 fractional Diff: 0.0027410153967788453 3020100800155094 3020100800156094 C05 10 fractional Diff: 9.780925429581089e-05 3020100800155094 3020100800156094 C04 11 fractional Diff: 9.78027520304181e-05 3020100800155094 3020100800156094 C03 12 fractional Diff: 9.779788296970793e-05 3020100800155094 3020100800156094 C02 13 fractional Diff: 9.7790810604792e-05 3020100800155094 3020100800156094 C01 14 fractional Diff: 9.779318450586771e-05 3020100800155094 3020100800156094 C00 15 fractional Diff: 9.881285862878286e-05 3020100800158094 3020100800159094 C10 0 fractional Diff: 9.896972210654553e-05 3020100800158094 3020100800159094 C11 1 fractional Diff: 9.780678542625765e-05 3020100800158094 3020100800159094 C12 2 fractional Diff: 3.2773899916427207 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 3.2773899916427207 3020100800158094 3020100800159094 C13 3 fractional Diff: 9.780038334739061e-05 3020100800158094 3020100800159094 C14 4 fractional Diff: 9.779741241278117e-05 3020100800158094 3020100800159094 C15 5 fractional Diff: 9.779597232029147e-05 3020100800158094 3020100800159094 C16 6 fractional Diff: 0.0030967523353075954 3020100800158094 3020100800159094 C17 7 fractional Diff: 0.7246736611467686 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7246736611467686 3020100800158094 3020100800159094 C07 8 fractional Diff: 9.913602951883149e-05 3020100800158094 3020100800159094 C06 9 fractional Diff: 9.779521492614407e-05 3020100800158094 3020100800159094 C05 10 fractional Diff: 9.780924414837244e-05 3020100800158094 3020100800159094 C04 11 fractional Diff: 9.780275644910574e-05 3020100800158094 3020100800159094 C03 12 fractional Diff: 9.77978900085219e-05 3020100800158094 3020100800159094 C02 13 fractional Diff: 9.779079890304132e-05 3020100800158094 3020100800159094 C01 14 fractional Diff: 9.77931900791873e-05 3020100800158094 3020100800159094 C00 15 fractional Diff: 9.881286426871583e-05 3020100800161094 3020100800162094 C10 0 fractional Diff: 1.5893083029948296 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.5893083029948296 3020100800161094 3020100800162094 C11 1 fractional Diff: 1.7771141248741795 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.7771141248741795 3020100800161094 3020100800162094 C12 2 fractional Diff: 5.787465570642314 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 5.787465570642314 3020100800161094 3020100800162094 C13 3 fractional Diff: 9.780038490170284e-05 3020100800161094 3020100800162094 C14 4 fractional Diff: 9.779741629856176e-05 3020100800161094 3020100800162094 C15 5 fractional Diff: 0.7519796717185057 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7519796717185057 3020100800161094 3020100800162094 C16 6 fractional Diff: 9.779369483098321e-05 3020100800161094 3020100800162094 C17 7 fractional Diff: 9.914228029650474e-05 3020100800161094 3020100800162094 C07 8 fractional Diff: 37.352008689423364 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 37.352008689423364 3020100800161094 3020100800162094 C06 9 fractional Diff: 2.5754245799020303 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.5754245799020303 3020100800161094 3020100800162094 C05 10 fractional Diff: 9.780923309055112e-05 3020100800161094 3020100800162094 C04 11 fractional Diff: 9.78027785647484e-05 3020100800161094 3020100800162094 C03 12 fractional Diff: 9.779790295372237e-05 3020100800161094 3020100800162094 C02 13 fractional Diff: 9.779082326133448e-05 3020100800161094 3020100800162094 C01 14 fractional Diff: 0.0030752463881733583 3020100800161094 3020100800162094 C00 15 fractional Diff: 9.881286879842577e-05 3020100800164094 3020100800165094 C10 0 fractional Diff: 9.896971902012552e-05 3020100800164094 3020100800165094 C11 1 fractional Diff: 9.780679099957723e-05 3020100800164094 3020100800165094 C12 2 fractional Diff: 9.780849061780117e-05 3020100800164094 3020100800165094 C13 3 fractional Diff: 0.4199973622268316 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.4199973622268316 3020100800164094 3020100800165094 C14 4 fractional Diff: 9.779741108051354e-05 3020100800164094 3020100800165094 C15 5 fractional Diff: 9.779595546710595e-05 3020100800164094 3020100800165094 C16 6 fractional Diff: 0.6731444474279313 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6731444474279313 3020100800164094 3020100800165094 C17 7 fractional Diff: 1.9569846353705067 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.9569846353705067 3020100800164094 3020100800165094 C07 8 fractional Diff: 9.913604408495758e-05 3020100800164094 3020100800165094 C06 9 fractional Diff: 9.779518337360571e-05 3020100800164094 3020100800165094 C05 10 fractional Diff: 9.780928063030103e-05 3020100800164094 3020100800165094 C04 11 fractional Diff: 9.78027691722616e-05 3020100800164094 3020100800165094 C03 12 fractional Diff: 0.6880864361574202 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6880864361574202 3020100800164094 3020100800165094 C02 13 fractional Diff: 9.779081098226783e-05 3020100800164094 3020100800165094 C01 14 fractional Diff: 9.779319256608687e-05 3020100800164094 3020100800165094 C00 15 fractional Diff: 0.03433559871320879 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.03433559871320879 3020100800167094 3020100800168094 C10 0 fractional Diff: 9.89697190867389e-05 3020100800167094 3020100800168094 C11 1 fractional Diff: 0.7472444758620522 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7472444758620522 3020100800167094 3020100800168094 C12 2 fractional Diff: 0.6746051094063565 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.6746051094063565 3020100800167094 3020100800168094 C13 3 fractional Diff: 0.32767688881890633 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.32767688881890633 3020100800167094 3020100800168094 C14 4 fractional Diff: 9.779741647619744e-05 3020100800167094 3020100800168094 C15 5 fractional Diff: 9.779598821868518e-05 3020100800167094 3020100800168094 C16 6 fractional Diff: 9.779368017603929e-05 3020100800167094 3020100800168094 C17 7 fractional Diff: 0.18652917705404004 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.18652917705404004 3020100800167094 3020100800168094 C07 8 fractional Diff: 0.47624519397281295 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.47624519397281295 3020100800167094 3020100800168094 C06 9 fractional Diff: 9.779520322439339e-05 3020100800167094 3020100800168094 C05 10 fractional Diff: 0.25426251599734995 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.25426251599734995 3020100800167094 3020100800168094 C04 11 fractional Diff: 9.780277427928752e-05 3020100800167094 3020100800168094 C03 12 fractional Diff: 9.779787049080113e-05 3020100800167094 3020100800168094 C02 13 fractional Diff: 9.779084908512203e-05 3020100800167094 3020100800168094 C01 14 fractional Diff: 0.7391955620787871 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.7391955620787871 3020100800167094 3020100800168094 C00 15 fractional Diff: 0.9518967445093174 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.9518967445093174 3020100800170094 3020100800171094 C10 0 fractional Diff: 0.47083621342879356 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.47083621342879356 3020100800170094 3020100800171094 C11 1 fractional Diff: 2.1533649212643113 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.1533649212643113 3020100800170094 3020100800171094 C12 2 fractional Diff: 9.780851515373001e-05 3020100800170094 3020100800171094 C13 3 fractional Diff: 0.8539846247759386 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.8539846247759386 3020100800170094 3020100800171094 C14 4 fractional Diff: 0.13341624991648482 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.13341624991648482 3020100800170094 3020100800171094 C15 5 fractional Diff: 4.070580087847942 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 4.070580087847942 3020100800170094 3020100800171094 C16 6 fractional Diff: 9.779366678674961e-05 3020100800170094 3020100800171094 C17 7 fractional Diff: 1.1848584723613276 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 1.1848584723613276 3020100800170094 3020100800171094 C07 8 fractional Diff: 0.40123721857229233 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.40123721857229233 3020100800170094 3020100800171094 C06 9 fractional Diff: 2.697308097265194 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 2.697308097265194 3020100800170094 3020100800171094 C05 10 fractional Diff: 9.780927903157988e-05 3020100800170094 3020100800171094 C04 11 fractional Diff: 0.2961000134249603 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 0.2961000134249603 3020100800170094 3020100800171094 C03 12 fractional Diff: 4.705865491175675 cpPtcExtract WARN: Fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 0.01%: 4.705865491175675 3020100800170094 3020100800171094 C02 13 fractional Diff: 9.779081300287373e-05 3020100800170094 3020100800171094 C01 14 fractional Diff: 9.779317009517285e-05
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            From the output above, it seems that the images that have large errors (tens of percent) have high flux and might be close to or perhaps past saturation.

            Focusing on a flat pair that has a fractional difference of about 94%,

            3020100800215094 3020100800216094 C16 6
            

            whose difference has a mean flux of 145645.3773790084 ADU, the difference diminishes to 79% when the sigma clipping (nSigmaClipPtc) is increased from 5.5 to 20 (i.e., discarding fewer outliers) in the options of afwMathc.VARIANCECLIP inside the function measureMeanVarCov:

                    diffImStatsCtrl = afwMath.StatisticsControl(nSigmaClipPtc,
                                                                nIterSigmaClipPtc,
                                                                diffImMaskVal)
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited From the output above, it seems that the images that have large errors (tens of percent) have high flux and might be close to or perhaps past saturation. Focusing on a flat pair that has a fractional difference of about 94% , 3020100800215094 3020100800216094 C16 6 whose difference has a mean flux of 145645.3773790084 ADU, the difference diminishes to 79% when the sigma clipping ( nSigmaClipPtc ) is increased from 5.5 to 20 (i.e., discarding fewer outliers) in the options of afwMathc.VARIANCECLIP inside the function measureMeanVarCov : diffImStatsCtrl = afwMath.StatisticsControl(nSigmaClipPtc, nIterSigmaClipPtc, diffImMaskVal)
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            I think I found the problem regarding the difference between the number given by afwMath.VARIANCECLIP and Cov00 by Astier’s code. First of all, I think that the images where the difference is of the order of tens of percent are where many pixels are saturated or close to saturation. For example, for images 3020100800215 and 3020100800216 and amp6 (same as in previous comment), where the fractional difference is ~94% }}, the {{SAT value is 143174.0 ADU and the mean of the difference image is 143528.84829382919 ADU. The other amplifiers are similar (meanDiff > SAT):

            muDiff, v1 (afwMATH.VARIANCECLIP), v2 (covAstier[0][3]*0.5), v1/v2, v1 - v2, 100*(np.fabs(v1/v2 - 1.0))
             
            0
            SAT 142002.0 vs meanDiff 146111.00770953728: 
            146111.00770953728 3988.966691156589 22062.494008375485 0.18080307193023037 -18073.527317218897 81.91969280697697
             
            1
            SAT 144371.0 vs meanDiff 146172.37764321262: 
            146172.37764321262 3983.5253021658045 10759.238496490132 0.3702423088274608 -6775.713194324328 62.97576911725392
             
            2
            SAT 144955.0 vs meanDiff 144370.89016559935: 
            144370.89016559935 3536.8688282224684 11194.216841541402 0.31595500411402205 -7657.348013318933 68.4044995885978
             
            3
            SAT 145516.0 vs meanDiff 144215.20472773016: 
            144215.20472773016 3365.666246779511 5666.235002843049 0.593986349858553 -2300.5687560635383 40.6013650141447
             
            4
            SAT 146022.0 vs meanDiff 143931.96538625384: 
            143931.96538625384 3353.129214197134 27337.928649638674 0.12265483816168539 -23984.79943544154 87.73451618383146
             
            5
            SAT 145477.0 vs meanDiff 143506.70749694807: 
            143506.70749694807 3332.5707766439436 19331.864807304002 0.17238744476346768 -15999.294030660058 82.76125552365323
             
            6
            SAT 143174.0 vs meanDiff 143528.84829382919: 
            143528.84829382919 3244.1983847322876 52588.21310313117 0.06169059934343203 -49344.01471839889 93.8309400656568
             
            7
            SAT 140048.0 vs meanDiff 141533.16402868764: 
            141533.16402868764 3597.0310225959956 6820.09318793747 0.5274166970266587 -3223.0621653414746 47.258330297334126
             
            8
            SAT 141798.0 vs meanDiff 145415.373742573: 
            145415.373742573 1922.4891113020303 15621.388891978384 0.12306774542238255 -13698.899780676355 87.69322545776174
             
            9
            SAT 142988.0 vs meanDiff 145056.4913993925: 
            145056.4913993925 1950.8817287622962 11228.575224567765 0.17374258886326283 -9277.693495805468 82.62574111367371
             
            10
            SAT 143252.0 vs meanDiff 146346.19244488003: 
            146346.19244488003 2005.9302192945202 6390.644026685123 0.31388545675810575 -4384.713807390603 68.61145432418942
             
            11
            SAT 143296.0 vs meanDiff 144642.57688154557: 
            144642.57688154557 2311.589051101791 4125.269013843146 0.5603486811029298 -1813.6799627413552 43.965131889707024
             
            12
            SAT 143316.0 vs meanDiff 145578.8111391311: 
            145578.8111391311 2783.5616308941712 4628.868872615999 0.6013481279112245 -1845.3072417218277 39.86518720887755
             
            13
            SAT 143825.0 vs meanDiff 146265.7014518256: 
            146265.7014518256 3100.314428596004 6064.6170776154695 0.511213550487677 -2964.3026490194657 48.878644951232296
             
            14
            SAT 144660.0 vs meanDiff 145866.65145004913: 
            145866.65145004913 2711.288139850514 10498.282284255147 0.25826016737202645 -7786.994144404633 74.17398326279736
             
            15
            SAT 144717.0 vs meanDiff 149183.22573232805: 
            149183.22573232805 2825.013535062864 6154.682874604512 0.45900229022675576 -3329.6693395416482 54.099770977324425
             
            

            Second, changing from afwMath.VARIANCECLIP to simply afwMath.VARIANCE in

                    
            varDiff = 0.5*(afwMath.makeStatistics(diffIm, afwMath.VARIANCE, diffImStatsCtrl).getValue())
            

            in the measureMeanVarCov function already brings the fractional difference down to 0.000102556% ! Here's the result for all the amps:

            0
            SAT 142002.0 vs meanDiff 146111.00770953728: 
            146111.00770953728 22062.516776475644 22062.494008375485 1.0000010319821573 0.022768100159737514 0.00010319821572846166
             
            1
            SAT 144371.0 vs meanDiff 146172.37764321262: 
            146172.37764321262 10759.249531459449 10759.238496490132 1.0000010256273546 0.011034969316824572 0.00010256273546005445
             
            2
            SAT 144955.0 vs meanDiff 144370.89016559935: 
            144370.89016559935 11194.228322824993 11194.216841541402 1.0000010256442013 0.011481283590910607 0.00010256442013467648
             
            3
            SAT 145516.0 vs meanDiff 144215.20472773016: 
            144215.20472773016 5666.24081413372 5666.235002843049 1.0000010256000091 0.005811290670862945 0.00010256000091413142
             
            4
            SAT 146022.0 vs meanDiff 143931.96538625384: 
            143931.96538625384 27337.95668684321 27337.928649638674 1.0000010255789638 0.028037204534484772 0.00010255789637536594
             
            5
            SAT 145477.0 vs meanDiff 143506.70749694807: 
            143506.70749694807 19331.88463329183 19331.864807304002 1.0000010255600287 0.01982598782706191 0.00010255600286779298
             
            6
            SAT 143174.0 vs meanDiff 143528.84829382919: 
            143528.84829382919 52588.26703583309 52588.21310313117 1.0000010255663527 0.05393270191416377 0.00010255663527303227
             
            7
            SAT 140048.0 vs meanDiff 141533.16402868764: 
            141533.16402868764 6820.100239753315 6820.09318793747 1.000001033976465 0.007051815844533849 0.00010339764648925609
             
            8
            SAT 141798.0 vs meanDiff 145415.373742573: 
            145415.373742573 15621.40504543011 15621.388891978384 1.000001034059893 0.016153451726495405 0.00010340598930458356
             
            9
            SAT 142988.0 vs meanDiff 145056.4913993925: 
            145056.4913993925 11228.586740282328 11228.575224567765 1.000001025572197 0.011515714562847279 0.00010255721969443243
             
            10
            SAT 143252.0 vs meanDiff 146346.19244488003: 
            146346.19244488003 6390.650581277063 6390.644026685123 1.0000010256543648 0.006554591939675447 0.00010256543647724214
             
            11
            SAT 143296.0 vs meanDiff 144642.57688154557: 
            144642.57688154557 4125.273244745861 4125.269013843146 1.0000010256065 0.004230902714880358 0.00010256064999492054
             
            12
            SAT 143316.0 vs meanDiff 145578.8111391311: 
            145578.8111391311 4628.8736198571505 4628.868872615999 1.0000010255726144 0.0047472411515627755 0.00010255726143881816
             
            13
            SAT 143825.0 vs meanDiff 146265.7014518256: 
            146265.7014518256 6064.623297161276 6064.6170776154695 1.0000010255463332 0.006219545806743554 0.00010255463331887427
             
            14
            SAT 144660.0 vs meanDiff 145866.65145004913: 
            145866.65145004913 10498.293050955008 10498.282284255147 1.0000010255677614 0.010766699861051165 0.00010255677613812964
             
            15
            SAT 144717.0 vs meanDiff 149183.22573232805: 
            149183.22573232805 6154.68921661151 6154.682874604512 1.0000010304360318 0.006342006998238503 0.0001030436031834725
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited I think I found the problem regarding the difference between the number given by afwMath.VARIANCECLIP and Cov00 by Astier’s code. First of all, I think that the images where the difference is of the order of tens of percent are where many pixels are saturated or close to saturation. For example, for images 3020100800215 and 3020100800216 and amp6 (same as in previous comment), where the fractional difference is ~94% }}, the {{SAT value is 143174.0 ADU and the mean of the difference image is 143528.84829382919 ADU. The other amplifiers are similar (meanDiff > SAT): muDiff, v1 (afwMATH.VARIANCECLIP), v2 (covAstier[0][3]*0.5), v1/v2, v1 - v2, 100*(np.fabs(v1/v2 - 1.0))   0 SAT 142002.0 vs meanDiff 146111.00770953728: 146111.00770953728 3988.966691156589 22062.494008375485 0.18080307193023037 -18073.527317218897 81.91969280697697 1 SAT 144371.0 vs meanDiff 146172.37764321262: 146172.37764321262 3983.5253021658045 10759.238496490132 0.3702423088274608 -6775.713194324328 62.97576911725392 2 SAT 144955.0 vs meanDiff 144370.89016559935: 144370.89016559935 3536.8688282224684 11194.216841541402 0.31595500411402205 -7657.348013318933 68.4044995885978 3 SAT 145516.0 vs meanDiff 144215.20472773016: 144215.20472773016 3365.666246779511 5666.235002843049 0.593986349858553 -2300.5687560635383 40.6013650141447 4 SAT 146022.0 vs meanDiff 143931.96538625384: 143931.96538625384 3353.129214197134 27337.928649638674 0.12265483816168539 -23984.79943544154 87.73451618383146 5 SAT 145477.0 vs meanDiff 143506.70749694807: 143506.70749694807 3332.5707766439436 19331.864807304002 0.17238744476346768 -15999.294030660058 82.76125552365323 6 SAT 143174.0 vs meanDiff 143528.84829382919: 143528.84829382919 3244.1983847322876 52588.21310313117 0.06169059934343203 -49344.01471839889 93.8309400656568 7 SAT 140048.0 vs meanDiff 141533.16402868764: 141533.16402868764 3597.0310225959956 6820.09318793747 0.5274166970266587 -3223.0621653414746 47.258330297334126 8 SAT 141798.0 vs meanDiff 145415.373742573: 145415.373742573 1922.4891113020303 15621.388891978384 0.12306774542238255 -13698.899780676355 87.69322545776174 9 SAT 142988.0 vs meanDiff 145056.4913993925: 145056.4913993925 1950.8817287622962 11228.575224567765 0.17374258886326283 -9277.693495805468 82.62574111367371 10 SAT 143252.0 vs meanDiff 146346.19244488003: 146346.19244488003 2005.9302192945202 6390.644026685123 0.31388545675810575 -4384.713807390603 68.61145432418942 11 SAT 143296.0 vs meanDiff 144642.57688154557: 144642.57688154557 2311.589051101791 4125.269013843146 0.5603486811029298 -1813.6799627413552 43.965131889707024 12 SAT 143316.0 vs meanDiff 145578.8111391311: 145578.8111391311 2783.5616308941712 4628.868872615999 0.6013481279112245 -1845.3072417218277 39.86518720887755 13 SAT 143825.0 vs meanDiff 146265.7014518256: 146265.7014518256 3100.314428596004 6064.6170776154695 0.511213550487677 -2964.3026490194657 48.878644951232296 14 SAT 144660.0 vs meanDiff 145866.65145004913: 145866.65145004913 2711.288139850514 10498.282284255147 0.25826016737202645 -7786.994144404633 74.17398326279736 15 SAT 144717.0 vs meanDiff 149183.22573232805: 149183.22573232805 2825.013535062864 6154.682874604512 0.45900229022675576 -3329.6693395416482 54.099770977324425 Second, changing from afwMath.VARIANCECLIP to simply afwMath.VARIANCE in varDiff = 0.5 * (afwMath.makeStatistics(diffIm, afwMath.VARIANCE, diffImStatsCtrl).getValue()) in the measureMeanVarCov function already brings the fractional difference down to 0.000102556% ! Here's the result for all the amps: 0 SAT 142002.0 vs meanDiff 146111.00770953728: 146111.00770953728 22062.516776475644 22062.494008375485 1.0000010319821573 0.022768100159737514 0.00010319821572846166 1 SAT 144371.0 vs meanDiff 146172.37764321262: 146172.37764321262 10759.249531459449 10759.238496490132 1.0000010256273546 0.011034969316824572 0.00010256273546005445 2 SAT 144955.0 vs meanDiff 144370.89016559935: 144370.89016559935 11194.228322824993 11194.216841541402 1.0000010256442013 0.011481283590910607 0.00010256442013467648 3 SAT 145516.0 vs meanDiff 144215.20472773016: 144215.20472773016 5666.24081413372 5666.235002843049 1.0000010256000091 0.005811290670862945 0.00010256000091413142 4 SAT 146022.0 vs meanDiff 143931.96538625384: 143931.96538625384 27337.95668684321 27337.928649638674 1.0000010255789638 0.028037204534484772 0.00010255789637536594 5 SAT 145477.0 vs meanDiff 143506.70749694807: 143506.70749694807 19331.88463329183 19331.864807304002 1.0000010255600287 0.01982598782706191 0.00010255600286779298 6 SAT 143174.0 vs meanDiff 143528.84829382919: 143528.84829382919 52588.26703583309 52588.21310313117 1.0000010255663527 0.05393270191416377 0.00010255663527303227 7 SAT 140048.0 vs meanDiff 141533.16402868764: 141533.16402868764 6820.100239753315 6820.09318793747 1.000001033976465 0.007051815844533849 0.00010339764648925609 8 SAT 141798.0 vs meanDiff 145415.373742573: 145415.373742573 15621.40504543011 15621.388891978384 1.000001034059893 0.016153451726495405 0.00010340598930458356 9 SAT 142988.0 vs meanDiff 145056.4913993925: 145056.4913993925 11228.586740282328 11228.575224567765 1.000001025572197 0.011515714562847279 0.00010255721969443243 10 SAT 143252.0 vs meanDiff 146346.19244488003: 146346.19244488003 6390.650581277063 6390.644026685123 1.0000010256543648 0.006554591939675447 0.00010256543647724214 11 SAT 143296.0 vs meanDiff 144642.57688154557: 144642.57688154557 4125.273244745861 4125.269013843146 1.0000010256065 0.004230902714880358 0.00010256064999492054 12 SAT 143316.0 vs meanDiff 145578.8111391311: 145578.8111391311 4628.8736198571505 4628.868872615999 1.0000010255726144 0.0047472411515627755 0.00010255726143881816 13 SAT 143825.0 vs meanDiff 146265.7014518256: 146265.7014518256 6064.623297161276 6064.6170776154695 1.0000010255463332 0.006219545806743554 0.00010255463331887427 14 SAT 144660.0 vs meanDiff 145866.65145004913: 145866.65145004913 10498.293050955008 10498.282284255147 1.0000010255677614 0.010766699861051165 0.00010255677613812964 15 SAT 144717.0 vs meanDiff 149183.22573232805: 149183.22573232805 6154.68921661151 6154.682874604512 1.0000010304360318 0.006342006998238503 0.0001030436031834725
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            The clipping algorithm in afwMATH.VARIANCECLIP already ignores the pixels in the mask planes. In addition to these masked pixels, we should also identify the pixels that have been clipped in the process. This is accomplished by:

                    varClip = afwMath.makeStatistics(diffIm, afwMath.VARIANCECLIP, diffImStatsCtrl).getValue()
                    meanClip = afwMath.makeStatistics(diffIm, afwMath.MEANCLIP, diffImStatsCtrl).getValue()
             
                    cut = meanClip + nSigmaClipPtc*np.sqrt(varClip)
                    unmasked = np.where(np.fabs(diffIm.image.array) <= cut, 1, 0)
            

            Then, the matrix `w` (defined as follows) should be passed to the Covariance calculations:

                    wDiff = np.where(diffIm.getMask().getArray() == 0, 1, 0)
                    w = wDiff*unmasked
            

            Including these modifications, the fractional difference goes below

            {1%}

            for all amplifiers (except one; 0.226% for amp 6). I'll add a warning in the code for the case when the difference is larger than 1%:

            0
            SAT 142002.0 vs meanDiff 148049.49503693567: 
            148049.49503693567 4113.619098129532 4124.8528603178065 0.9972765665664475 -11.23376218827434 0.27234334335525245
             
            1
            SAT 144371.0 vs meanDiff 147845.09956447146: 
            147845.09956447146 4138.948451455616 4157.531618939209 0.9955302402514659 -18.58316748359266 0.4469759748534141
             
            2
            SAT 144955.0 vs meanDiff 146294.38210178568: 
            146294.38210178568 3637.4780519048513 3646.6700047809695 0.9974793570945364 -9.191952876118194 0.25206429054636104
             
            3
            SAT 145516.0 vs meanDiff 146373.49562630185: 
            146373.49562630185 3425.5526407749344 3431.180479595854 0.9983597951625143 -5.627838820919806 0.1640204837485726
             
            4
            SAT 146022.0 vs meanDiff 144879.95020856708: 
            144879.95020856708 3408.484298284876 3390.9913228028795 1.0051586612340657 17.492975481996382 0.515866123406572
             
            5
            SAT 145477.0 vs meanDiff 144201.99522342085: 
            144201.99522342085 3380.96070882588 3372.1831245363237 1.0026029382051318 8.777584289556216 0.26029382051317995
             
            6
            SAT 143174.0 vs meanDiff 144219.2800343227: 
            144219.2800343227 3271.670691511372 3264.264277571426 1.0022689382078636 7.406413939945651 0.22689382078635578
             
            7
            SAT 140048.0 vs meanDiff 143247.12360096083: 
            143247.12360096083 3736.8108764375993 3423.3841236906374 1.0915546551080768 313.42675274696194 9.15546551080768
             
            8
            SAT 141798.0 vs meanDiff 145785.97266838083: 
            145785.97266838083 1939.4518418819803 1935.6243057743025 1.0019774168449216 3.8275361076778154 0.19774168449215868
             
            9
            SAT 142988.0 vs meanDiff 145427.4797746908: 
            145427.4797746908 1977.1675410522487 1972.7115474516563 1.002258816605169 4.455993600592365 0.22588166051689118
             
            10
            SAT 143252.0 vs meanDiff 146687.08830933238: 
            146687.08830933238 2038.3095264059475 2034.7059315895585 1.001771064191853 3.603594816388977 0.17710641918529113
             
            11
            SAT 143296.0 vs meanDiff 145111.99779136135: 
            145111.99779136135 2360.4288614212046 2354.9684209445572 1.00231868946865 5.4604404766473635 0.23186894686499926
             
            12
            SAT 143316.0 vs meanDiff 146008.9060246319: 
            146008.9060246319 2819.666517175569 2813.5489876615857 1.0021743106449579 6.1175295139833 0.21743106449578775
             
            13
            SAT 143825.0 vs meanDiff 147121.24589549645: 
            147121.24589549645 3190.9693868805393 3162.304178877705 1.0090646586733498 28.665208002834333 0.9064658673349824
             
            14
            SAT 144660.0 vs meanDiff 146577.15637499627: 
            146577.15637499627 2768.8894534582087 2761.1743842910937 1.0027941260106596 7.715069167114962 0.27941260106596477
             
            15
            SAT 144717.0 vs meanDiff 151161.137353045: 
            151161.137353045 2985.228450413262 3010.5688313578426 0.9915828594647507 -25.340380944580374 0.8417140535249334
            

            Updated notebook: afwVar-vs-astierCov00-DM-29272-2021APR01.pdf

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - The clipping algorithm in afwMATH.VARIANCECLIP already ignores the pixels in the mask planes. In addition to these masked pixels, we should also identify the pixels that have been clipped in the process. This is accomplished by: varClip = afwMath.makeStatistics(diffIm, afwMath.VARIANCECLIP, diffImStatsCtrl).getValue() meanClip = afwMath.makeStatistics(diffIm, afwMath.MEANCLIP, diffImStatsCtrl).getValue()   cut = meanClip + nSigmaClipPtc * np.sqrt(varClip) unmasked = np.where(np.fabs(diffIm.image.array) < = cut, 1 , 0 ) Then, the matrix `w` (defined as follows) should be passed to the Covariance calculations: wDiff = np.where(diffIm.getMask().getArray() = = 0 , 1 , 0 ) w = wDiff * unmasked Including these modifications, the fractional difference goes below {1%} for all amplifiers (except one; 0.226% for amp 6). I'll add a warning in the code for the case when the difference is larger than 1% : 0 SAT 142002.0 vs meanDiff 148049.49503693567: 148049.49503693567 4113.619098129532 4124.8528603178065 0.9972765665664475 -11.23376218827434 0.27234334335525245 1 SAT 144371.0 vs meanDiff 147845.09956447146: 147845.09956447146 4138.948451455616 4157.531618939209 0.9955302402514659 -18.58316748359266 0.4469759748534141 2 SAT 144955.0 vs meanDiff 146294.38210178568: 146294.38210178568 3637.4780519048513 3646.6700047809695 0.9974793570945364 -9.191952876118194 0.25206429054636104 3 SAT 145516.0 vs meanDiff 146373.49562630185: 146373.49562630185 3425.5526407749344 3431.180479595854 0.9983597951625143 -5.627838820919806 0.1640204837485726 4 SAT 146022.0 vs meanDiff 144879.95020856708: 144879.95020856708 3408.484298284876 3390.9913228028795 1.0051586612340657 17.492975481996382 0.515866123406572 5 SAT 145477.0 vs meanDiff 144201.99522342085: 144201.99522342085 3380.96070882588 3372.1831245363237 1.0026029382051318 8.777584289556216 0.26029382051317995 6 SAT 143174.0 vs meanDiff 144219.2800343227: 144219.2800343227 3271.670691511372 3264.264277571426 1.0022689382078636 7.406413939945651 0.22689382078635578 7 SAT 140048.0 vs meanDiff 143247.12360096083: 143247.12360096083 3736.8108764375993 3423.3841236906374 1.0915546551080768 313.42675274696194 9.15546551080768 8 SAT 141798.0 vs meanDiff 145785.97266838083: 145785.97266838083 1939.4518418819803 1935.6243057743025 1.0019774168449216 3.8275361076778154 0.19774168449215868 9 SAT 142988.0 vs meanDiff 145427.4797746908: 145427.4797746908 1977.1675410522487 1972.7115474516563 1.002258816605169 4.455993600592365 0.22588166051689118 10 SAT 143252.0 vs meanDiff 146687.08830933238: 146687.08830933238 2038.3095264059475 2034.7059315895585 1.001771064191853 3.603594816388977 0.17710641918529113 11 SAT 143296.0 vs meanDiff 145111.99779136135: 145111.99779136135 2360.4288614212046 2354.9684209445572 1.00231868946865 5.4604404766473635 0.23186894686499926 12 SAT 143316.0 vs meanDiff 146008.9060246319: 146008.9060246319 2819.666517175569 2813.5489876615857 1.0021743106449579 6.1175295139833 0.21743106449578775 13 SAT 143825.0 vs meanDiff 147121.24589549645: 147121.24589549645 3190.9693868805393 3162.304178877705 1.0090646586733498 28.665208002834333 0.9064658673349824 14 SAT 144660.0 vs meanDiff 146577.15637499627: 146577.15637499627 2768.8894534582087 2761.1743842910937 1.0027941260106596 7.715069167114962 0.27941260106596477 15 SAT 144717.0 vs meanDiff 151161.137353045: 151161.137353045 2985.228450413262 3010.5688313578426 0.9915828594647507 -25.340380944580374 0.8417140535249334 Updated notebook: afwVar-vs-astierCov00-DM-29272-2021APR01.pdf
            Hide
            kannawad Arun Kannawadi added a comment -

            It looks good overall. I just have some minor suggestions which I commented inline.

            Show
            kannawad Arun Kannawadi added a comment - It looks good overall. I just have some minor suggestions which I commented inline.
            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33959/pipeline

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              plazas Andrés Alejandro Plazas Malagón
              Reviewers:
              Arun Kannawadi
              Watchers:
              Andrés Alejandro Plazas Malagón, Arun Kannawadi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.