Details
-
Type:
Improvement
-
Status: In Progress
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: ip_isr
-
Labels:None
-
Epic Link:
-
Team:Data Release Production
Description
A couple months ago it was noticed that the variance planes in calexps were 10-20% lower than empirical for HSC, and 10% higher than empirical for DECam. Brighter fatter is the most likely source, so the suggestion is to run ScaleVarianceTask on the images right before BF and right after OR on calexps with isr.BF turned on or off to confirm this is the cause of the differences.
The prior analysis can be found in https://github.com/lsst-dm/diffimTests/tree/master/tickets/DM-22396_ScaleVariance_debug
Attachments
Issue Links
Activity
HSC: Linearity data for detector 73:
(at tiger2-sumire)
ls /projects/HSC/HSC-Public/Linearity/2018-04-23/*[24680]00.fits
|
Create gen3 repo (in /project/plazas/DM-29695-missing-variance-BFE/):
butler create HSCGen3Test2
|
butler register-instrument ./HSCGen3Test2 lsst.obs.subaru.HyperSuprimeCam
|
butler write-curated-calibrations ./HSCGen3Test2 lsst.obs.subaru.HyperSuprimeCam
|
butler ingest-raws ./HSCGen3Test2 data/HSC/flats2/*.fits --transfer link
|
Polynomial (deg 3) fit:
Note: expId1 and inputDims (for HSC) differ, so expId1 = (expId1 - detNum)/200 shoudl be set in cpPtcExtract
expId1: 29383273
|
np.array(inputDims): [146916 146920 146924 146928 146932 146936 146940 146952 146956 146960
|
146964 146968 146972 146976 147000 147002 147004 147010 147014 147018
|
147022 147026 147030 147034 147038 147042 147046 147050 147054 147058
|
147062 147064]
|
pipetask run -j 9 -d "detector IN (73) AND instrument='HSC' AND exposure IN (146916,146920,146924,146928,146932,146936,146940,146952,146956,146960,146964,146968,146972,146976,147000,147002,147004,147010,147014,147018,147022,147026,147030,147034,147038,147042,147046,147050,147054,147058,147062,147064)" -b /project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2 -i HSC/raw/all,HSC/calib -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=POLYNOMIAL -c isr:doLinearize=False -c isr:doCrosstalk=False -c isr:doDefect=False -c isr:doBias=False -c isr:doDark=False -c isr:doFlat=False -c ptcSolve:initialNonLinearityExclusionThresholdPositive=0.1 -c ptcSolve:initialNonLinearityExclusionThresholdNegative=0.1 -c ptcSolve:minVarPivotSearch=30000 -c ptcSolve:minMeanRatioTest=1000 -c ptcSolve:sigmaCutPtcOutliers=10 -o ptc_2021MAY04.PRUEBA.2 --register-dataset-types
|
plotPhotonTransferCurve.py /project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2/ptc_2021MAY04.PRUEBA.2/20210506T194613Z/ptc/ptc_HSC_0_00_ptc_2021MAY04_PRUEBA_2_20210506T194613Z.fits --detNum=73 --outDir=/project/plazas/DM-29695-missing-variance-BFE/plots/HSC
|
Gains:
from lsst.ip.isr import PhotonTransferCurveDataset |
filename="/project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2/ptc_2021MAY04.PRUEBA.2/20210506T194613Z/ptc/ptc_HSC_0_00_ptc_2021MAY04_PRUEBA_2_20210506T194613Z.fits" |
fromFits = PhotonTransferCurveDataset.readFits(filename) |
|
|
In [5]: fromFits.gain |
Out[5]: |
{'0': 3.8659529635915617, |
'1': 3.6346835961670223, |
'2': 3.547171434932095, |
'3': 3.54736331982458} |
|
|
In [6]: fromFits.noise |
Out[6]: |
{'0': 5.567376066742054, |
'1': 5.129006317833421, |
'2': 4.981057943709491, |
'3': 4.957983067826489} |
(from: HSC_PTC_det73.pdf
)
Exponential approximation model:
pipetask run -j 9 -d "detector IN (73) AND instrument='HSC' AND exposure IN (146916,146920,146924,146928,146932,146936,146940,146952,146956,146960,146964,146968,146972,146976,147000,147002,147004,147010,147014,147018,147022,147026,147030,147034,147038,147042,147046,147050,147054,147058,147062,147064)" -b /project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2 -i HSC/raw/all,HSC/calib -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c isr:doLinearize=False -c isr:doCrosstalk=False -c isr:doDefect=False -c isr:doBias=False -c isr:doDark=False -c isr:doFlat=False -c ptcSolve:initialNonLinearityExclusionThresholdPositive=0.1 -c ptcSolve:initialNonLinearityExclusionThresholdNegative=0.1 -c ptcSolve:minVarPivotSearch=30000 -c ptcSolve:minMeanRatioTest=1000 -c ptcSolve:sigmaCutPtcOutliers=10 -o ptc_2021MAY06.EXPAPPROXIMATION --register-dataset-types
|
In [1]: from lsst.ip.isr import PhotonTransferCurveDataset |
|
|
In [2]: filename="/project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2/ptc_2021MAY06.EXPAPPROXIMATION/20210506T205523Z/ptc/ptc_HSC_0_00_ptc_2021MAY06_EXPAPPRO |
...: XIMATION_20210506T205523Z.fits"
|
|
|
In [3]: fromFits = PhotonTransferCurveDataset.readFits(filename) |
|
|
In [4]: fromFits.gain |
Out[4]: |
{'0': 3.7942044770779497, |
'1': 3.644914978067487, |
'2': 3.55430931333964, |
'3': 3.5425723722236246} |
|
|
In [5]: fromFits.noise |
Out[5]: |
{'0': 5.144589314594767, |
'1': 5.193140499919786, |
'2': 5.028407308359781, |
'3': 4.9258378112961445} |
plotPhotonTransferCurve.py "/project/plazas/DM-29695-missing-variance-BFE/HSCGen3Test2/ptc_2021MAY06.EXPAPPROXIMATION/20210506T205523Z/ptc/ptc_HSC_0_00_ptc_2021MAY06_EXPAPPRO --detNum=73 --outDir=/project/plazas/DM-29695-missing-variance-BFE/plots/HSC
|
(from: HSC_EXPAPPROXIMATION_PTC_det73.pdf )
This shows that the gain might be a bit overestimated in HSC and might help understand what ScaleVarianceTask reports (for DECam, it's still unclear). These are the gains in detector 73 (thanks to Chris, using ftool ~/tmp/trash/obs_subaru/hsc/camera/0_00.fits -T -A -c gain)
##gain
|
4.2097
|
4.054
|
3.9717
|
3.9878
|
Aside: the work on making the PTC task function with DEcam data and HSC data that was developed in this ticket was moved to another ticket, DM-30130, to separate things.
Chris checked the math used in Bernstein+17 ("Instrumental response model and detrending for the Dark Energy Camera") and confirmed that their formula for the variance and what's in our ISR code coincide. Therefore, we seem to agree on the methods.
We discussed with Chris checking the values after the background subtraction that is performed by a context manager in the scale variance factor task before calculating the pixel and image scale factors. This distribution should be centered around zero with a standard deviation close to 1 if we understand the noise distribution. Chris calculated this distribution for DEcam images, and we found that it is indeed centered around zero but with a sigma of ~0.82, which means that the variance plane will be higher by a factor of ~ 0.82^2 = 0.679, which is consistent with the scale factor that we have been finding for DECam images.
Another check, suggested by Chris, is to look at the scale factor reported by the task after making the variance plane, before making the flat, and after making the flat, and do this for the whole focal plane for a range of DECam exposures. If all the values at each step are the same for a detector, then it's not a feature dependent on the sky, sources, etc.
Command:
nohup pipetask run -j 9 -d "detector IN (1..62) AND exposure IN (288976,289016,289409,289493,289614,289820,289871) AND instrument='DECam'" -b /project/mrawls/hits2014-3/butler.yaml -i DECam/raw/all,DECam/calib,DECam/calib/DM-26971 -p /home/plazas/lsst_devel/LSST/obs_decam/pipelines/DRP.yaml#isr -c isr:doBrighterFatter=False -c isr:doLinearize=False -c isr:connections.bias='bias' -c isr:biasDataProductName='bias' -c isr:connections.flat='flat' -c isr:flatDataProductName='flat' -o u/andres/DM-29695/postISR.decam.2021MAY13.6 --register-dataset-types 2>&1 | tee log.DM-29695.2021MAY13.6
|
Full output in /home/plazas/lsst_devel/LSST/ip_isr/log.DM-29695.2021MAY13.6
For one detector, the scale factors seem to be consistent across different exposures:
isr INFO: DM-29695: before var map scaleFactors 57.96983820109894, 58.452972001601424 28987161 N30 61 A
|
isr INFO: DM-29695: after var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 A
|
isr INFO: DM-29695: before var map scaleFactors 3.9956834833591035, 6.610897797054294 28987161 N30 61 B
|
isr INFO: DM-29695: after var map scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61 B
|
isr INFO: DM-29695: before flat scaleFactors 0.7007764489930125, 0.7000842876415234 28987161 N30 61
|
isr INFO: DM-29695: after flat scaleFactors 0.681931749483376, 0.6841051717491675 28987161 N30 61)
|
Picking one exposure, 289614, the results seem to also be consistent across different detectors. For example,
isr INFO: DM-29695: before var map scaleFactors 50.29182536205966, 50.7197337712512 28961411 S23 11 A
|
isr INFO: DM-29695: after var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 A
|
isr INFO: DM-29695: before var map scaleFactors 3.339859758058494, 0.7119860497296152 28961411 S23 11 B
|
isr INFO: DM-29695: after var map scaleFactors 0.6629060554064795, 0.6629273259263699 28961411 S23 11 B
|
isr INFO: DM-29695: before flat scaleFactors 0.6626422372744426, 0.6626858302814864 28961411 S23 11
|
isr INFO: DM-29695: after flat scaleFactors 0.6357515313246155, 0.6352227458010834 28961411 S23 11
|
isr INFO: DM-29695: before var map scaleFactors 54.73647988872145, 55.15204801863784 28961452 N21 52 A
|
isr INFO: DM-29695: after var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 A
|
isr INFO: DM-29695: before var map scaleFactors 3.4847922766913997, 0.734293353779192 28961452 N21 52 B
|
isr INFO: DM-29695: after var map scaleFactors 0.6934252927812252, 0.6933824287273015 28961452 N21 52 B
|
isr INFO: DM-29695: before flat scaleFactors 0.6931716272864122, 0.6931485679424054 28961452 N21 52
|
isr INFO: DM-29695: after flat scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52
|
isr INFO: DM-29695: scaleFactors 0.6718722785740711, 0.6717430952072597 28961452 N21 52
|
isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A
|
isr INFO: DM-29695: after var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 A
|
isr INFO: DM-29695: before var map scaleFactors 43.68462012015279, 45.55426321422043 28961431 S7 31 B
|
isr INFO: DM-29695: after var map scaleFactors 0.4203686394123188, 0.4204171251468196 28961431 S7 31 B
|
isr INFO: DM-29695: before flat scaleFactors 0.4203600888247675, 0.4204082965977526 28961431 S7 31
|
isr INFO: DM-29695: after flat scaleFactors 0.40519949367064567, 0.4051799370697797 28961431 S7 31
|
Tunning, some parameters of the task to include more points:
pipetask run -j 9 -d "detector IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30) AND instrument='DECam' AND exposure IN (153088,153089,153090,153091,153092,153095,153096,153097,153098,153099,153100,153101,153102,153103,153104,153105,153106,153107,153108,153109,153110,153111,153112,153115,153116,153117,153118,153119,153120,153121,153122,153123,153124,153125,153126,153127,153128,153129,153130,153131,153132,153133,153134,153135,153136,153030,153035,153039,153040,153079,153080,153081,153082,153085,153086,153087)" -b /project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2 -i DECam/raw/all,DECam/calib -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=POLYNOMIAL -c isr:doLinearize=False -c isr:doCrosstalk=False -c isr:doDefect=False -c isr:doBias=False -c isr:doDark=False -c isr:doFlat=False -c ptcSolve:initialNonLinearityExclusionThresholdPositive=0.1 -c ptcSolve:initialNonLinearityExclusionThresholdNegative=0.1 -c ptcSolve:minVarPivotSearch=30000 -c ptcSolve:minMeanRatioTest=1000 -c ptcSolve:sigmaCutPtcOutliers=10 -o ptc_2021MAY02.1 --register-dataset-types/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S15_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.7691257061912116, 'B': 4.379643086796402}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S12_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.8853582259221846, 'B': 4.036745233708877}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S2_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.708636707802179, 'B': 4.232237727675842}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S1_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.110615662456239, 'B': 4.231633914126678}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S22_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.8712298424856035, 'B': 3.879834161178448}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S24_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.9117313074935613, 'B': 4.323051613806082}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S18_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.8222934245058218, 'B': 4.255891634025443}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S3_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.968435572369203, 'B': 4.844259856149896}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S9_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.948265923301593, 'B': 4.691855851771984}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S6_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.078603484276802, 'B': 4.017349300843371}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S16_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.7860595049020467, 'B': 4.288615818398926}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S27_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.091113792966569, 'B': 3.8424935670053855}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S23_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.329057241294821, 'B': 4.351307410574284}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S20_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.9835806393154787, 'B': 3.954467513965731}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S28_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.504289676810856, 'B': 5.073777576201708}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S8_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.5023768222396345, 'B': 4.78126148258539}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S26_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 7.861917457134321, 'B': 2.5233502681504154}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S13_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.7634589508573493, 'B': 4.050310801663693}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S10_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.189219804733647, 'B': 4.267447661493404}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S29_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.70280385658764, 'B': 3.883326985692564}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S17_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.7289241076969395, 'B': 4.16530190782093}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S14_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.110853521280339, 'B': 3.9771693150586387}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S5_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.4097033880917476, 'B': 4.28367590268997}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S25_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.458610935902888, 'B': 4.680367428420633}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S21_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.9305194052354597, 'B': 4.210282897540109}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S19_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.020262405997158, 'B': 3.927274153011615}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S31_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.605842308779766, 'B': 3.9892267421284355}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S4_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 3.8015382611784374, 'B': 4.867303448274548}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S30_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.909222097138484, 'B': 4.4677547172303065}/project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY02.1/20210502T013357Z/ptc/ptc_DECam_S11_ptc_2021MAY02_1_20210502T013357Z.fitsGain:{'A': 4.339995904816039, 'B': 4.320460204120186}S29 PTC_detS29.pdf
plotPhotonTransferCurve.py /project/plazas/DM-29695-missing-variance-BFE/DECamGen3Test2/ptc_2021MAY01.6/20210502T012402Z/ptc/ptc_DECam_S29_ptc_2021MAY01_6_20210502T012402Z.fits --detNum=1 --outDir=/project/plazas/DM-29695-missing-variance-BFE