Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-37644

ci_cpp_gen3 ptc seems to be returning almost entirely all NaN values

    XMLWordPrintable

Details

    • Improvement
    • Status: Invalid
    • Resolution: Done
    • None
    • ci_cpp
    • None

    Description

      The PTC step appears to be generating little usable results, with many entries being NaN, despite the images looking reasonable:

      (Pdb++) self.inputExpIdPairs[ampName]
      [(2021052500077, 2021052500078)]
      (Pdb++) self.expIdMask[ampName]
      [True]
      (Pdb++) self.rawExpTimes[ampName]
      [0.2]
      (Pdb++) self.rawMeans[ampName]
      [4924.4157451060455]
      (Pdb++) self.rawVars
      {'C10': [4804.287179942229], 'C11': [4915.693658767317], 'C12': [4853.79077967877], 'C13': [4834.87824835332], 'C14': [4934.428637557369], 'C15': [4973.902904489969], 'C16': [4935.354497102363], 'C17': [5220.200201536287], 'C07': [nan], 'C06': [4647.658652756437], 'C05': [4696.717638139308], 'C04': [4656.033331091465], 'C03': [4620.489260915196], 'C02': [4693.405212676453], 'C01': [4677.0195909135755], 'C00': [4664.458146511158]}
      (Pdb++) self.gain
      {'C10': 1.0450390914170966, 'C11': 0.009923194956800442, 'C12': 1.0403032526907565, 'C13': 1.0426238566994317, 'C14': 1.0264718756797735, 'C15': 1.0280014594413431, 'C16': 1.0356306564073268, 'C17': 1.0078169853256589, 'C07': 1.0573568259010764, 'C06': 1.058426648373386, 'C05': 1.041615233633119, 'C04': 0.2322371541220943, 'C03': 1.0630493760787727, 'C02': 1.0532586276631128, 'C01': 1.053884652198826, 'C00': 1.0550876439578698}
      (Pdb++) self.gainErr
      {'C10': nan, 'C11': nan, 'C12': nan, 'C13': nan, 'C14': nan, 'C15': nan, 'C16': nan, 'C17': nan, 'C07': nan, 'C06': nan, 'C05': nan, 'C04': nan, 'C03': nan, 'C02': nan, 'C01': nan, 'C00': nan}
      (Pdb++) self.noise
      {'C10': None, 'C11': None, 'C12': None, 'C13': None, 'C14': None, 'C15': None, 'C16': None, 'C17': None, 'C07': None, 'C06': None, 'C05': None, 'C04': None, 'C03': None, 'C02': None, 'C01': None, 'C00': None}
      (Pdb++) self.noiseErr
      {'C10': nan, 'C11': nan, 'C12': nan, 'C13': nan, 'C14': nan, 'C15': nan, 'C16': nan, 'C17': nan, 'C07': nan, 'C06': nan, 'C05': nan, 'C04': nan, 'C03': nan, 'C02': nan, 'C01': nan, 'C00': nan}
      (Pdb++) self.ptcFitPars
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      (Pdb++) self.ptcFitParsError
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      (Pdb++) self.ptcFitChiSq
      {'C10': nan, 'C11': nan, 'C12': nan, 'C13': nan, 'C14': nan, 'C15': nan, 'C16': nan, 'C17': nan, 'C07': nan, 'C06': nan, 'C05': nan, 'C04': nan, 'C03': nan, 'C02': nan, 'C01': nan, 'C00': nan}
      (Pdb++) self.ptcTurnoff
      {'C10': nan, 'C11': nan, 'C12': nan, 'C13': nan, 'C14': nan, 'C15': nan, 'C16': nan, 'C17': nan, 'C07': nan, 'C06': nan, 'C05': nan, 'C04': nan, 'C03': nan, 'C02': nan, 'C01': nan, 'C00': nan}
      (Pdb++) self.covariances
      {'C10': array([[[ 4.80428246e+03, 9.31569328e+00, 5.27296875e+00,
       -1.41296388e+01, -7.46880188e+00, 6.63498961e-02,
       -2.27999811e-01, 8.32039848e+00],}}
      [...]
      (Pdb++) self.finalVars
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      (Pdb++) self.finalModelVars
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      (Pdb++) nPadPoints
      {'C10': 0, 'C11': 0, 'C12': 0, 'C13': 0, 'C14': 0, 'C15': 0, 'C16': 0, 'C17': 0, 'C07': 0, 'C06': 0, 'C05': 0, 'C04': 0, 'C03': 0, 'C02': 0, 'C01': 0, 'C00': 0}
      (Pdb++) self.finalMeans
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      (Pdb++) self.badAmps
      [nan]
      (Pdb++) self.photoCharge
      {'C10': [nan], 'C11': [nan], 'C12': [nan], 'C13': [nan], 'C14': [nan], 'C15': [nan], 'C16': [nan], 'C17': [nan], 'C07': [nan], 'C06': [nan], 'C05': [nan], 'C04': [nan], 'C03': [nan], 'C02': [nan], 'C01': [nan], 'C00': [nan]}
      

      Attachments

        Activity

          The example seems to show an intermediate partial PTC dataset, from cpPtcExtract , with a single flat pair, so, in that case, most of the entries are expected to be filled with nan’s (except for the means, the vars, the covariances, the expTimes, which in this case they are not.)

          Also, from Chris, this is a recent ci_cpp run, which looks fine except for C07, which is BAD:

          # ftool -T ci_cpp_ptc/20230428T181814Z/ptc/ptc_LATISS_RXX_S00_ci_cpp_ptc_20230428T181814Z.fits  -x 1 -c AMPLIFIER_NAME,COVARIANCES,BAD_AMPS
          ##AMPLIFIER_NAME        BAD_AMPS        COVARIANCES
          C10     C07     [4.80598670e+03 1.10211649e+01 6.97612526e+00 ... 1.73682862e+04
           1.60145869e+04 1.46038125e+04]
          C11     C07     [4.91109807e+03 4.25096389e+01 1.79788294e+01 ... 2.14972432e+04
           1.97942662e+04 1.80426345e+04]
          C12     C07     [4.85547487e+03 5.85349388e+00 5.82267787e+00 ... 2.03550929e+04
           1.87439152e+04 1.70499764e+04]
          C13     C07     [4.83705583e+03 1.20644313e+01 1.28739936e+01 ... 2.91589375e+04
           2.68124524e+04 2.44132762e+04]
          C14     C07     [ 4.93656178e+03  1.99087852e-01 -6.36413530e+00 ...  2.57860297e+04
            2.37165786e+04  2.16183521e+04]
          C15     C07     [4.97595508e+03 1.58937842e+01 3.90988008e+00 ... 2.68221585e+04
           2.46657334e+04 2.24726341e+04]
          C16     C07     [4.93735948e+03 1.17941423e+01 3.55293893e+00 ... 3.30295203e+04
           3.04054685e+04 2.76793669e+04]
          C17     C07     [5.22172100e+03 8.07167823e+00 2.31194482e+00 ... 3.33054006e+04
           3.06234399e+04 2.78684839e+04]
          C07     C07     [nan nan nan ... nan nan nan]
          C06     C07     [4.64988663e+03 7.09312999e+00 2.38197207e+00 ... 1.19687557e+04
           1.09673214e+04 9.98740767e+03]
          C05     C07     [4699.42959048   12.38999652   10.68713951 ... 9545.67094099 8737.8297648
           7946.79567312]
          C04     C07     [4.65785811e+03 1.21301478e+01 4.85162859e+00 ... 6.65273805e+03
           6.07159374e+03 5.53045866e+03]
          C03     C07     [4.62312155e+03 1.33230675e+01 2.91784979e+00 ... 6.02761463e+03
           5.51978988e+03 5.02001392e+03]
          C02     C07     [4695.60856073    9.26658062    6.41362007 ... 5611.40960517 5132.45566749
           4668.98891166]
          C01     C07     [4.67969713e+03 2.53149337e+00 5.17346539e+00 ... 4.52998349e+03
           4.14881759e+03 3.77469483e+03]
          C00     C07     [4.66671145e+03 9.55170673e+00 2.95277638e+00 ... 2.36465573e+02
           2.29158893e+02 2.10280917e+02]
          

          plazas Andrés Alejandro Plazas Malagón added a comment - The example seems to show an intermediate partial PTC dataset, from cpPtcExtract , with a single flat pair, so, in that case, most of the entries are expected to be filled with nan’s (except for the means, the vars, the covariances, the expTimes, which in this case they are not.) Also, from Chris, this is a recent ci_cpp run, which looks fine except for C07 , which is BAD: # ftool -T ci_cpp_ptc/20230428T181814Z/ptc/ptc_LATISS_RXX_S00_ci_cpp_ptc_20230428T181814Z.fits -x 1 -c AMPLIFIER_NAME,COVARIANCES,BAD_AMPS ##AMPLIFIER_NAME BAD_AMPS COVARIANCES C10 C07 [4.80598670e+03 1.10211649e+01 6.97612526e+00 ... 1.73682862e+04 1.60145869e+04 1.46038125e+04] C11 C07 [4.91109807e+03 4.25096389e+01 1.79788294e+01 ... 2.14972432e+04 1.97942662e+04 1.80426345e+04] C12 C07 [4.85547487e+03 5.85349388e+00 5.82267787e+00 ... 2.03550929e+04 1.87439152e+04 1.70499764e+04] C13 C07 [4.83705583e+03 1.20644313e+01 1.28739936e+01 ... 2.91589375e+04 2.68124524e+04 2.44132762e+04] C14 C07 [ 4.93656178e+03 1.99087852e-01 -6.36413530e+00 ... 2.57860297e+04 2.37165786e+04 2.16183521e+04] C15 C07 [4.97595508e+03 1.58937842e+01 3.90988008e+00 ... 2.68221585e+04 2.46657334e+04 2.24726341e+04] C16 C07 [4.93735948e+03 1.17941423e+01 3.55293893e+00 ... 3.30295203e+04 3.04054685e+04 2.76793669e+04] C17 C07 [5.22172100e+03 8.07167823e+00 2.31194482e+00 ... 3.33054006e+04 3.06234399e+04 2.78684839e+04] C07 C07 [nan nan nan ... nan nan nan] C06 C07 [4.64988663e+03 7.09312999e+00 2.38197207e+00 ... 1.19687557e+04 1.09673214e+04 9.98740767e+03] C05 C07 [4699.42959048 12.38999652 10.68713951 ... 9545.67094099 8737.8297648 7946.79567312] C04 C07 [4.65785811e+03 1.21301478e+01 4.85162859e+00 ... 6.65273805e+03 6.07159374e+03 5.53045866e+03] C03 C07 [4.62312155e+03 1.33230675e+01 2.91784979e+00 ... 6.02761463e+03 5.51978988e+03 5.02001392e+03] C02 C07 [4695.60856073 9.26658062 6.41362007 ... 5611.40960517 5132.45566749 4668.98891166] C01 C07 [4.67969713e+03 2.53149337e+00 5.17346539e+00 ... 4.52998349e+03 4.14881759e+03 3.77469483e+03] C00 C07 [4.66671145e+03 9.55170673e+00 2.95277638e+00 ... 2.36465573e+02 2.29158893e+02 2.10280917e+02]

          People

            plazas Andrés Alejandro Plazas Malagón
            czw Christopher Waters
            Andrés Alejandro Plazas Malagón, Christopher Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.