Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38486

Combined dark seems to not have exposure time

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      While working on DM-37684, expTime for a combined dark was found to be nan: 

      ipdb> exp.getInfo().getVisitInfo()
      VisitInfo(exposureId=0, exposureTime=nan, darkTime=nan, date=<invalid>, UT1=nan, ERA=nan rad, boresightRaDec=(nan, +nan), boresightAzAlt=(nan, +nan), boresightAirmass=nan, boresightRotAngle=nan rad, rotType=0, observatory=nanN, nanE  nan, weather=Weather(nan, nan, nan), instrumentLabel='', id=0, focusZ=nan, observationType='dark', scienceProgram='', observationReason='', object='', hasSimulatedContent=false)
      

      This can be found by runninf DM-37684 branch of cp_pipe with the following command:

      pipetask run -j 1 -d "detector IN (54) AND instrument='LSSTCam'" -i u/cslage/dark_20211117/20211117T173016Z,u/cslage/flat_20211117/20211118T005839Z,LSSTCam/raw/all -p $CP_PIPE_DIR/pipelines/findDefectsCombined.yaml --register-dataset-types -b /sdf/group/rubin/repo/main/butler.yaml -o u/plazas/DM-37684.2023MAR29.6
      

       

       

        Attachments

          Issue Links

            Activity

            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            The new pipeline in DM-37684 runs (although with type=STDEV so far, still need to try the new VALUE option), but the nan's below apear due to expTime being nan:

            (lsst-scipipe-5.1.0) [plazas@sdfrome002 cp_pipe]$ pipetask run -j 1 -d "detector IN (54) AND instrument='LSSTCam'" -i u/cslage/dark_20211117/20211117T173016Z,u/cslage/flat_20211117/20211118T005839Z,LSSTCam/raw/all -p $CP_PIPE_DIR/pipelines/findDefectsCombined.yaml --register-dataset-types -b /sdf/group/rubin/repo/main/butler.yaml -o u/plazas/DM-37684.2023MAR29.5
            lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 3 quanta for 3 tasks, graph ID: '1680134678.2149749-3883427'
            Quanta        Tasks        
            ------ --------------------
                 1   measureDefectsFlat
                 1   measureDefectsDark
                 1 mergeDefectsCombined
            lsst.measureDefectsDark INFO: Amp: C10. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C11. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C12. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C13. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C14. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C15. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C16. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C17. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C07. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C06. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C05. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C04. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C03. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C02. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C01. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Amp: C00. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan]
            lsst.measureDefectsDark INFO: Found 519 defects containing 1876 pixels in DARK
            lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'measureDefectsDark' on quantum {instrument: 'LSSTCam', detector: 54} took 2.834 seconds
            lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 1 quanta successfully, 0 failed and 2 remain out of total 3 quanta.
            py.warnings WARNING: /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/obs_base/g4f3ce6f906+9f9ad907f2/python/lsst/obs/base/formatters/fitsExposure.py:643: UserWarning: Reading file:///sdf/group/rubin/repo/main/u/cslage/flat_20211117/20211118T005839Z/flat/i~nd/SDSSi~ND_OD0.5/flat_LSSTCam_i~nd_SDSSi~ND_OD0_5_R13_S00_u_cslage_flat_20211117_20211118T005839Z.fits with data ID {instrument: 'LSSTCam', detector: 54, physical_filter: 'SDSSi~ND_OD0.5', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i~nd", physical="SDSSi~ND_OD0.5")).  This is probably a bug in the code that produced it.
              warnings.warn(
             
            lsst.measureDefectsFlat INFO: Amp: C10. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7854026000683105]
            lsst.measureDefectsFlat INFO: Amp: C11. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.78616105695217]
            lsst.measureDefectsFlat INFO: Amp: C12. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7848000877249174]
            lsst.measureDefectsFlat INFO: Amp: C13. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.787147634786462]
            lsst.measureDefectsFlat INFO: Amp: C14. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7863203841591773]
            lsst.measureDefectsFlat INFO: Amp: C15. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7865870251337173]
            lsst.measureDefectsFlat INFO: Amp: C16. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7858335747457372]
            lsst.measureDefectsFlat INFO: Amp: C17. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7836665587058274]
            lsst.measureDefectsFlat INFO: Amp: C07. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7810149580313949]
            lsst.measureDefectsFlat INFO: Amp: C06. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7826530038124825]
            lsst.measureDefectsFlat INFO: Amp: C05. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7824735076337688]
            lsst.measureDefectsFlat INFO: Amp: C04. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7830237667193115]
            lsst.measureDefectsFlat INFO: Amp: C03. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.781793648075024]
            lsst.measureDefectsFlat INFO: Amp: C02. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7815996285212605]
            lsst.measureDefectsFlat INFO: Amp: C01. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7836849865620805]
            lsst.measureDefectsFlat INFO: Amp: C00. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7826639161985226]
            lsst.measureDefectsFlat INFO: Found 792 defects containing 1003 pixels in FLAT
            lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'measureDefectsFlat' on quantum {instrument: 'LSSTCam', detector: 54} took 3.362 seconds
            lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 2 quanta successfully, 0 failed and 1 remain out of total 3 quanta.
            lsst.mergeDefectsCombined INFO: Pre-merge 1003 pixels with non-zero detections for FLAT
            lsst.mergeDefectsCombined INFO: Post-merge 1003 pixels marked as defects for FLAT
            lsst.mergeDefectsCombined INFO: Pre-merge 1876 pixels with non-zero detections for DARK
            lsst.mergeDefectsCombined INFO: Post-merge 1876 pixels marked as defects for DARK
            lsst.mergeDefectsCombined INFO: Pre-final merge 2798 pixels with non-zero detections
            lsst.mergeDefectsCombined INFO: Post-final merge 2798 pixels marked as defects
            lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'mergeDefectsCombined' on quantum {instrument: 'LSSTCam', detector: 54} took 1.465 seconds
            lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 3 quanta successfully, 0 failed and 0 remain out of total 3 quanta.
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - The new pipeline in DM-37684 runs (although with type=STDEV so far, still need to try the new VALUE option), but the nan's below apear due to expTime being nan: (lsst-scipipe-5.1.0) [plazas@sdfrome002 cp_pipe]$ pipetask run -j 1 -d "detector IN (54) AND instrument='LSSTCam'" -i u/cslage/dark_20211117/20211117T173016Z,u/cslage/flat_20211117/20211118T005839Z,LSSTCam/raw/all -p $CP_PIPE_DIR/pipelines/findDefectsCombined.yaml --register-dataset-types -b /sdf/group/rubin/repo/main/butler.yaml -o u/plazas/DM-37684.2023MAR29.5 lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 3 quanta for 3 tasks, graph ID: '1680134678.2149749-3883427' Quanta Tasks ------ -------------------- 1 measureDefectsFlat 1 measureDefectsDark 1 mergeDefectsCombined lsst.measureDefectsDark INFO: Amp: C10. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C11. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C12. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C13. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C14. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C15. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C16. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C17. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C07. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C06. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C05. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C04. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C03. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C02. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C01. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Amp: C00. Threshold Type: STDEV Sigma values and Pixel Values: [4.8], [nan] lsst.measureDefectsDark INFO: Found 519 defects containing 1876 pixels in DARK lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'measureDefectsDark' on quantum {instrument: 'LSSTCam', detector: 54} took 2.834 seconds lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 1 quanta successfully, 0 failed and 2 remain out of total 3 quanta. py.warnings WARNING: /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/obs_base/g4f3ce6f906+9f9ad907f2/python/lsst/obs/base/formatters/fitsExposure.py:643: UserWarning: Reading file:///sdf/group/rubin/repo/main/u/cslage/flat_20211117/20211118T005839Z/flat/i~nd/SDSSi~ND_OD0.5/flat_LSSTCam_i~nd_SDSSi~ND_OD0_5_R13_S00_u_cslage_flat_20211117_20211118T005839Z.fits with data ID {instrument: 'LSSTCam', detector: 54, physical_filter: 'SDSSi~ND_OD0.5', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i~nd", physical="SDSSi~ND_OD0.5")). This is probably a bug in the code that produced it. warnings.warn(   lsst.measureDefectsFlat INFO: Amp: C10. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7854026000683105] lsst.measureDefectsFlat INFO: Amp: C11. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.78616105695217] lsst.measureDefectsFlat INFO: Amp: C12. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7848000877249174] lsst.measureDefectsFlat INFO: Amp: C13. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.787147634786462] lsst.measureDefectsFlat INFO: Amp: C14. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7863203841591773] lsst.measureDefectsFlat INFO: Amp: C15. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7865870251337173] lsst.measureDefectsFlat INFO: Amp: C16. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7858335747457372] lsst.measureDefectsFlat INFO: Amp: C17. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7836665587058274] lsst.measureDefectsFlat INFO: Amp: C07. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7810149580313949] lsst.measureDefectsFlat INFO: Amp: C06. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7826530038124825] lsst.measureDefectsFlat INFO: Amp: C05. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7824735076337688] lsst.measureDefectsFlat INFO: Amp: C04. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7830237667193115] lsst.measureDefectsFlat INFO: Amp: C03. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.781793648075024] lsst.measureDefectsFlat INFO: Amp: C02. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7815996285212605] lsst.measureDefectsFlat INFO: Amp: C01. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7836849865620805] lsst.measureDefectsFlat INFO: Amp: C00. Threshold Type: STDEV Sigma values and Pixel Values: [4.8, -5.0], [nan, 0.7826639161985226] lsst.measureDefectsFlat INFO: Found 792 defects containing 1003 pixels in FLAT lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'measureDefectsFlat' on quantum {instrument: 'LSSTCam', detector: 54} took 3.362 seconds lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 2 quanta successfully, 0 failed and 1 remain out of total 3 quanta. lsst.mergeDefectsCombined INFO: Pre-merge 1003 pixels with non-zero detections for FLAT lsst.mergeDefectsCombined INFO: Post-merge 1003 pixels marked as defects for FLAT lsst.mergeDefectsCombined INFO: Pre-merge 1876 pixels with non-zero detections for DARK lsst.mergeDefectsCombined INFO: Post-merge 1876 pixels marked as defects for DARK lsst.mergeDefectsCombined INFO: Pre-final merge 2798 pixels with non-zero detections lsst.mergeDefectsCombined INFO: Post-final merge 2798 pixels marked as defects lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'mergeDefectsCombined' on quantum {instrument: 'LSSTCam', detector: 54} took 1.465 seconds lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 3 quanta successfully, 0 failed and 0 remain out of total 3 quanta.
            Show
            czw Christopher Waters added a comment - Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38367/pipeline

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              plazas Andrés Alejandro Plazas Malagón
              Reviewers:
              Tim Jenness
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.