Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36277

Reading calibrations produced at NCSA gives a PROGRAM header error

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_base
    • Labels:
      None
    • Story Points:
      1.5
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      When trying to access calibrations produced at NCSA at USDF, we are getting a PROGRAM header error. From a notebook, with this query:

      butler = Butler('/repo/main', collections=["LSSTComCam/raw/all", "LSSTComCam/calib", 
                                                           "u/cslage/comcam/calib_20220218"])
      bias = butler.get('bias', detector=0, exposure=2021093000079, instrument="LSSTComCam")
      

      We are getting this error stream:

      timer.lsst.daf.butler.datastores.fileDatastore ERROR: Reading from location file:///sdf/group/rubin/repo/main/u/cslage/comcam/bias_20220218/20220221T102105Z/bias/bias_LSSTComCam_R22_S00_u_cslage_comcam_bias_20220218_20220221T102105Z.fits  with formatter lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter: Took 0.0022 seconds
      ---------------------------------------------------------------------------
      TypeError                                 Traceback (most recent call last)
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g4ca570b7d7+17c7ad0751/python/lsst/daf/butler/datastores/fileDatastore.py:1341, in FileDatastore._read_artifact_into_memory(self, getInfo, ref, isComponent, cache_ref)
         1331         with time_this(
         1332             log,
         1333             msg="Reading%s from location %s %s with formatter %s",
         (...)
         1339             ),
         1340         ):
      -> 1341             result = formatter.read(component=getInfo.component if isComponent else None)
         1342 except Exception as e:
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/obs_base/g188f660963+c805e22280/python/lsst/obs/base/formatters/fitsExposure.py:102, in FitsImageFormatterBase.read(self, component)
           96         raise ValueError(
           97             "Storage class inconsistency ({} vs {}) but no"
           98             " component requested".format(
           99                 self.fileDescriptor.readStorageClass.name, self.fileDescriptor.storageClass.name
          100             )
          101         )
      --> 102 return self.readFull()
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/obs_base/g188f660963+c805e22280/python/lsst/obs/base/formatters/fitsExposure.py:566, in FitsExposureFormatter.readFull(self)
          562 def readFull(self):
          563     # Docstring inherited.
          564     amplifier, detector, _ = standardizeAmplifierParameters(
          565         self.checked_parameters,
      --> 566         self.reader.readDetector(),
          567     )
          568     if amplifier is not None:
       
      TypeError: 
        File "src/PropertySet.cc", line 208, in T lsst::daf::base::PropertySet::get(const string&) const [with T = std::__cxx11::basic_string<char>; std::string = std::__cxx11::basic_string<char>]
          PROGRAM {0}
      lsst::pex::exceptions::TypeError: 'PROGRAM'
       
       
      The above exception was the direct cause of the following exception:
       
      ValueError                                Traceback (most recent call last)
      Input In [9], in <cell line: 3>()
            1 butler = Butler('/repo/main', collections=["LSSTComCam/raw/all", "LSSTComCam/calib", 
            2                                                      "u/cslage/comcam/calib_20220218"])
      ----> 3 bias = butler.get('bias', detector=0, exposure=2021093000079, instrument="LSSTComCam")
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g4ca570b7d7+17c7ad0751/python/lsst/daf/butler/_butler.py:1388, in Butler.get(self, datasetRefOrType, dataId, parameters, collections, **kwargs)
         1386 log.debug("Butler get: %s, dataId=%s, parameters=%s", datasetRefOrType, dataId, parameters)
         1387 ref = self._findDatasetRef(datasetRefOrType, dataId, collections=collections, **kwargs)
      -> 1388 return self.getDirect(ref, parameters=parameters)
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g4ca570b7d7+17c7ad0751/python/lsst/daf/butler/_butler.py:1246, in Butler.getDirect(self, ref, parameters)
         1226 def getDirect(self, ref: DatasetRef, *, parameters: Optional[Dict[str, Any]] = None) -> Any:
         1227     """Retrieve a stored dataset.
         1228 
         1229     Unlike `Butler.get`, this method allows datasets outside the Butler's
         (...)
         1244         The dataset.
         1245     """
      -> 1246     return self.datastore.get(ref, parameters=parameters)
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g4ca570b7d7+17c7ad0751/python/lsst/daf/butler/datastores/fileDatastore.py:2133, in FileDatastore.get(self, ref, parameters)
         2126 else:
         2127     # For an assembled composite this could be a derived
         2128     # component derived from a real component. The validity
         2129     # of the parameters is not clear. For now validate against
         2130     # the composite storage class
         2131     getInfo.formatter.fileDescriptor.storageClass.validateParameters(parameters)
      -> 2133 return self._read_artifact_into_memory(getInfo, ref, isComponent=isComponent, cache_ref=cache_ref)
       
      File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g4ca570b7d7+17c7ad0751/python/lsst/daf/butler/datastores/fileDatastore.py:1343, in FileDatastore._read_artifact_into_memory(self, getInfo, ref, isComponent, cache_ref)
         1341             result = formatter.read(component=getInfo.component if isComponent else None)
         1342 except Exception as e:
      -> 1343     raise ValueError(
         1344         f"Failure from formatter '{formatter.name()}' for dataset {ref.id}"
         1345         f" ({ref.datasetType.name} from {uri}): {e}"
         1346     ) from e
         1348 # File was read successfully so can move to cache
         1349 if can_be_cached:
       
      ValueError: Failure from formatter 'lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter' for dataset 202535f2-e947-4abb-b146-db8590e3ec6a (bias from file:///sdf/group/rubin/repo/main/u/cslage/comcam/bias_20220218/20220221T102105Z/bias/bias_LSSTComCam_R22_S00_u_cslage_comcam_bias_20220218_20220221T102105Z.fits): 
        File "src/PropertySet.cc", line 208, in T lsst::daf::base::PropertySet::get(const string&) const [with T = std::__cxx11::basic_string<char>; std::string = std::__cxx11::basic_string<char>]
          PROGRAM {0}
      lsst::pex::exceptions::TypeError: 'PROGRAM'
      

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            This can be triggered even more simply:

            >>> ExposureF("bias_LSSTComCam_R22_S00_u_cslage_comcam_bias_20220218_20220221T102105Z.fits")
            Traceback (most recent call last):
              File "<stdin>", line 1, in <module>
            lsst.pex.exceptions.wrappers.TypeError: 
              File "src/PropertySet.cc", line 208, in T lsst::daf::base::PropertySet::get(const std::string &) const [T = std::string]
                PROGRAM {0}
            lsst::pex::exceptions::TypeError: 'PROGRAM'
            

            Show
            tjenness Tim Jenness added a comment - This can be triggered even more simply: >>> ExposureF( "bias_LSSTComCam_R22_S00_u_cslage_comcam_bias_20220218_20220221T102105Z.fits" ) Traceback (most recent call last): File "<stdin>" , line 1 , in <module> lsst.pex.exceptions.wrappers.TypeError: File "src/PropertySet.cc" , line 208 , in T lsst::daf::base::PropertySet::get(const std::string &) const [T = std::string] PROGRAM { 0 } lsst::pex::exceptions::TypeError: 'PROGRAM'
            Hide
            Parejkoj John Parejko added a comment -

            Echoing my comment on the slack thread: we should definitely catch empty fields like this when reading FITS header values into our own datatypes, and we could perhaps switch to using HIERARCH keys (though does that further reduce the number of characters that can be in that field?). Normally this isn't a problem because we generate the VisitInfo when reading the raw, and serialize it on its own to the post-raw products, but it looks like we treat calibrations differently?

            Show
            Parejkoj John Parejko added a comment - Echoing my comment on the slack thread: we should definitely catch empty fields like this when reading FITS header values into our own datatypes, and we could perhaps switch to using HIERARCH keys (though does that further reduce the number of characters that can be in that field?). Normally this isn't a problem because we generate the VisitInfo when reading the raw, and serialize it on its own to the post-raw products, but it looks like we treat calibrations differently?
            Hide
            tjenness Tim Jenness added a comment -

            I have a fix. I'm trying to work out how to test it.

            Show
            tjenness Tim Jenness added a comment - I have a fix. I'm trying to work out how to test it.
            Hide
            tjenness Tim Jenness added a comment -

            John Parejko small review. I added defined-ness checks around all the PropertySet gets and added tests for keyword values that are undefined.

            Show
            tjenness Tim Jenness added a comment - John Parejko small review. I added defined-ness checks around all the PropertySet gets and added tests for keyword values that are undefined.
            Show
            tjenness Tim Jenness added a comment - PR: https://github.com/lsst/afw/pull/658 Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37307/pipeline/
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the quick fix: as I said on that PR, please file a ticket if you want us to switch to HIERARCH for anything in afw that uses header keys. I don't know how straightforward of a change that would actually be, but we're probably better off doing it sooner than later.

            Show
            Parejkoj John Parejko added a comment - Thanks for the quick fix: as I said on that PR, please file a ticket if you want us to switch to HIERARCH for anything in afw that uses header keys. I don't know how straightforward of a change that would actually be, but we're probably better off doing it sooner than later.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              cslage Craig Lage
              Reviewers:
              John Parejko
              Watchers:
              Andrés Alejandro Plazas Malagón, Craig Lage, John Parejko, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.