Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23928

PropertySet not robust against duplicate and empty FITS cards

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Templates:
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      For the PFS project, we have issues writing certain postISRCCD exposures to file using the LSST stack v18.1.0 . For certain visits taken in 2019 on instrument hardware on a testbench, I get the following error:

        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py", line 1434, in runDataRef
          sensorRef.put(result.exposure, "postISRCCD")
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butlerSubset.py", line 224, in put
          self.butlerSubset.butler.put(obj, datasetType, self.dataId, doBackup=doBackup, **rest)
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py", line 1434, in put
          location.getRepository().write(location, obj)
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/repository.py", line 185, in write
          return butlerLocationStorage.write(butlerLocation, obj)
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/posixStorage.py", line 258, in write
          writeFormatter(butlerLocation, obj)
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/posixStorage.py", line 634, in writeFitsStorage
          obj.writeFitsWithOptions(logLoc.locString(), options=additionalData)
        File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/afw/18.1.0/python/lsst/afw/image/image/fitsIoWithOptions.py", line 127, in exposureWriteFitsWithOptions
          self.writeFits(dest, **writeOptionDict)
      lsst.pex.exceptions.wrappers.TypeError: 
        File "src/PropertySet.cc", line 232, in std::vector<T> lsst::daf::base::PropertySet::getArray(const string&) const [with T = std::basic_string<char>; std::string = std::basic_string<char>]
          W_XHP2FR {0}
      lsst::pex::exceptions::TypeError: 'W_XHP2FR'
      

      Now, I've found the error is raised when the keyword 'W_XHP2FR' appears twice in the FITS header, and has a value that is empty:

      W_XHP2FR=  / frac power to spreader heater: 0 or 1 NOT CURRENT
      [...]
      W_XHP2FR=  / frac power to shield: 0 or 1 NOT CURRENT
      

      The fact that the keyword is duplicate and empty are bugs in the PFS software that have subsequently been fixed. However it is not possible to process historical data that are 'corrupted' this way using v18.1.0 stack. It is also not possible to fix the metadata in the PFS layer before the IsrTask sees that, as there is no way to intercept the metadata when the IsrTask reads in the raw exposure.

      Is it possible to make PropertySet robust against such scenarios?

        Attachments

          Activity

            People

            • Assignee:
              ktl Kian-Tat Lim
              Reporter:
              hassans Hassan Siddiqui
              Reviewers:
              Tim Jenness
              Watchers:
              Hassan Siddiqui, John Swinbank, Kian-Tat Lim, Paul Price, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel