Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21048

butler.get('raw',...) raises a lsst::pex::exceptions::NotFoundError for BOT data at lsst-dev

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      For the BOT data at lsst-dev, the following code raises a NotFoundError:

      from lsst.daf.persistence import Butler
       
      BUTLER_BOT_REPO = '/lsstdata/offline/teststand/BOT/gen2repo'
      butler = Butler(BUTLER_BOT_REPO)
       
      datarefs = butler.subset('raw', run='6545D', raftName='R10',
                               imagetype='BIAS', testtype='DARK')
       
      for dataref in datarefs:
          print(dataref.dataId)
          exp = butler.get('raw', dataref.dataId)
      

      Here is the tail-end of the error output:

         File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/18.1.0-11-g311e899+3/python/lsst/obs/base/cameraMapper.py", line 1076, in _standardizeExposure
          self._setFilter(mapping, item, dataId)
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/obs_base/18.1.0-11-g311e899+3/python/lsst/obs/base/cameraMapper.py", line 1030, in _setFilter
          item.setFilter(afwImage.Filter(filterName))
      lsst.pex.exceptions.wrappers.NotFoundError: 
        File "src/image/Filter.cc", line 335, in static int lsst::afw::image::Filter::_lookup(const string&, bool)
          Unable to find filter SDSSu+ND_OD4.0 {0}
      lsst::pex::exceptions::NotFoundError: 'Unable to find filter SDSSu+ND_OD4.0'
      

      This is using `lsst_distrib w_2019_33`.

      Camera I&T will start producing lots of BOT data with 9 science rafts and 4 corner rafts starting the second week of September 2019, so it would be good to have a fix for this before then.

        Attachments

          Activity

          Hide
          erykoff Eli Rykoff added a comment -

          After James Chiang told me about this, I took a quick look, and it seems that we should be able to leave the filter as UNKNOWN and log a warning and continue with reading the raw without crashing. I tried this on a user branch on obs_base and it seems to work. Is there any reason I can't turn this into a ticket branch to fix this? It would certainly be useful for all sorts of testing situations.

          https://github.com/lsst/obs_base/commit/8f9f4da566e951fa8daf63b21e42e8e0e935ac65

          Show
          erykoff Eli Rykoff added a comment - After James Chiang told me about this, I took a quick look, and it seems that we should be able to leave the filter as UNKNOWN and log a warning and continue with reading the raw without crashing. I tried this on a user branch on obs_base and it seems to work. Is there any reason I can't turn this into a ticket branch to fix this? It would certainly be useful for all sorts of testing situations. https://github.com/lsst/obs_base/commit/8f9f4da566e951fa8daf63b21e42e8e0e935ac65
          Hide
          tjenness Tim Jenness added a comment -

          That might mess up gen 3 butler ingest but I'll let Jim Bosch comment on that one.

          Show
          tjenness Tim Jenness added a comment - That might mess up gen 3 butler ingest but I'll let Jim Bosch comment on that one.
          Hide
          jbosch Jim Bosch added a comment -

          Probably prudent to come back to this after DM-20763 lands (I think that should happen sometime today).  That changes the way filters are defined in obs packages quite a bit (in order to make it so Gen2 and Gen3 can use the same declarations).

          Show
          jbosch Jim Bosch added a comment - Probably prudent to come back to this after DM-20763 lands (I think that should happen sometime today).  That changes the way filters are defined in obs packages quite a bit (in order to make it so Gen2 and Gen3 can use the same declarations).
          Hide
          erykoff Eli Rykoff added a comment -

          I will watch for that. But scanning the PR right now, I don't think it'll change that particular aspect of the reading of raws (but might change what tests are done).

          Show
          erykoff Eli Rykoff added a comment - I will watch for that. But scanning the PR right now, I don't think it'll change that particular aspect of the reading of raws (but might change what tests are done).
          Hide
          erykoff Eli Rykoff added a comment -

          I've rebased against DM-20763 and things work just the same. Anybody want to review https://github.com/lsst/obs_base/pull/168 ?

          Show
          erykoff Eli Rykoff added a comment - I've rebased against DM-20763 and things work just the same. Anybody want to review https://github.com/lsst/obs_base/pull/168 ?

            People

            Assignee:
            erykoff Eli Rykoff
            Reporter:
            jchiang James Chiang
            Reviewers:
            Jim Bosch
            Watchers:
            Eli Rykoff, Eric Charles, James Chiang, Jim Bosch, Kian-Tat Lim, Robert Lupton, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.