Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15764

fail to retrieve calibration data with obs_lsstCam

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None

      Description

      obs_lsstCam calibrations policy uses as date as obsTimeName, not dateObs. 

      RuntimeError: Unable to retrieve bias for {'visit': 185615, 'detector': 29, 'run': '185615', 'raftName': 'R10', 'detectorName': 'S02', 'snap': 0}: no such column: date

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15691 The raw_visit table in obs_lsstCam registry is not optimized

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            rhl Robert Lupton added a comment -

            I don't understand.  The code works for me – what's the how-to-repeat?  If it was caused by DM-15691, then I think you may end up fixing it!

             

            Show
            rhl Robert Lupton added a comment - I don't understand.  The code works for me – what's the how-to-repeat?  If it was caused by DM-15691 , then I think you may end up fixing it!  
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Hmm.. I ingested imsim data from scratch, tried "processCcd" with some ISR calibrations on, and got the above error. I added date to ingest config.register.visit and then went pass this error. Let me try again, maybe it was my user error.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Hmm.. I ingested imsim data from scratch, tried "processCcd" with some ISR calibrations on, and got the above error. I added date to ingest config.register.visit and then went pass this error. Let me try again, maybe it was my user error.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            I tried again and got the same error as above (no such column: date), which makes sense to me and is easy to fix: either fix the calibration lookup policy like this commit, or fix the ingest raw_visit schema to include date at this line (which I changed in DM-15691).

            Then, my "processCcd" halted this error, which I don't fully understand yet:
            $ processCcd.py . --rerun test --id visit=185615 raftName=R10 detectorName=S02 

            Traceback (most recent call last):
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ip_isr/16.0-9-gcc4efb7+5/python/lsst/ip/isr/isrTask.py", line 787, in getIsrExposure
            		 
                exp = dataRef.get(datasetType, immediate=immediate)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butlerSubset.py", line 198, in get
            		 
                return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butler.py", line 1378, in get
            		 
                location = self._locate(datasetType, dataId, write=False)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butler.py", line 1297, in _locate
            		 
                location = repoData.repo.map(datasetType, dataId, write=write)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/repository.py", line 246, in map
            		 
                loc = self._mapper.map(*args, **kwargs)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/mapper.py", line 166, in map
            		 
                return func(self.validate(dataId), write)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/cameraMapper.py", line 394, in mapClosure
            		 
                return mapping.map(mapper, dataId, write)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 464, in map
            		 
                location = Mapping.map(self, mapper, dataId, write=write)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 156, in map
            		 
                actualId = self.need(iter(self.keyDict.keys()), dataId)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 320, in need
            		 
                lookups = self.lookup(newProps, newId)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 513, in lookup
            		 
                lookups = self.refRegistry.lookup(columns, self.reference, lookupDataId)
            		 
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/registries.py", line 369, in lookup
            		 
                cursor.execute(cmd, valueList)
            		 
            sqlite3.OperationalError: no such column: detector

            I think this might be a problem from here. My understanding is that the code looks up the columns in the reference table given the refCols. So here it wanted the detector column (refCols) in the raw_visit table (reference). Expecting detector in the raw_visit table seems weird to me as I expected only one row for each visit in the raw_visit table. Any thoughts?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I tried again and got the same error as above ( no such column: date ), which makes sense to me and is easy to fix: either fix the calibration lookup policy like this commit , or fix the ingest raw_visit schema to include date at this line (which I changed in DM-15691 ). Then, my "processCcd" halted this error, which I don't fully understand yet: $ processCcd.py . --rerun test --id visit=185615 raftName=R10 detectorName=S02 Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ip_isr/16.0-9-gcc4efb7+5/python/lsst/ip/isr/isrTask.py", line 787, in getIsrExposure exp = dataRef.get(datasetType, immediate=immediate) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butlerSubset.py", line 198, in get return self.butlerSubset.butler.get(datasetType, self.dataId, **rest) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butler.py", line 1378, in get location = self._locate(datasetType, dataId, write=False) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/butler.py", line 1297, in _locate location = repoData.repo.map(datasetType, dataId, write=write) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/repository.py", line 246, in map loc = self._mapper.map(*args, **kwargs) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/mapper.py", line 166, in map return func(self.validate(dataId), write) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/cameraMapper.py", line 394, in mapClosure return mapping.map(mapper, dataId, write) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 464, in map location = Mapping.map(self, mapper, dataId, write=write) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 156, in map actualId = self.need(iter(self.keyDict.keys()), dataId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 320, in need lookups = self.lookup(newProps, newId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/16.0-13-gde155d7+1/python/lsst/obs/base/mapping.py", line 513, in lookup lookups = self.refRegistry.lookup(columns, self.reference, lookupDataId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/16.0-3-g3806c63+8/python/lsst/daf/persistence/registries.py", line 369, in lookup cursor.execute(cmd, valueList) sqlite3.OperationalError: no such column: detector I think this might be a problem from here . My understanding is that the code looks up the columns in the reference table given the refCols . So here it wanted the detector column ( refCols ) in the raw_visit table ( reference ). Expecting detector in the raw_visit table seems weird to me as I expected only one row for each visit in the raw_visit table. Any thoughts?
            Hide
            Permalink
            rhl Robert Lupton added a comment -

            Does it all work if you use a version of `obs_lsstCam` from before your changes? I can look at it later (maybe) if that's difficult for you – note that I had to fix `obs_lsstCam` and Jim had to fix DM-15776, so you can't just use master and the latest weekly. 

            Show
            rhl Robert Lupton added a comment - Does it all work if you use a version of `obs_lsstCam` from before your changes? I can look at it later (maybe) if that's difficult for you – note that I had to fix `obs_lsstCam` and Jim had to fix  DM-15776 , so you can't just use master and the latest weekly. 
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            I think this works now, with w_2018_37 + the master daf_persistence (post-DM-15776).  Would you please review it? 

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I think this works now, with w_2018_37 + the master daf_persistence (post- DM-15776 ).  Would you please review it? 
            Hide
            Permalink
            rhl Robert Lupton added a comment -

            Looks good.

            Show
            rhl Robert Lupton added a comment - Looks good.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for reviewing and merging it for me!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for reviewing and merging it for me!

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Robert Lupton
              Watchers:
              Hsin-Fang Chiang, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.