Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24395

HSC defects_visit table has unexpected filter column

    Details

    • Story Points:
      4
    • Urgent?:
      No

      Description

      While working on DM-24259, I performed automated Gen 2 ingest of an HSC dataset in the following order:

      1. Call obs.subaru.ingest.HscIngestTask on the raws, with the obs_subaru config overrides
      2. Call pipe.tasks.ingestCalibs.IngestCalibsTask on the calibs, with the obs_subaru config overrides
      3. Call lsst.pipe.tasks.ingestCuratedCalibs.IngestCuratedCalibsTask on the defects in obs_subaru_data, with the obs_subaru config overrides

      This led to the following error on defects ingest:

      datasetIngest.defectIngester INFO: defects written for sensor: 0_01 and calibDate: 2016-11-22T00:00:00
      datasetIngest.defectIngester INFO: defects written for sensor: 0_01 and calibDate: 2014-06-01T00:00:00
      datasetIngest.defectIngester.register INFO: Table "defects" exists.  Skipping creation
      datasetIngest.defectIngester INFO: /tmp/tmpq7y7eqj3/defects_0_56_2013-01-31T00:00:00.fits --<move>--> /scratch/krzys001/_test_hsc/calibingested/defects/2013-01-31T00:00:00/defects-2013-01-31T00:00:00-096.fits
      Traceback (most recent call last):
        File "/scratch/krzys001/ap_verify/bin/ingest_dataset.py", line 29, in <module>
          result = runIngestion()
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ap_verify.py", line 210, in runIngestion
          ingestDataset(args.dataset, workspace)
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 432, in ingestDataset
          ingester.run(dataset, workspace)
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 148, in run
          self._ingestDefects(dataset, workspace)
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 306, in _ingestDefects
          self._doIngestDefects(workspace.dataRepo, workspace.calibRepo, self.config.textDefectPath)
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 330, in _doIngestDefects
          _runIngestTask(self.defectIngester, defectargs)
        File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 480, in _runIngestTask
          task.run(parsedCmd)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCuratedCalibs.py", line 56, in run
          IngestCalibsTask.run(self, args)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCalibs.py", line 247, in run
          create=args.create, table=calibType)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCalibs.py", line 103, in addRow
          RegisterTask.addRow(self, conn, info, *args, **kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingest.py", line 362, in addRow
          conn.cursor().execute(sql, values)
      sqlite3.OperationalError: table defects_visit has 2 columns but 1 values were supplied
      

      Investigation revealed that the table had columns for date and filter, but the obs_subaru config for IngestCuratedCalibsTask expected the defects_visit table to only have a date column. Simon Krughoff speculates that the table was created during calib ingest (which is configured to have its *_visit tables be (date, filter)). However, if this is the case then it's not clear how ingestion passes when building obs_subaru; there are no instrument-specific config files for ingestion.

      Given that defects should not be filter-dependent, this appears to be a bug in HSC ingestion. Investigate the issue to confirm the cause, and fix. One plausible option is to create the defects-related registry tables only when ingesting defects, not calibs.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I was under the impression that the obs_x_data packages force filter to be present. I was a bit surprised by this when I was looking at the Linearizer code that also required it. ingestCuratedCalibs is forcing filter to default to NONE for all curated calibs, possibly because ingestCalibs also forces filter to be used. If you look at bias entries in a calibration registry you'll also see that filter is present (always set to NONE).

            Show
            tjenness Tim Jenness added a comment - I was under the impression that the obs_x_data packages force filter to be present. I was a bit surprised by this when I was looking at the Linearizer code that also required it. ingestCuratedCalibs is forcing filter to default to NONE for all curated calibs, possibly because ingestCalibs also forces filter to be used. If you look at bias entries in a calibration registry you'll also see that filter is present (always set to NONE).
            Hide
            krzys Krzysztof Findeisen added a comment -
            Show
            krzys Krzysztof Findeisen added a comment - So are you saying that https://github.com/lsst/obs_subaru/blob/cbbde586f25690534e76e3fd2bcdc4ed751fbffb/config/ingestCuratedCalibs.py#L14 should be changed to ['calibDate', 'filter'] ?
            Hide
            tjenness Tim Jenness added a comment -

            I have never looked at this before but it's likely that since ingestCalibs is doing this and since we are creating a filter for defects just like for bias, then it's possible that we have to be closer to the ingestCalibs.py config here.

            Show
            tjenness Tim Jenness added a comment - I have never looked at this before but it's likely that since ingestCalibs is doing this and since we are creating a filter for defects just like for bias, then it's possible that we have to be closer to the ingestCalibs.py config here.
            Hide
            tjenness Tim Jenness added a comment -

            Does this all mean that obs_subaru has never been switched over to obs_subaru_data?

            Show
            tjenness Tim Jenness added a comment - Does this all mean that obs_subaru has never been switched over to obs_subaru_data?

              People

              • Assignee:
                Unassigned
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                John Swinbank, Krzysztof Findeisen, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel