HSC defects_visit table has unexpected filter column

XMLWordPrintable

Details

• Type: Bug
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Urgent?:
No

Description

While working on DM-24259, I performed automated Gen 2 ingest of an HSC dataset in the following order:

1. Call obs.subaru.ingest.HscIngestTask on the raws, with the obs_subaru config overrides
3. Call lsst.pipe.tasks.ingestCuratedCalibs.IngestCuratedCalibsTask on the defects in obs_subaru_data, with the obs_subaru config overrides

This led to the following error on defects ingest:

 datasetIngest.defectIngester INFO: defects written for sensor: 0_01 and calibDate: 2016-11-22T00:00:00 datasetIngest.defectIngester INFO: defects written for sensor: 0_01 and calibDate: 2014-06-01T00:00:00 datasetIngest.defectIngester.register INFO: Table "defects" exists. Skipping creation datasetIngest.defectIngester INFO: /tmp/tmpq7y7eqj3/defects_0_56_2013-01-31T00:00:00.fits ----> /scratch/krzys001/_test_hsc/calibingested/defects/2013-01-31T00:00:00/defects-2013-01-31T00:00:00-096.fits Traceback (most recent call last):  File "/scratch/krzys001/ap_verify/bin/ingest_dataset.py", line 29, in   result = runIngestion()  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ap_verify.py", line 210, in runIngestion  ingestDataset(args.dataset, workspace)  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 432, in ingestDataset  ingester.run(dataset, workspace)  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 148, in run  self._ingestDefects(dataset, workspace)  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 306, in _ingestDefects  self._doIngestDefects(workspace.dataRepo, workspace.calibRepo, self.config.textDefectPath)  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 330, in _doIngestDefects  _runIngestTask(self.defectIngester, defectargs)  File "/scratch/krzys001/ap_verify/python/lsst/ap/verify/ingestion.py", line 480, in _runIngestTask  task.run(parsedCmd)  File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCuratedCalibs.py", line 56, in run  IngestCalibsTask.run(self, args)  File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCalibs.py", line 247, in run  create=args.create, table=calibType)  File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingestCalibs.py", line 103, in addRow  RegisterTask.addRow(self, conn, info, *args, **kwargs)  File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_tasks/19.0.0-39-g09c7e8d9+3/python/lsst/pipe/tasks/ingest.py", line 362, in addRow  conn.cursor().execute(sql, values) sqlite3.OperationalError: table defects_visit has 2 columns but 1 values were supplied 

Investigation revealed that the table had columns for date and filter, but the obs_subaru config for IngestCuratedCalibsTask expected the defects_visit table to only have a date column. Simon Krughoff speculates that the table was created during calib ingest (which is configured to have its *_visit tables be (date, filter)). However, if this is the case then it's not clear how ingestion passes when building obs_subaru; there are no instrument-specific config files for ingestion.

Given that defects should not be filter-dependent, this appears to be a bug in HSC ingestion. Investigate the issue to confirm the cause, and fix. One plausible option is to create the defects-related registry tables only when ingesting defects, not calibs.

Activity

Hide
Tim Jenness added a comment -

I was under the impression that the obs_x_data packages force filter to be present. I was a bit surprised by this when I was looking at the Linearizer code that also required it. ingestCuratedCalibs is forcing filter to default to NONE for all curated calibs, possibly because ingestCalibs also forces filter to be used. If you look at bias entries in a calibration registry you'll also see that filter is present (always set to NONE).

Show
Tim Jenness added a comment - I was under the impression that the obs_x_data packages force filter to be present. I was a bit surprised by this when I was looking at the Linearizer code that also required it. ingestCuratedCalibs is forcing filter to default to NONE for all curated calibs, possibly because ingestCalibs also forces filter to be used. If you look at bias entries in a calibration registry you'll also see that filter is present (always set to NONE).
Hide
Krzysztof Findeisen added a comment -
Show
Krzysztof Findeisen added a comment - So are you saying that https://github.com/lsst/obs_subaru/blob/cbbde586f25690534e76e3fd2bcdc4ed751fbffb/config/ingestCuratedCalibs.py#L14 should be changed to ['calibDate', 'filter'] ?
Hide
Tim Jenness added a comment -

I have never looked at this before but it's likely that since ingestCalibs is doing this and since we are creating a filter for defects just like for bias, then it's possible that we have to be closer to the ingestCalibs.py config here.

Show
Tim Jenness added a comment - I have never looked at this before but it's likely that since ingestCalibs is doing this and since we are creating a filter for defects just like for bias, then it's possible that we have to be closer to the ingestCalibs.py config here.
Hide
Tim Jenness added a comment -

Does this all mean that obs_subaru has never been switched over to obs_subaru_data?

Show
Tim Jenness added a comment - Does this all mean that obs_subaru has never been switched over to obs_subaru_data?

People

• Assignee:
Unassigned
Reporter:
Krzysztof Findeisen
Watchers:
John Swinbank, Krzysztof Findeisen, Simon Krughoff, Tim Jenness