Details
-
Type:
Bug
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_decam
-
Labels:
Description
Decam defect ingestion is a bit of a mess. In this particular example, it turns out the calibDate (a required registry field, from which both validStart and validEnd are derived) is set by looking for a date string in the file path. Not the file name, but the path to the file.
For example, consider the defect file D_n20150105t0115_c47_r2134p01_bpm.fits, for ccd 47. Let's say you are trying to ingest it (and friends) using the command
ingestCalibs.py . --calib calibrepo --calibType defect calibrepo/defects/*fits --validity 0 --mode=skip |
The ingestion will "work," but the calib registry fields calibDate, validStart, and validEnd will all be set to unknown (despite something resembling a date in the filename), and any subsequent processing will fail.
If instead, you have the same friendly defect file living in a differently-named directory and run the equivalent command
ingestCalibs.py . --calib calibrepo --calibType defect calibrepo/defects_2014-12-05/*fits --validity 0 --mode=skip |
The ingestion will work and the calibDate field will be "correctly" set to 2014-12-05.
This problem arguably arises from the lack of a DATE-OBS header keyword (or any timestamp-related header keyword whatsoever) in the defect files.
~~~~~
While we're on a roll, here are some other decam defect ingestion fun facts for the poor souls who stumble across this ticket when trying to solve their problems, but which this ticket is not explicitly trying to address.
- ingestCalibs.py prints out lots of warnings while ingesting decam defects which appear to relate to an attempt to read a multi-extension FITS file when in fact the defects have only a single extension. The warnings don't break anything but they are rather scary looking.
- You have to ingest defects with --mode=skip because getDestination requires the OBSTYPE header keyword which, surprise, doesn't exist.
- You must include the --validity flag with some value for defect ingestion to work, but the value is not used. Instead, validStart is set equal to calibDate and validEnd is set to some time far in the future. Incidentally, this is probably why the directories on lsst-dev in /datasets/decam/_internal/calib/bpmDes are named with dates which tend to be a few months earlier than the dates in the defect filenames themselves.
- You must specify --calibType defect because, newsflash, there is still no OBSTYPE header keyword (also see
DM-13975). - Finally, you have to ensure the argument used for the path to the defect files when you run ingestCalibs.py is relative to the repository you are working in. It cannot be an absolute path, or you will run into "No location for get" for the defects when you try to do any subsequent processing. (This problem also spawned
DM-14848.)
I too don't know if/where that's documented either, but I think Gregory Dubois-Felsmann wouldn't be a bad person to ask?! (Sorry Gregory!).
Also, I think that we all somehow managed to agree a little while ago that if we do do snaps, that they would not be two components of a single visit number, but would instead each be given unique (contiguous) visit numbers - just thought I'd throw that in there (because this is a good thing).