Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: astro_metadata_translator, daf_butler
-
Labels:
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
Description
We need to make sure the type of exposure (e.g. "science", "bias", "flat") is extracted from metadata and put into the registry's exposure table in ingest. I think this is what's called OBSTYPE in FITS, and I'm not sure if ObservationInfo already includes it and we drop it on the floor in ingest, or if we need to add it to astro_metadata_translator (either way, I'm sure that's on Tim Jenness's radar).
In any case, this is becoming particular important as we prototype Gen3 CPP pipelines, which we of course want to run only on raws with specific types.
Attachments
Issue Links
- relates to
-
- In Progress
Activity
I don't really understand. Gen2 stores visit numbers for all raw data including calibrations. The construct bias scripts take visit numbers as arguments. We also have spurious warnings about WCS not working when reading raw bias frames because VisitInfo doesn't have an observation type in it.
Gen2 visit != Gen3 visit != VisitInfo visit; at least as it stands now, Gen3 CPP pipelines should not deal with visits at all - only exposures.
Anyhow, I've dropped the last commit from obs_base and added an obs_subaru branch with the replacement test: https://github.com/lsst/obs_subaru/pull/225. Running Jenkins again.
In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.
Yeah, that'd be the same fundamental problem as DM-21832; I think we need to fix that query logic. In the meantime I just won't mess with the logic that sets exposure.visit more often than I expected.
I can build a repo that matches the new schema, load various calibration raw data into it, and get different numbers of quanta that will be run depending on how I restrict the input data (i.e., whether or not "AND exposure.observation_type = 'bias'" is included).
In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.
Edit to add: for gen3 processing, such as with `cp_pipe`/`cpBiasTask.py`.