Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21836

Add OBSTYPE/purpose to Gen3 Registry exposure table

    XMLWordPrintable

Details

    Description

      We need to make sure the type of exposure (e.g. "science", "bias", "flat") is extracted from metadata and put into the registry's exposure table in ingest.  I think this is what's called OBSTYPE in FITS, and I'm not sure if ObservationInfo already includes it and we drop it on the floor in ingest, or if we need to add it to astro_metadata_translator (either way, I'm sure that's on tjenness's radar).

      In any case, this is becoming particular important as we prototype Gen3 CPP pipelines, which we of course want to run only on raws with specific types.

      Attachments

        Issue Links

          Activity

            czw Christopher Waters added a comment - - edited

            In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.

            Edit to add: for gen3 processing, such as with `cp_pipe`/`cpBiasTask.py`.

            czw Christopher Waters added a comment - - edited In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data. Edit to add: for gen3 processing, such as with `cp_pipe`/`cpBiasTask.py`.
            tjenness Tim Jenness added a comment -

            I don't really understand. Gen2 stores visit numbers for all raw data including calibrations. The construct bias scripts take visit numbers as arguments. We also have spurious warnings about WCS not working when reading raw bias frames because VisitInfo doesn't have an observation type in it.

            tjenness Tim Jenness added a comment - I don't really understand. Gen2 stores visit numbers for all raw data including calibrations. The construct bias scripts take visit numbers as arguments. We also have spurious warnings about WCS not working when reading raw bias frames because VisitInfo doesn't have an observation type in it.
            jbosch Jim Bosch added a comment -

            Gen2 visit != Gen3 visit != VisitInfo visit; at least as it stands now, Gen3 CPP pipelines should not deal with visits at all - only exposures.

            Anyhow, I've dropped the last commit from obs_base and added an obs_subaru branch with the replacement test: https://github.com/lsst/obs_subaru/pull/225.  Running Jenkins again.

             

            jbosch Jim Bosch added a comment - Gen2 visit != Gen3 visit != VisitInfo visit; at least as it stands now, Gen3 CPP pipelines should not deal with visits at all - only exposures. Anyhow, I've dropped the last commit from obs_base and added an obs_subaru branch with the replacement test: https://github.com/lsst/obs_subaru/pull/225 .  Running Jenkins again.  
            jbosch Jim Bosch added a comment -

            In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.

            Yeah, that'd be the same fundamental problem as DM-21832; I think we need to fix that query logic.  In the meantime I just won't mess with the logic that sets exposure.visit more often than I expected.

            jbosch Jim Bosch added a comment - In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data. Yeah, that'd be the same fundamental problem as DM-21832 ; I think we need to fix that query logic.  In the meantime I just won't mess with the logic that sets exposure.visit more often than I expected.

            I can build a repo that matches the new schema, load various calibration raw data into it, and get different numbers of quanta that will be run depending on how I restrict the input data (i.e., whether or not "AND exposure.observation_type = 'bias'" is included).

             

            czw Christopher Waters added a comment - I can build a repo that matches the new schema, load various calibration raw data into it, and get different numbers of quanta that will be run depending on how I restrict the input data (i.e., whether or not "AND exposure.observation_type = 'bias'" is included).  

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Christopher Waters
              Christopher Waters, Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.