Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21836

Add OBSTYPE/purpose to Gen3 Registry exposure table

    XMLWordPrintable

    Details

      Description

      We need to make sure the type of exposure (e.g. "science", "bias", "flat") is extracted from metadata and put into the registry's exposure table in ingest.  I think this is what's called OBSTYPE in FITS, and I'm not sure if ObservationInfo already includes it and we drop it on the floor in ingest, or if we need to add it to astro_metadata_translator (either way, I'm sure that's on Tim Jenness's radar).

      In any case, this is becoming particular important as we prototype Gen3 CPP pipelines, which we of course want to run only on raws with specific types.

        Attachments

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment - - edited

            In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.

            Edit to add: for gen3 processing, such as with `cp_pipe`/`cpBiasTask.py`.

            Show
            czw Christopher Waters added a comment - - edited In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data. Edit to add: for gen3 processing, such as with `cp_pipe`/`cpBiasTask.py`.
            Hide
            tjenness Tim Jenness added a comment -

            I don't really understand. Gen2 stores visit numbers for all raw data including calibrations. The construct bias scripts take visit numbers as arguments. We also have spurious warnings about WCS not working when reading raw bias frames because VisitInfo doesn't have an observation type in it.

            Show
            tjenness Tim Jenness added a comment - I don't really understand. Gen2 stores visit numbers for all raw data including calibrations. The construct bias scripts take visit numbers as arguments. We also have spurious warnings about WCS not working when reading raw bias frames because VisitInfo doesn't have an observation type in it.
            Hide
            jbosch Jim Bosch added a comment -

            Gen2 visit != Gen3 visit != VisitInfo visit; at least as it stands now, Gen3 CPP pipelines should not deal with visits at all - only exposures.

            Anyhow, I've dropped the last commit from obs_base and added an obs_subaru branch with the replacement test: https://github.com/lsst/obs_subaru/pull/225.  Running Jenkins again.

             

            Show
            jbosch Jim Bosch added a comment - Gen2 visit != Gen3 visit != VisitInfo visit; at least as it stands now, Gen3 CPP pipelines should not deal with visits at all - only exposures. Anyhow, I've dropped the last commit from obs_base and added an obs_subaru branch with the replacement test: https://github.com/lsst/obs_subaru/pull/225 .  Running Jenkins again.  
            Hide
            jbosch Jim Bosch added a comment -

            In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data.

            Yeah, that'd be the same fundamental problem as DM-21832; I think we need to fix that query logic.  In the meantime I just won't mess with the logic that sets exposure.visit more often than I expected.

            Show
            jbosch Jim Bosch added a comment - In my tests with ISR, any exposure with `exposure.visit IS NULL` fails to generate a quantum graph, so having a populated `visit_id` is currently an essential field for `raw` data. Yeah, that'd be the same fundamental problem as DM-21832 ; I think we need to fix that query logic.  In the meantime I just won't mess with the logic that sets exposure.visit more often than I expected.
            Hide
            czw Christopher Waters added a comment -

            I can build a repo that matches the new schema, load various calibration raw data into it, and get different numbers of quanta that will be run depending on how I restrict the input data (i.e., whether or not "AND exposure.observation_type = 'bias'" is included).

             

            Show
            czw Christopher Waters added a comment - I can build a repo that matches the new schema, load various calibration raw data into it, and get different numbers of quanta that will be run depending on how I restrict the input data (i.e., whether or not "AND exposure.observation_type = 'bias'" is included).  

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Christopher Waters
              Watchers:
              Christopher Waters, Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.