Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27350

Fix handling of bad metadata translations in gen3 ingest

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      In IHS-4230 there is an error condition where metadata translation fails but completes and later on ingestion fails because the date of the observation is unset.

      • I think we need to turn on pedantic translation mode so that we get early reports of failures and can determine whether to fix them. I think this is better than trying to carry on and hope that everything will be fine with unset values. It might be best to modify astro_metadata_translator to construct an ObservationInfo with a restricted set of properties since ingest doesn't need every property.
      • Put try blocks around the code that uses the ObservationInfo just in case bad data sneaks through.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          There are two parts to this: metadata translation can now be given a subset of values to calculate and a requirement that specific translations must result in non-None. Then obs_base simply lists the properties it needs to exist. This will fix the problem where downstream we weren't expecting the exposure time to be None.

          Show
          tjenness Tim Jenness added a comment - There are two parts to this: metadata translation can now be given a subset of values to calculate and a requirement that specific translations must result in non-None. Then obs_base simply lists the properties it needs to exist. This will fix the problem where downstream we weren't expecting the exposure time to be None.
          Hide
          rhl Robert Lupton added a comment -

          Only picky comments

          Show
          rhl Robert Lupton added a comment - Only picky comments

            People

            Assignee:
            tjenness Tim Jenness
            Reporter:
            tjenness Tim Jenness
            Reviewers:
            Robert Lupton
            Watchers:
            Robert Lupton, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.