Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35071

ap_verify can't import from old ap_verify dataset

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP F22-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Our ap_verify datasets currently use version 1.0.1 of the registry schema. However, when we try to copy the dataset to a sandbox repository, it gets created with version 1.0.2, which causes a conflict between the imported exposure records and those created in raw ingest. For now, hand-patch the export.yaml file to match the 1.0.2 records that would have been created.

      (This problem will also affect our test data for the Google Cloud prototype, but fixing that is out of scope for this issue.)

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            Following discussion with Tim Jenness, I propose instead modifying DatasetIngestTask to overwrite the exposure records on raw ingest. We only have preexisting exposure records in order to support the visitSsoObjects dataset, and these records are supposed to be based on the same raws. The risk that updated records will disagree in a way that breaks e.g. solar system processing is small.

            Show
            krzys Krzysztof Findeisen added a comment - Following discussion with Tim Jenness , I propose instead modifying DatasetIngestTask to overwrite the exposure records on raw ingest. We only have preexisting exposure records in order to support the visitSsoObjects dataset, and these records are supposed to be based on the same raws. The risk that updated records will disagree in a way that breaks e.g. solar system processing is small.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Tim Jenness, can you take a look, since you understand the implications of overwriting the records?

            Show
            krzys Krzysztof Findeisen added a comment - Hi Tim Jenness , can you take a look, since you understand the implications of overwriting the records?
            Hide
            tjenness Tim Jenness added a comment -

            Looks like a good pragmatic fix to me.

            Show
            tjenness Tim Jenness added a comment - Looks like a good pragmatic fix to me.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Tim Jenness
              Watchers:
              Eric Bellm, Ian Sullivan, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.