Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13451

Make ap_verify responsible for ingestion

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S18-3
    • Team:
      Alert Production

      Description

      Currently, ap_pipe ingests and processes data. For its ongoing conversion to a CmdLineTask (DM-13163) and forward-compatibility with Pipeline classes, ap_pipe should work on an externally provided repository.

      For the time being, this functionality should be moved to ap_verify (which currently requires uningested data), and the final responsibility for ingestion will be determined later.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Hi Meredith Rawls, can you review this change when you have the chance? It's about 500 lines of code total, more if you do it commit by commit.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Hi Meredith Rawls , can you review this change when you have the chance? It's about 500 lines of code total, more if you do it commit by commit.
            Hide
            mrawls Meredith Rawls added a comment -

            This looks good to me. The remaining edge cases (e.g., template "ingestion") are handled nicely and it seems like everything is self consistent. I left a note over on GitHub to please make a ticket to write some tests to go along with ingestion now that it lives in ap_verify. I probably should have done that myself a while ago, but testing ingestion is a tricky business, and I don't expect you to do it here.

            Show
            mrawls Meredith Rawls added a comment - This looks good to me. The remaining edge cases (e.g., template "ingestion") are handled nicely and it seems like everything is self consistent. I left a note over on GitHub to please make a ticket to write some tests to go along with ingestion now that it lives in ap_verify . I probably should have done that myself a while ago, but testing ingestion is a tricky business, and I don't expect you to do it here.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Thanks for the quick response! The ingestion tests are DM-13524.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for the quick response! The ingestion tests are DM-13524 .

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Meredith Rawls
                Watchers:
                Krzysztof Findeisen, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel