Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13451

Make ap_verify responsible for ingestion

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S18-3
    • Team:
      Alert Production

      Description

      Currently, ap_pipe ingests and processes data. For its ongoing conversion to a CmdLineTask (DM-13163) and forward-compatibility with Pipeline classes, ap_pipe should work on an externally provided repository.

      For the time being, this functionality should be moved to ap_verify (which currently requires uningested data), and the final responsibility for ingestion will be determined later.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13524 Add unit tests for ingestion

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13163 Refactor ap_pipe to use CmdLineTask primitives

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Hi Meredith Rawls, can you review this change when you have the chance? It's about 500 lines of code total, more if you do it commit by commit.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Hi Meredith Rawls , can you review this change when you have the chance? It's about 500 lines of code total, more if you do it commit by commit.
            Hide
            Permalink
            mrawls Meredith Rawls added a comment -

            This looks good to me. The remaining edge cases (e.g., template "ingestion") are handled nicely and it seems like everything is self consistent. I left a note over on GitHub to please make a ticket to write some tests to go along with ingestion now that it lives in ap_verify. I probably should have done that myself a while ago, but testing ingestion is a tricky business, and I don't expect you to do it here.

            Show
            mrawls Meredith Rawls added a comment - This looks good to me. The remaining edge cases (e.g., template "ingestion") are handled nicely and it seems like everything is self consistent. I left a note over on GitHub to please make a ticket to write some tests to go along with ingestion now that it lives in ap_verify . I probably should have done that myself a while ago, but testing ingestion is a tricky business, and I don't expect you to do it here.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Thanks for the quick response! The ingestion tests are DM-13524.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for the quick response! The ingestion tests are DM-13524 .

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Meredith Rawls
              Watchers:
              Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.