Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33095

Streamline updating of ap_verify CI datasets

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      DM-29857 implements scripts for regenerating repositories from scratch, as well as individual scripts for each step. However, the scripts for ap_verify_ci_hits2015 and ap_verify_hits2015 are redundant, in that the CI script reprocesses a subset of the HiTS data from scratch. This adds about 10 hours of unnecessary runtime to any update of both repositories.

      Add a way for an update of ap_verify_ci_hits2015 to reuse the collections created for updating ap_verify_hits2015; this may be as simple as adding an optional dataset filter to the CI ingest scripts' CLI.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Assuming we stop maintaining ap_verify_hits2015 per RFC-882 and DM-36385, there will no longer be any redundancy and we will just want to use the CI dataset's standalone script as-is.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Assuming we stop maintaining ap_verify_hits2015 per RFC-882 and DM-36385 , there will no longer be any redundancy and we will just want to use the CI dataset's standalone script as-is.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Watchers:
              Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.