Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25806

Support parallel ap_verify ingestion in Gen 3

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      The Gen 3 lsst.obs.base.RawIngestTask can be passed the number of processes to use for ingesting in parallel. This could potentially greatly speed up the processing of large datasets like HiTS2015. Propagate the existing ap_verify.py -j flag to Gen 3 ingestion.

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-26237 Can't rerun ap_verify on same repository in Gen 3

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-26526 Can't run RawIngestTask with processes != 1

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21915 Support Gen 3 ingestion of ap_verify datasets

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-26662 ap_verify import slow in Gen 3

          • Done

            Activity

            Ascending order - Click to sort in descending order
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-25145 [ 435263 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21915 [ DM-21915 ]
            krzys Krzysztof Findeisen made changes -
            Sprint AP F20-3 (August) [ 1033 ]
            krzys Krzysztof Findeisen made changes -
            Rank Ranked lower
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-26237 [ DM-26237 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-26526 [ DM-26526 ]
            krzys Krzysztof Findeisen made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            swinbank John Swinbank made changes -
            Sprint AP F20-3 (August) [ 1033 ] AP F20-4 (September) [ 1039 ]
            swinbank John Swinbank made changes -
            Rank Ranked lower
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-26662 [ DM-26662 ]
            krzys Krzysztof Findeisen made changes -
            Reviewers Ian Sullivan [ sullivan ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            sullivan Ian Sullivan made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Ian Sullivan
              Watchers:
              Ian Sullivan, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.