Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25806

Support parallel ap_verify ingestion in Gen 3

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      The Gen 3 lsst.obs.base.RawIngestTask can be passed the number of processes to use for ingesting in parallel. This could potentially greatly speed up the processing of large datasets like HiTS2015. Propagate the existing ap_verify.py -j flag to Gen 3 ingestion.

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-26237 Can't rerun ap_verify on same repository in Gen 3

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-26526 Can't run RawIngestTask with processes != 1

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21915 Support Gen 3 ingestion of ap_verify datasets

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-26662 ap_verify import slow in Gen 3

          • Done

            Activity

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Ian Sullivan
              Watchers:
              Ian Sullivan, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.