Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25806

Support parallel ap_verify ingestion in Gen 3

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      The Gen 3 lsst.obs.base.RawIngestTask can be passed the number of processes to use for ingesting in parallel. This could potentially greatly speed up the processing of large datasets like HiTS2015. Propagate the existing ap_verify.py -j flag to Gen 3 ingestion.

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-26237 Can't rerun ap_verify on same repository in Gen 3

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-26526 Can't run RawIngestTask with processes != 1

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21915 Support Gen 3 ingestion of ap_verify datasets

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-26662 ap_verify import slow in Gen 3

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use.

            While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.

            Show
            krzys Krzysztof Findeisen added a comment - It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use. While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.
            Hide
            Permalink
            sullivan Ian Sullivan added a comment -

            Looks good. I left a few very minor comments on the pull request.

            Show
            sullivan Ian Sullivan added a comment - Looks good. I left a few very minor comments on the pull request.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Ian Sullivan
              Watchers:
              Ian Sullivan, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.