Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25806

Support parallel ap_verify ingestion in Gen 3

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      The Gen 3 lsst.obs.base.RawIngestTask can be passed the number of processes to use for ingesting in parallel. This could potentially greatly speed up the processing of large datasets like HiTS2015. Propagate the existing ap_verify.py -j flag to Gen 3 ingestion.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use.

            While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.

            Show
            krzys Krzysztof Findeisen added a comment - It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use. While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.
            Hide
            sullivan Ian Sullivan added a comment -

            Looks good. I left a few very minor comments on the pull request.

            Show
            sullivan Ian Sullivan added a comment - Looks good. I left a few very minor comments on the pull request.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Ian Sullivan
                Watchers:
                Ian Sullivan, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: