Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25806

Support parallel ap_verify ingestion in Gen 3

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • ap_verify
    • None

    Description

      The Gen 3 lsst.obs.base.RawIngestTask can be passed the number of processes to use for ingesting in parallel. This could potentially greatly speed up the processing of large datasets like HiTS2015. Propagate the existing ap_verify.py -j flag to Gen 3 ingestion.

      Attachments

        Issue Links

          Activity

            It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use.

            While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.

            krzys Krzysztof Findeisen added a comment - It turns out that multiprocessing RawIngestTask requires creating parallel copies of its parent task(s) (see Slack for discussion). This is a problem, since Gen3DatasetIngestTask (in particular, its initialization) was not designed for parallel use. While not geared toward race conditions, DM-26237 should at least prevent initialization of multiple Gen3DatasetIngestTask instances from causing a problem. Delay this issue until then.
            sullivan Ian Sullivan added a comment -

            Looks good. I left a few very minor comments on the pull request.

            sullivan Ian Sullivan added a comment - Looks good. I left a few very minor comments on the pull request.

            People

              krzys Krzysztof Findeisen
              krzys Krzysztof Findeisen
              Ian Sullivan
              Ian Sullivan, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.