Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19616

Make IngestIndexReferenceObjectsTask multiprocessing capable

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S19-6, AP F19-1
    • Team:
      Alert Production

      Description

      In order to be able to generate a Gaia (or PS1-DR2) reference catalog in a reasonable amount of time, we need a way to process files in parallel. The existing IngestIndexReferenceObjectsTask is not capable of running in parallel. One approach that should be fairly simple is to use multiprocessing plus file locking: lock each output catalog before appending.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Christopher Waters
                Watchers:
                Christopher Waters, Colin Slater, Jim Bosch, John Parejko, John Swinbank, Paul Price, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel