Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19616

Make IngestIndexReferenceObjectsTask multiprocessing capable

    Details

    • Templates:
    • Story Points:
      8
    • Sprint:
      AP S19-6, AP F19-1
    • Team:
      Alert Production

      Description

      In order to be able to generate a Gaia (or PS1-DR2) reference catalog in a reasonable amount of time, we need a way to process files in parallel. The existing IngestIndexReferenceObjectsTask is not capable of running in parallel. One approach that should be fairly simple is to use multiprocessing plus file locking: lock each output catalog before appending.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  Parejkoj John Parejko
                  Reporter:
                  Parejkoj John Parejko
                  Reviewers:
                  Christopher Waters
                  Watchers:
                  Christopher Waters, Colin Slater, Jim Bosch, John Parejko, John Swinbank, Paul Price, Simon Krughoff
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel