Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19616

Make IngestIndexReferenceObjectsTask multiprocessing capable

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP S19-6, AP F19-1
    • Team:
      Alert Production

      Description

      In order to be able to generate a Gaia (or PS1-DR2) reference catalog in a reasonable amount of time, we need a way to process files in parallel. The existing IngestIndexReferenceObjectsTask is not capable of running in parallel. One approach that should be fairly simple is to use multiprocessing plus file locking: lock each output catalog before appending.

        Attachments

          Issue Links

            Activity

            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Epic Link DM-19473 [ 263512 ]
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Parejkoj John Parejko made changes -
            Attachment test_filelock.py [ 38045 ]
            Parejkoj John Parejko made changes -
            Attachment test_filelock.py [ 38047 ]
            Parejkoj John Parejko made changes -
            Attachment test_filelock.py [ 38045 ]
            Parejkoj John Parejko made changes -
            Reviewers John Swinbank [ swinbank ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-19761 [ DM-19761 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-19768 [ DM-19768 ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] In Progress [ 3 ]
            Parejkoj John Parejko made changes -
            Link This issue is blocked by DM-20019 [ DM-20019 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-20025 [ DM-20025 ]
            Parejkoj John Parejko made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-6 [ 834 ] AP S19-6, AP F19-1 [ 834, 923 ]
            Parejkoj John Parejko made changes -
            Reviewers John Swinbank [ swinbank ] Christopher Waters [ cwaters ]
            Status In Review [ 10004 ] In Review [ 10004 ]
            czw Christopher Waters made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Christopher Waters
                Watchers:
                Christopher Waters, Colin Slater, Jim Bosch, John Parejko, John Swinbank, Paul Price, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel