Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18624

speed up Table pickling

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Story Points:
      2
    • Sprint:
      AP S19-4, AP S19-5
    • Team:
      Alert Production

      Description

      With Schema pickleable (DM-17950), I can now use ProcessPools to parallelize reading data. But pickle/unpickle of SourceCatalogs seems to be preventing that from actually providing a speed boost. I've found at least one obvious place to gain a big chunk of speed. I'll spend a little more time looking for others.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment - - edited

            Nate Lust: do you mind doing this short review? Sadly, it doesn't gain enough to help me with my
            parallelization problems, but it's probably worth incorporating anyway.

            afw PR if Jira doesn't pick it up: https://github.com/lsst/afw/pull/445

            Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29651/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited Nate Lust : do you mind doing this short review? Sadly, it doesn't gain enough to help me with my parallelization problems, but it's probably worth incorporating anyway. afw PR if Jira doesn't pick it up: https://github.com/lsst/afw/pull/445 Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29651/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Nate Lust
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel