Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5819

Incorporate Price suggestions to make `validate_drp` faster

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      Increase the loading and processing speed of validate_drp following suggestions by Paul Price

      1. Don't read in footprints
      Pass flags=lsst.afw.table.SOURCE_IO_NO_FOOTPRINTS to butler.get

      2. Work on speed of calculation of RMS and other expensive quantities. Current suggestions:
      a. calcRmsDistances
      b. multiMatch
      c. matchVisitComputeDistance
      d. Consider boolean indexing in afw's multiMatch.py

         objById = {record.get(self.objectKey): record for record in self.reference}
      to:
         objById = dict(zip(self.reference[self.objectKey], self.reference))
      

      Note that while this ticket will involve work to reduce the memory footprint of the processing, it will not cover work to re-architect things to enable efficient processing beyond the memory on one node.

        Attachments

          Activity

            People

            • Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              wmwood-vasey Michael Wood-Vasey
              Reviewers:
              Angelo Fausti
              Watchers:
              Angelo Fausti, Michael Wood-Vasey, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel