Details
-
Type:
Improvement
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: validate_drp, Verification
-
Labels:None
-
Templates:customfield_11800 156072
-
Epic Link:
-
Team:DM Science
Description
The majority of the time when running validate_drp's MatchedVisitsMetricsTask is spent on ingest and building the MultiMatch catalog. This should be a trivially parallelizable step (modulo MultiMatch's threadsafeness), and could produce a drastic reduction in runtime for validate_drp on large datasets.
First step is to try a ThreadPool (to see if I/O parallelization gains us anything), and the next is to try a ProcessPool (and see whether the returned objects are pickleable).