Details
-
Type:
Story
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: jointcal
-
Story Points:4
-
Epic Link:
-
Sprint:AP S19-6, AP F19-1
-
Team:Alert Production
-
Urgent?:No
Description
In order to inform future improvements in jointcal's performance, we need to do some profiling on 10, 50, and 100 visits, to see how the performance of the different components (e.g. outlier rejection, matrix construction, solving) scale with the number of visits. In addition to the general profiling code in the stack, jointcal has a task-specific profiling option that separates the algorithm profiling from the data reading portion.
Hsin-Fang Chiang: do you know if there is a tract of data on lsst-dev that has roughly 100 visits in it? If we do not have 100 visits available on lsst-dev, we can use whatever tract has the largest number.
Attachments
Issue Links
Activity
Now that the gen3 middleware and job submission system is relatively stable, I'll put together some scripts to run jointcal on a few different numbers of visits in gen3, with the profiling options turned on. Then Clare Saunders and I can use those to more easily test how our outlier rejection changes affect the overall runtime on large numbers of visits, and where the new pain points are. This might require pulling down a larger HSC dataset to run on.
There are visits from PDR2 available in /repo/main on lsst-devl, but they haven't been processed in gen3 single frame yet.
This document should provide a start for the necessary dataIds: https://confluence.lsstcorp.org/display/DM/S20+HSC+PDR2+Reprocessing
John Parejko as far as I know for HSC one of the largest numbers would be UD_COSMOS tract=9813 filter=HSC-Y which has 52 visits; see this table here
Or, the simulated LSST data from DC2 (at /datasets/DC2/repo/) has some tracts with >100 overlapping visits. For example, any filter of tract=4849. Some of those require really large memory to run.