Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19591

Profile jointcal performance on different numbers of visits

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Sprint:
      AP S19-6, AP F19-1
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      In order to inform future improvements in jointcal's performance, we need to do some profiling on 10, 50, and 100 visits, to see how the performance of the different components (e.g. outlier rejection, matrix construction, solving) scale with the number of visits. In addition to the general profiling code in the stack, jointcal has a task-specific profiling option that separates the algorithm profiling from the data reading portion.

      Hsin-Fang Chiang: do you know if there is a tract of data on lsst-dev that has roughly 100 visits in it? If we do not have 100 visits available on lsst-dev, we can use whatever tract has the largest number.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            John Parejko as far as I know for HSC one of the largest numbers would be UD_COSMOS tract=9813 filter=HSC-Y which has 52 visits; see this table here

            Or, the simulated LSST data from DC2 (at /datasets/DC2/repo/) has some tracts with >100 overlapping visits. For example, any filter of tract=4849. Some of those require really large memory to run.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - John Parejko as far as I know for HSC one of the largest numbers would be UD_COSMOS tract=9813 filter=HSC-Y which has 52 visits; see this table here Or, the simulated LSST data from DC2 (at /datasets/DC2/repo/) has some tracts with >100 overlapping visits. For example, any filter of tract=4849. Some of those require really large memory to run.
            Hide
            Parejkoj John Parejko added a comment - - edited

            Now that the gen3 middleware and job submission system is relatively stable, I'll put together some scripts to run jointcal on a few different numbers of visits in gen3, with the profiling options turned on. Then Clare Saunders and I can use those to more easily test how our outlier rejection changes affect the overall runtime on large numbers of visits, and where the new pain points are. This might require pulling down a larger HSC dataset to run on.

            Show
            Parejkoj John Parejko added a comment - - edited Now that the gen3 middleware and job submission system is relatively stable, I'll put together some scripts to run jointcal on a few different numbers of visits in gen3, with the profiling options turned on. Then Clare Saunders and I can use those to more easily test how our outlier rejection changes affect the overall runtime on large numbers of visits, and where the new pain points are. This might require pulling down a larger HSC dataset to run on.
            Hide
            Parejkoj John Parejko added a comment -

            There are visits from PDR2 available in /repo/main on lsst-devl, but they haven't been processed in gen3 single frame yet.

            This document should provide a start for the necessary dataIds: https://confluence.lsstcorp.org/display/DM/S20+HSC+PDR2+Reprocessing

            Show
            Parejkoj John Parejko added a comment - There are visits from PDR2 available in /repo/main on lsst-devl, but they haven't been processed in gen3 single frame yet. This document should provide a start for the necessary dataIds: https://confluence.lsstcorp.org/display/DM/S20+HSC+PDR2+Reprocessing

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Clare Saunders, Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.