Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29453

validate_drp_gen3 takes 3X longer than before w_2021_03_13

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: faro
    • Labels:
      None

      Description

      Starting with the w_2021_03_13 build, validate_drp_gen3 (which executes scripts in faro) takes 3X longer than before (more than 13 hours rather than 4+ hours).

      The time appears to be consumed in the processCcd step at the end of https://github.com/lsst/faro/blob/master/bin/hsc_gen2_to_gen3.sh#L12. But the total time taken by "calibrate", "characterizeImage", and "isr" quanta in that step as reported by ctrl.mpexec.singleQuantumExecutor is 77769.208 sec prior to the weekly and 78151.527 sec for the weekly, so the delay does not appear to be in the algorithmic code.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          lguy Leanne Guy added a comment -

          Note from Jim on slack that the processCcd pipeline subset on HSC should be run as : $OBS_SUBARU/pipelines/DRP.yaml

          Show
          lguy Leanne Guy added a comment - Note from Jim on slack that the processCcd pipeline subset on HSC should be run as : $OBS_SUBARU/pipelines/DRP.yaml
          Hide
          Permalink
          lguy Leanne Guy added a comment -

          --long-log has been turned on and the run script updated. A new ci job is running

          Show
          lguy Leanne Guy added a comment - --long-log has been turned on and the run script updated. A new ci job is running
          Hide
          Permalink
          ktl Kian-Tat Lim added a comment - - edited

          It turns out that the time is actually spent in the query graph generation of the next step after processCcd. After using --long-log, --log-level=pipe.base.graphBuilder=DEBUG, and --log=level=sqlalchemy.engine=DEBUG along with pipetask qgraph, this was traced to a change in the sqlite dependency.

          Show
          ktl Kian-Tat Lim added a comment - - edited It turns out that the time is actually spent in the query graph generation of the next step after processCcd. After using --long-log , --log-level=pipe.base.graphBuilder=DEBUG , and --log=level=sqlalchemy.engine=DEBUG along with pipetask qgraph , this was traced to a change in the sqlite dependency.

            People

            Assignee:
            lguy Leanne Guy
            Reporter:
            ktl Kian-Tat Lim
            Watchers:
            Jim Bosch, Kian-Tat Lim, Leanne Guy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.