# validate_drp_gen3 takes 3X longer than before w_2021_03_13

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
DM Science
• Urgent?:
No

#### Description

Starting with the w_2021_03_13 build, validate_drp_gen3 (which executes scripts in faro) takes 3X longer than before (more than 13 hours rather than 4+ hours).

The time appears to be consumed in the processCcd step at the end of https://github.com/lsst/faro/blob/master/bin/hsc_gen2_to_gen3.sh#L12. But the total time taken by "calibrate", "characterizeImage", and "isr" quanta in that step as reported by ctrl.mpexec.singleQuantumExecutor is 77769.208 sec prior to the weekly and 78151.527 sec for the weekly, so the delay does not appear to be in the algorithmic code.

#### Activity

Hide
Leanne Guy added a comment -

Note from Jim on slack that the processCcd pipeline subset on HSC should be run as : $OBS_SUBARU/pipelines/DRP.yaml Show Leanne Guy added a comment - Note from Jim on slack that the processCcd pipeline subset on HSC should be run as :$OBS_SUBARU/pipelines/DRP.yaml
Hide
Leanne Guy added a comment -

--long-log has been turned on and the run script updated. A new ci job is running

Show
Leanne Guy added a comment - --long-log has been turned on and the run script updated. A new ci job is running
Hide
Kian-Tat Lim added a comment - - edited

It turns out that the time is actually spent in the query graph generation of the next step after processCcd. After using --long-log, --log-level=pipe.base.graphBuilder=DEBUG, and --log=level=sqlalchemy.engine=DEBUG along with pipetask qgraph, this was traced to a change in the sqlite dependency.

Show
Kian-Tat Lim added a comment - - edited It turns out that the time is actually spent in the query graph generation of the next step after processCcd. After using --long-log , --log-level=pipe.base.graphBuilder=DEBUG , and --log=level=sqlalchemy.engine=DEBUG along with pipetask qgraph , this was traced to a change in the sqlite dependency.

#### People

Assignee:
Leanne Guy
Reporter:
Kian-Tat Lim
Watchers:
Jim Bosch, Kian-Tat Lim, Leanne Guy