# gen3_to_job.py does not create uploadable Jobs

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Sprint:
AP S21-4 (March)
• Team:
• Urgent?:
No

#### Description

Following the resolution of DM-29004, gen3_to_job.py runs in ap_verify Gen 3 runs, but no Job files get detected and uploaded. This could be either because gen3_to_job.py does not generate any files, or because they do not get picked up by the upload selector.

The command line invoked was:

 gen3_to_job.py '/j/ws/scipipe/ap_verify/cosmos_pdr2-master^gen3^ap_verify-installed/run/CI-CosmosPDR2/repo' ap_verify-output --dataset_name CI-CosmosPDR2 

I've confirmed on lsst-dev that this syntax (absolute path, single quotes, no trailing slash, etc.) should work, producing .verify.json files in the working directory. I've also confirmed that the repository location was cosmos_pdr2-master^gen3^ap_verify-installed/run/CI-CosmosPDR2/repo/, so the absolute version of the path is probably correct.

#### Activity

Hide
Krzysztof Findeisen added a comment -

I think I found the problem: the files were created in /j/ws/scipipe/ap_verify (a.k.a. jobDir in the script), but the search for the files was relative to /j/ws/scipipe/ap_verify/cosmos_pdr2-master^gen3^ap_verify-installed/run/ (a.k.a. runDir). I need to be calling the script from runDir as well.

I think this means I need to modify util.groovy:runGen3ToJob to take separate arguments for the working directory and the Gen 3 repo. It's pointlessly redundant, but it's what's consistent with other functions (e.g., runApVerify, runDispatchVerify), and the only other way to handle the PWD internally is to let the output go into the repo directory.

Show
Krzysztof Findeisen added a comment - I think I found the problem: the files were created in /j/ws/scipipe/ap_verify (a.k.a. jobDir in the script), but the search for the files was relative to /j/ws/scipipe/ap_verify/cosmos_pdr2-master^gen3^ap_verify-installed/run/ (a.k.a. runDir ). I need to be calling the script from runDir as well. I think this means I need to modify util.groovy:runGen3ToJob to take separate arguments for the working directory and the Gen 3 repo. It's pointlessly redundant, but it's what's consistent with other functions (e.g., runApVerify , runDispatchVerify ), and the only other way to handle the PWD internally is to let the output go into the repo directory.
Hide
Krzysztof Findeisen added a comment -

Kian-Tat Lim, could you review the (merged) jenkins-dm-jobs#770 a posteriori? Happy to make any needed changes on a "b" branch.

This should the last review I ask you for, since I've confirmed that we now have Gen 3 metrics in SQuaSH.

Show
Krzysztof Findeisen added a comment - Kian-Tat Lim , could you review the (merged) jenkins-dm-jobs#770 a posteriori? Happy to make any needed changes on a "b" branch. This should the last review I ask you for, since I've confirmed that we now have Gen 3 metrics in SQuaSH.
Hide
Kian-Tat Lim added a comment -

Looks fine, no objections.

Show
Kian-Tat Lim added a comment - Looks fine, no objections.

#### People

Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
Kian-Tat Lim
Watchers:
Kian-Tat Lim, Krzysztof Findeisen