Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: faro
-
Labels:
-
Epic Link:
-
Team:DM Science
-
Urgent?:No
Description
To confirm that gen3_RC2_subset will meet our needs for a CI dataset, run `faro` on those data and check the resulting metrics.
Make necessary changes to the pipeline in gen3_RC2_subset and set it up to regularly run in Jenkins.
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
| Field | Original Value | New Value |
|---|---|---|
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Attachment | metrics_RC2_subset_faro.pdf [ 51524 ] |
| Watchers | Jeffrey Carlin [ Jeffrey Carlin ] | Jeffrey Carlin, Leanne Guy, Simon Krughoff [ Jeffrey Carlin, Leanne Guy, Simon Krughoff ] |
| Attachment | metrics_RC2_subset_faro.pdf [ 51524 ] |
| Attachment | metrics_RC2_subset_faro.pdf [ 51525 ] |
Let me know if you think there are other things I should check before we move forward with `gen3_RC2_subset` as our CI dataset.
| Reviewers | Leanne Guy, Simon Krughoff [ lguy, krughoff ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
It would be good to add a script that does the processing you need to run faro. That will make it easier to swap out in jenkins
I'm not sure I understand what you mean? Do you mean to add a faro script to the `gen3_RC2_subset` Jenkins job?
For this purpose, I ran "DRP.yaml#singleFrame", "DRP.yaml#jointcal", and "DRP.yaml#fgcm" from the repo. Then I ran "validate_drp_metrics_pipeline.yaml" from faro. (Just calculated matched-visit metrics for now.)
I mean we have scripts in faro that make it easy for jenkins to run processing and metric measurement. I think we need those same scripts (probably in faro) for doing the processing and measurement on this repository instead
Oh, so the script (DRP.yaml) that is in gen3_RC2_subset is different from what would be needed in Jenkins? (Sorry if I'm being dense...)
| Summary | Run faro on gen3_RC2_subset | Run faro on gen3_RC2_subset and set it up in Jenkins |
| Description | To confirm that [gen3_RC2_subset|https://github.com/lsst-dm/gen3_rc2_subset] will meet our needs for a CI dataset, run `faro` on those data and check the resulting metrics. |
To confirm that [gen3_RC2_subset|https://github.com/lsst-dm/gen3_rc2_subset] will meet our needs for a CI dataset, run `faro` on those data and check the resulting metrics.
Make necessary changes to the pipeline in gen3_RC2_subset and set it up to regularly run in Jenkins. |
| Status | In Review [ 10004 ] | In Progress [ 3 ] |
| Remote Link | This issue links to "Page (Confluence)" [ 30202 ] |
I added the `faro` steps to the DRP.yaml file in rc2_subset (PR here), and added a shell script to run all the processing in `faro` (PR here).
I have confirmed that the script runs to completion and generates the expected output data. The next step is to set up a Jenkins job (under the name `verify_drp_metrics`).
| Reviewers | Leanne Guy, Simon Krughoff [ lguy, krughoff ] | Simon Krughoff [ krughoff ] |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
I'd like you to take a look at some of the comments I have in the faro PR, but it all looks generally OK to me.
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
Thanks for the suggestions – I implemented them, and confirmed that it runs to completion. With just patch 40, it took ~4.5 hours on lsst-devl with 8 cores.
Successful Jenkins run here.
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
Both PRs have been merged.
I've attached a table of `faro` metrics measured on that dataset. The best point of comparison would probably be the last Characterization Metric Report.
As expected, the values are typically higher than the full RC2 results, but they mostly seem reasonable.