# Run Gen 3 single frame measurement on on validation_data_cfht

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
8
• Sprint:
AP F21-5 (October), AP F21-6 (November)
• Team:
• Urgent?:
No

#### Description

Convert the validation_data_hsc pipeline to work on the CFHT dataset, and run them. The output of this will be the input for DM-29535.

#### Activity

Hide
John Parejko added a comment - - edited

Notes from talking with Simon Krughoff:

• validation_data_hsc should be deprecated, as it is not used by the verification team for anything.
• faro is run on rc2_subset via a script in the bin directory there. This just runs pipetask on the obs_subaru DRP.yaml.

I should reorganize the validation_data datasets (assuming we're still going to use them, which might not be feasible!) to have the necessary scripts/configs in them.

A guess at the necessary steps to run on the decam/cfht validation datasets:

• Find out if anyone has a working DECam/CFHT DRP.yaml, and how much does it look like the obs_subaru one?
• Make a gen3 repo that contains the raws/calibs/refcat/skymap.
• Decide whether to always assume the repo is clean (i.e. a new Jenkins checkout every time) or whether to export/import the repo to be able to run repeatedly. The latter would be more future-proofed (like how testdata_jointcal is used for testing), but might be harder to actually do (e.g. calibrations).
• Run pipetask REPO obs_subaru/DPR.yaml#step1 and see what happens. We'll almost certainly have to override that for DECam.
Show
John Parejko added a comment - - edited Notes from talking with Simon Krughoff : validation_data_hsc should be deprecated, as it is not used by the verification team for anything. faro is run on rc2_subset via a script in the bin directory there. This just runs pipetask on the obs_subaru DRP.yaml. I should reorganize the validation_data datasets (assuming we're still going to use them, which might not be feasible!) to have the necessary scripts/configs in them. A guess at the necessary steps to run on the decam/cfht validation datasets: Find out if anyone has a working DECam/CFHT DRP.yaml, and how much does it look like the obs_subaru one? Make a gen3 repo that contains the raws/calibs/refcat/skymap. Decide whether to always assume the repo is clean (i.e. a new Jenkins checkout every time) or whether to export/import the repo to be able to run repeatedly. The latter would be more future-proofed (like how testdata_jointcal is used for testing), but might be harder to actually do (e.g. calibrations). Run pipetask REPO obs_subaru/DPR.yaml#step1 and see what happens. We'll almost certainly have to override that for DECam.
Hide
John Parejko added a comment -

As part of this, we might want to move the CFHT defects out of obs_cfht, and into an obs_cfht_data, to match our convention on that.

Show
John Parejko added a comment - As part of this, we might want to move the CFHT defects out of obs_cfht, and into an obs_cfht_data, to match our convention on that.
Hide
John Parejko added a comment -

Simon Krughoff: do you mind reviewing these two tickets? It's about 80 lines of changes in obs_cfht to get it to work in gen3, and a bunch of changes and removals in validation_data_cfht demonstrating that we can run a gen3 singleframe pipeline on it. I did the minimum I thought was necessary to get jointcal to run on this data, so if you want to run faro on this output, you'll need to make more changes.

The commits should all be atomic, which should make the validation_data_cfht changes easier to view, since I changed lfs files.

Show
John Parejko added a comment - Simon Krughoff : do you mind reviewing these two tickets? It's about 80 lines of changes in obs_cfht to get it to work in gen3, and a bunch of changes and removals in validation_data_cfht demonstrating that we can run a gen3 singleframe pipeline on it. I did the minimum I thought was necessary to get jointcal to run on this data, so if you want to run faro on this output, you'll need to make more changes. The commits should all be atomic, which should make the validation_data_cfht changes easier to view, since I changed lfs files. https://github.com/lsst/obs_cfht/pull/94 https://github.com/lsst/validation_data_cfht/pull/13
Hide
John Parejko added a comment - - edited

Jenkins (though v_d_cfht isn't included and doesn't have anything to run anyway): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35449/pipeline

Show
John Parejko added a comment - - edited Jenkins (though v_d_cfht isn't included and doesn't have anything to run anyway): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35449/pipeline
Hide
Simon Krughoff added a comment -

Very sorry for taking so long to get to this. It all looks good. Tim has a comment on obs_cfht that I think you should consider

Show
Simon Krughoff added a comment - Very sorry for taking so long to get to this. It all looks good. Tim has a comment on obs_cfht that I think you should consider
Hide
Ian Sullivan added a comment -

Note that this ticket was completed using an unmerged ticket branch of DM-28862

Show
Ian Sullivan added a comment - Note that this ticket was completed using an unmerged ticket branch of DM-28862

#### People

Assignee:
John Parejko
Reporter:
Ian Sullivan
Reviewers:
Simon Krughoff
Watchers:
Ian Sullivan, John Parejko, Leanne Guy, Simon Krughoff