HSC's ApTemplate pipeline is missing skyCorr step

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
2
• Sprint:
AP S22-5 (April)
• Team:
• Urgent?:
No

Description

When Karlo Mrakovcic was trying to build HSC goodSeeing coadds for use as AP templates, he ran into a problem with no skyCorr dataset. This ticket is to update the HSC-specific ApTemplate pipeline to fix this, and test it to make sure nothin awful happened. (While I'm at it, I should probably fix the comment in the subset definition area that errantly refers to DECam... oy).

Options include: (1) adding the skyCorr step or (2) turning skyCorr off in makeWarp. If (1) is chosen, it's important to keep in mind that skyCorr does not play well with patch and tract dataIds, per comments in the DRP-RC2 pipeline. So unless something changes in that regard and/or we desperately need skyCorr, it would be simplest to just explicitly turn it off in ApTemplate.

Relevant line to potentially add to pipeline:

  skyCorr: # This comes from the obs_subaru DRP.yaml. Alternative is doApplySkyCorr: False in makeWarp.  class: lsst.pipe.drivers.skyCorrection.SkyCorrectionTask

Activity

Hide
Krzysztof Findeisen added a comment -

Just to be clear, is skyCorr something that's only needed for HSC, or should it be part of template generation in general (i.e., everything except DECam)?

Show
Krzysztof Findeisen added a comment - Just to be clear, is skyCorr something that's only needed for HSC, or should it be part of template generation in general (i.e., everything except DECam)?
Hide
Meredith Rawls added a comment -

A wonderful question that I do not know the answer to!

Show
Meredith Rawls added a comment - A wonderful question that I do not know the answer to!
Hide
Lee Kelvin added a comment -

Krzysztof Findeisen, currently we only have sky frames constructed for HSC data. The sky dataset types which live in HSC/calib/DM-32378 are one of the two inputs required (alongside raw's) to produce the skyCorr dataset type using the SkyCorrectionTask. As such, full-focal-plane sky correction is only supported for HSC data reductions at present.

With that said, we may want to generalize this task in the future to work with both LSST data, and potentially other survey data such as DECam. From discussions I've had with other survey team members, it seems like this form of sky correction is used elsewhere such as in the Community Pipeline for DECam, and so might be worth considering for the science pipelines as well.

Show
Lee Kelvin added a comment - Krzysztof Findeisen , currently we only have sky frames constructed for HSC data. The sky dataset types which live in HSC/calib/ DM-32378 are one of the two inputs required (alongside raw 's) to produce the skyCorr dataset type using the SkyCorrectionTask . As such, full-focal-plane sky correction is only supported for HSC data reductions at present. With that said, we may want to generalize this task in the future to work with both LSST data, and potentially other survey data such as DECam. From discussions I've had with other survey team members, it seems like this form of sky correction is used elsewhere such as in the Community Pipeline for DECam, and so might be worth considering for the science pipelines as well.
Hide
Meredith Rawls added a comment -

Thanks very much for the additional context, Lee!

For now, I think it makes sense to just add the skyCorrectionTask step to HSC's ApTemplate.yaml. This is similar to how only the DRP-RC2.yaml has this step, vanilla DRP.yaml does not.

I updated the pipeline and verified that this successfully begins to process data on lsst-devl:

 pipetask run -j 4 -b /repo/main -i HSC/RC2/defaults -o u/mrawls/DM-33692-take1 -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -d "skymap='hsc_rings_v1' AND exposure IN (30482, 30484, 30486)"  Karlo Mrakovcic, would you be willing to review this ticket, since you originally noticed the problem? In this situation, I think a review consists of (1) looking at the pipeline/code changes on the ticket branch and agreeing it makes sense, and (2) trying steps similar to what you tried before to make sure the problem is resolved. (More info on our code review process: https://developer.lsst.io/work/flow.html?#workflow-code-review). I am not running Jenkins for this small change, because Jenkins does not run or test pipelines in the scope of its CI. Show Meredith Rawls added a comment - Thanks very much for the additional context, Lee! For now, I think it makes sense to just add the skyCorrectionTask step to HSC's ApTemplate.yaml. This is similar to how only the DRP-RC2.yaml has this step, vanilla DRP.yaml does not. I updated the pipeline and verified that this successfully begins to process data on lsst-devl: pipetask run -j 4 -b /repo/main -i HSC/RC2/defaults -o u/mrawls/DM- 33692 -take1 -p$AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -d "skymap='hsc_rings_v1' AND exposure IN (30482, 30484, 30486)" Karlo Mrakovcic , would you be willing to review this ticket, since you originally noticed the problem? In this situation, I think a review consists of (1) looking at the pipeline/code changes on the ticket branch and agreeing it makes sense, and (2) trying steps similar to what you tried before to make sure the problem is resolved. (More info on our code review process: https://developer.lsst.io/work/flow.html?#workflow-code-review ). I am not running Jenkins for this small change, because Jenkins does not run or test pipelines in the scope of its CI.
Hide
Karlo Mrakovcic added a comment - - edited

I used this command

 pipetask run -j 4 -b /repo/main -d "instrument='HSC' AND skymap='hsc_rings_v1' AND tract=9813 AND band='g' AND patch=50" -i HSC/RC2/defaults -o u/kmrakovcic/DM-33692-take2 -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml  The process successfully ended. Thank you for your patience. Show Karlo Mrakovcic added a comment - - edited I used this command pipetask run -j 4 -b /repo/main -d "instrument='HSC' AND skymap='hsc_rings_v1' AND tract=9813 AND band='g' AND patch=50" -i HSC/RC2/defaults -o u/kmrakovcic/DM- 33692 -take2 -p$AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml The process successfully ended. Thank you for your patience.
Hide
Meredith Rawls added a comment -

Thanks for the review! You errantly assigned yourself to the ticket (you're the reviewer, not the assignee) and put it all the way to Done instead of just Reviewed - no problem. I'm supposed to move it to Done once I actually merge the ticket, which I will do now. Glad your test run ran through to completion - awesome!

Show
Meredith Rawls added a comment - Thanks for the review! You errantly assigned yourself to the ticket (you're the reviewer, not the assignee) and put it all the way to Done instead of just Reviewed - no problem. I'm supposed to move it to Done once I actually merge the ticket, which I will do now. Glad your test run ran through to completion - awesome!

People

Assignee:
Meredith Rawls
Reporter:
Meredith Rawls
Reviewers:
Karlo Mrakovcic
Watchers:
Ian Sullivan, Karlo Mrakovcic, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls