Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ip_isr
-
Labels:None
-
Story Points:0.5
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
I'm trying to run the PTC task with the linearizer. I successfully created the linearizers and certified them. I know they are there, because I can butler.get them with the same set of collections and plot them. I have set doLinearize: true in the yaml file. When I tried to run it, the results were the same as with doLinearize: false. Merlin Fisher-Levine pointed out that when the code can't find the linearizer, it just creates a default linearizer. So there are really two issues:
(1) Why is the code not finding the linearizer?
(2) Why does it just drive on and use the default when doLinearize is true?
Below is the set of pipetasks I ran:
pipetask run -j 16 -d "detector IN (0..188) AND instrument='LSSTCam' AND exposure=3021120600576" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/13144,u/cslage/bps_13144B -o u/cslage/bps_13144D -p /project/cslage/BOT_LSSTCam/pipelines/cpLinearitySolve.yaml --register-dataset-types
butler certify-calibrations /repo/main u/cslage/bps_13144D LSSTCam/calib/u/cslage/13144 --begin-date 1980-01-01 --end-date 2050-01-01 linearity
pipetask run -j 32 -d "detector IN (55,74) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/13144 -o u/cslage/bps_13144D -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types
Attachments
Activity
OK, I think we understand what happened. Merlin and I hypothesized that the problem is that when calculating and certifying the linearizer, it is called "linearity", but the ISR task is looking for "linearizer" . So I hacked line 202 in isrTask.py to change the name from "linearizer" to "linearity, and then it successfully found the linearizer and applied the linearity correction when doing the ISR.
But then I found the root cause of the problem. I copied the yaml file from $CP_PIPE_DIR/pipelines/cpLinearitySolve.yaml. It looks like this:
description: cp_pipe linearity calibration construction.
|
tasks:
|
linearitySolve:
|
class: lsst.cp.pipe.LinearitySolveTask
|
config:
|
connections.inputPtc: ptc
|
connections.outputLinearizer: linearity
|
If I change the last lline to be "linearizer" instead of "linearity", then it runs OK, I can certify it as "linearizer", and then the ISR task finds it as "linearizer". Merlin has submitted a PR to update the yaml file. Chris Waters also has to update the calibration document, which says to certify it as "linearity". This document is at:
https://lsst.ncsa.illinois.edu/~czw/pipelines_lsst_io/_build/html/modules/lsst.cp.pipe/constructing-calibrations.html#constructing-a-linearity-correction
I think this is just me misremembering the dataset_type_name. This ticket should also change the calibration documentation (it's in cp_pipe as well.
Doing that will ensure that the "proper" documentation is updated: https://pipelines.lsst.io/v/daily/modules/lsst.cp.pipe/constructing-calibrations.html
Christopher Waters I think/hope I've done it right, but could you check that's what you meant please?
Craig Lage Could you just take a quick glance at the PR and check/confirm there wasn't anything else you had to change to make things work please?
I think this is clear after the rebase.
Great, thanks both for comments on the PR. Have rebased, am running Jenkins now and will merge once it's cooked.
Christopher Waters it seems like the draft PR I've made might be all that's needed here, but could you just give that a look and see if that looks like it will break anything else?