# PTC task not finding linearizer

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
0.5
• Team:
Data Release Production
• Urgent?:
No

#### Description

I'm trying to run the PTC task with the linearizer. I successfully created the linearizers and certified them. I know they are there, because I can butler.get them with the same set of collections and plot them. I have set doLinearize: true in the yaml file. When I tried to run it, the results were the same as with doLinearize: false. Merlin Fisher-Levine pointed out that when the code can't find the linearizer, it just creates a default linearizer. So there are really two issues:

(1) Why is the code not finding the linearizer?
(2) Why does it just drive on and use the default when doLinearize is true?

Below is the set of pipetasks I ran:

pipetask run -j 16 -d "detector IN (0..188) AND instrument='LSSTCam' AND exposure=3021120600576" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/13144,u/cslage/bps_13144B -o u/cslage/bps_13144D -p /project/cslage/BOT_LSSTCam/pipelines/cpLinearitySolve.yaml --register-dataset-types

butler certify-calibrations /repo/main u/cslage/bps_13144D LSSTCam/calib/u/cslage/13144 --begin-date 1980-01-01 --end-date 2050-01-01 linearity

pipetask run -j 32 -d "detector IN (55,74) AND instrument='LSSTCam' AND exposure IN (3021120600575..3021120700825) AND exposure.observation_type='flat'" -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/calib/u/cslage/13144 -o u/cslage/bps_13144D -p /project/cslage/BOT_LSSTCam/pipelines/measurePhotonTransferCurve.yaml --register-dataset-types

#### Activity

Hide
Merlin Fisher-Levine added a comment -

Christopher Waters it seems like the draft PR I've made might be all that's needed here, but could you just give that a look and see if that looks like it will break anything else?

Show
Merlin Fisher-Levine added a comment - Christopher Waters it seems like the draft PR I've made might be all that's needed here, but could you just give that a look and see if that looks like it will break anything else?
Hide
Craig Lage added a comment -

OK, I think we understand what happened. Merlin and I hypothesized that the problem is that when calculating and certifying the linearizer, it is called "linearity", but the ISR task is looking for "linearizer" . So I hacked line 202 in isrTask.py to change the name from "linearizer" to "linearity, and then it successfully found the linearizer and applied the linearity correction when doing the ISR.

But then I found the root cause of the problem. I copied the yaml file from $CP_PIPE_DIR/pipelines/cpLinearitySolve.yaml. It looks like this:  description: cp_pipe linearity calibration construction. tasks:  linearitySolve:  class: lsst.cp.pipe.LinearitySolveTask  config:  connections.inputPtc: ptc  connections.outputLinearizer: linearity  If I change the last lline to be "linearizer" instead of "linearity", then it runs OK, I can certify it as "linearizer", and then the ISR task finds it as "linearizer". Merlin has submitted a PR to update the yaml file. Chris Waters also has to update the calibration document, which says to certify it as "linearity". This document is at: https://lsst.ncsa.illinois.edu/~czw/pipelines_lsst_io/_build/html/modules/lsst.cp.pipe/constructing-calibrations.html#constructing-a-linearity-correction Show Craig Lage added a comment - OK, I think we understand what happened. Merlin and I hypothesized that the problem is that when calculating and certifying the linearizer, it is called "linearity", but the ISR task is looking for "linearizer" . So I hacked line 202 in isrTask.py to change the name from "linearizer" to "linearity, and then it successfully found the linearizer and applied the linearity correction when doing the ISR. But then I found the root cause of the problem. I copied the yaml file from$CP_PIPE_DIR/pipelines/cpLinearitySolve.yaml. It looks like this: description: cp_pipe linearity calibration construction. tasks: linearitySolve: class: lsst.cp.pipe.LinearitySolveTask config: connections.inputPtc: ptc connections.outputLinearizer: linearity If I change the last lline to be "linearizer" instead of "linearity", then it runs OK, I can certify it as "linearizer", and then the ISR task finds it as "linearizer". Merlin has submitted a PR to update the yaml file. Chris Waters also has to update the calibration document, which says to certify it as "linearity". This document is at: https://lsst.ncsa.illinois.edu/~czw/pipelines_lsst_io/_build/html/modules/lsst.cp.pipe/constructing-calibrations.html#constructing-a-linearity-correction
Hide
Christopher Waters added a comment -

I think this is just me misremembering the dataset_type_name.  This ticket should also change the calibration documentation (it's in cp_pipe as well.

Doing that will ensure that the "proper" documentation is updated: https://pipelines.lsst.io/v/daily/modules/lsst.cp.pipe/constructing-calibrations.html

Show
Christopher Waters added a comment - I think this is just me misremembering the dataset_type_name .  This ticket should also change the calibration documentation (it's in cp_pipe as well. Doing that will ensure that the "proper" documentation is updated:  https://pipelines.lsst.io/v/daily/modules/lsst.cp.pipe/constructing-calibrations.html
Hide
Merlin Fisher-Levine added a comment -

Christopher Waters I think/hope I've done it right, but could you check that's what you meant please?

Craig Lage Could you just take a quick glance at the PR and check/confirm there wasn't anything else you had to change to make things work please?

Show
Merlin Fisher-Levine added a comment - Christopher Waters I think/hope I've done it right, but could you check that's what you meant please? Craig Lage Could you just take a quick glance at the PR and check/confirm there wasn't anything else you had to change to make things work please?
Hide
Christopher Waters added a comment -

I think this is clear after the rebase.

Show
Christopher Waters added a comment - I think this is clear after the rebase.
Hide
Merlin Fisher-Levine added a comment -

Great, thanks both for comments on the PR. Have rebased, am running Jenkins now and will merge once it's cooked.

Show
Merlin Fisher-Levine added a comment - Great, thanks both for comments on the PR. Have rebased, am running Jenkins now and will merge once it's cooked.

#### People

Assignee:
Merlin Fisher-Levine
Reporter:
Craig Lage
Reviewers:
Christopher Waters, Craig Lage
Watchers:
Christopher Waters, Craig Lage, Merlin Fisher-Levine
0 Vote for this issue
Watchers:
3 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### Jenkins

No builds found.