# Save intermediate data product with pre-interpolation pixel values

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Team:
Data Release Production

#### Description

An easy path towards supporting DESC's desire to look at pre-interpolation pixels would be to save a mostly-zeros MaskedImage with those values as a separate intermediate data product somewhere in processCcd.py.  Exactly what values to save (as a diff? with or without background? with or without a mask?) is still slightly TBD.

Should come with a Python snippet to combine this image with a calexp and/or postISRCCD, ideally with options to control what to re-insert by mask plane.

Storing these as HeavyFootprints with the Exposure may ultimately be a better solution to this problem, but is sufficiently harder to implement that we should at least do this first, and evaluate whether we need that after we've tried it out.

#### Attachments

1. interp_difference.png
13 kB

#### Activity

Hide
Lauren MacArthur added a comment -

Colin Slater: we are unsure of where we stand on this issue. Are you aware if it has been addressed or become moot, or is there still work to be done here?

Show
Lauren MacArthur added a comment - Colin Slater : we are unsure of where we stand on this issue. Are you aware if it has been addressed or become moot, or is there still work to be done here?
Hide
Colin Slater added a comment -

I'm pretty sure that this hasn't been addressed and is also not moot, as the use case hasn't gone away. I assume we have not yet reached the ISR reordering described in Yusra's option C? That seems like the scenario in which this would be easy enough to be worthwhile.

Show
Colin Slater added a comment - I'm pretty sure that this hasn't been addressed and is also not moot, as the use case hasn't gone away. I assume we have not yet reached the ISR reordering described in Yusra's option C? That seems like the scenario in which this would be easy enough to be worthwhile.
Hide
Lauren MacArthur added a comment -

Having been "in" this code lately, I actually think the functionality was indeed added on DM-19382.  See in particular the doSaveInterpPixels config value that was added on this commit.  Perhaps Christopher Waters can confirm and, if we all agree, I think we can mark this as invalid?

Show
Lauren MacArthur added a comment - Having been "in" this code lately, I actually think the functionality was indeed added on DM-19382 .  See in particular the doSaveInterpPixels config value  that was added on this commit .  Perhaps Christopher Waters  can confirm and, if we all agree, I think we can mark this as invalid?
Hide
Christopher Waters added a comment -

I had thought that this was solved, and in attempting a test, realized that there's a typo in the gen3 handling.  Running the command:

 pipetask --long-log run -b /repo/main -i HSC/calib,HSC/raw/all -o u/czw/DM-17471/test.20210707 -d "instrument='HSC' AND detector=24 AND exposure=36234" -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#isr -c isr:doSaveInterpPixels=True --register-dataset-types >& ../interp_test.20210707.log followed by grabbing the two flavors of exposure and differencing them yields the attached image. Show Christopher Waters added a comment - I had thought that this was solved, and in attempting a test, realized that there's a typo in the gen3 handling. Running the command: pipetask -- long -log run -b /repo/main -i HSC/calib,HSC/raw/all -o u/czw/DM- 17471 /test. 20210707 -d "instrument='HSC' AND detector=24 AND exposure=36234" -p$OBS_SUBARU_DIR/pipelines/DRP.yaml#isr -c isr:doSaveInterpPixels=True --register-dataset-types >& ../interp_test. 20210707 .log followed by grabbing the two flavors of exposure and differencing them yields the attached image.
Hide
Christopher Waters added a comment - - edited
Show
Christopher Waters added a comment - - edited Jenkins run checking that I didn't make a mistake with the six characters I removed:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34581/pipeline ETA: version that actually succeeded:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34583/pipeline

#### People

Assignee:
Christopher Waters
Reporter:
Jim Bosch
Reviewers:
Watchers:
Arun Kannawadi, Christopher Waters, Colin Slater, Jim Bosch, Lauren MacArthur, Yusra AlSayyad