Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17471

Save intermediate data product with pre-interpolation pixel values

    XMLWordPrintable

    Details

      Description

      An easy path towards supporting DESC's desire to look at pre-interpolation pixels would be to save a mostly-zeros MaskedImage with those values as a separate intermediate data product somewhere in processCcd.py.  Exactly what values to save (as a diff? with or without background? with or without a mask?) is still slightly TBD.

      Should come with a Python snippet to combine this image with a calexp and/or postISRCCD, ideally with options to control what to re-insert by mask plane.

      Storing these as HeavyFootprints with the Exposure may ultimately be a better solution to this problem, but is sufficiently harder to implement that we should at least do this first, and evaluate whether we need that after we've tried it out.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Colin Slater: we are unsure of where we stand on this issue. Are you aware if it has been addressed or become moot, or is there still work to be done here?

            Show
            lauren Lauren MacArthur added a comment - Colin Slater : we are unsure of where we stand on this issue. Are you aware if it has been addressed or become moot, or is there still work to be done here?
            Hide
            ctslater Colin Slater added a comment -

            I'm pretty sure that this hasn't been addressed and is also not moot, as the use case hasn't gone away. I assume we have not yet reached the ISR reordering described in Yusra's option C? That seems like the scenario in which this would be easy enough to be worthwhile.

            Show
            ctslater Colin Slater added a comment - I'm pretty sure that this hasn't been addressed and is also not moot, as the use case hasn't gone away. I assume we have not yet reached the ISR reordering described in Yusra's option C? That seems like the scenario in which this would be easy enough to be worthwhile.
            Hide
            lauren Lauren MacArthur added a comment -

            Having been "in" this code lately, I actually think the functionality was indeed added on DM-19382.  See in particular the doSaveInterpPixels config value that was added on this commit.  Perhaps Christopher Waters can confirm and, if we all agree, I think we can mark this as invalid?

            Show
            lauren Lauren MacArthur added a comment - Having been "in" this code lately, I actually think the functionality was indeed added on DM-19382 .  See in particular the doSaveInterpPixels config value  that was added on this commit .  Perhaps Christopher Waters  can confirm and, if we all agree, I think we can mark this as invalid?
            Hide
            czw Christopher Waters added a comment -

            I had thought that this was solved, and in attempting a test, realized that there's a typo in the gen3 handling.  Running the command:

            pipetask --long-log run -b /repo/main -i HSC/calib,HSC/raw/all -o u/czw/DM-17471/test.20210707 -d "instrument='HSC' AND detector=24 AND exposure=36234" -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#isr -c isr:doSaveInterpPixels=True --register-dataset-types  >&  ../interp_test.20210707.log

            followed by grabbing the two flavors of exposure and differencing them yields the attached image.

            Show
            czw Christopher Waters added a comment - I had thought that this was solved, and in attempting a test, realized that there's a typo in the gen3 handling.  Running the command: pipetask -- long -log run -b /repo/main -i HSC/calib,HSC/raw/all -o u/czw/DM- 17471 /test. 20210707 -d "instrument='HSC' AND detector=24 AND exposure=36234" -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#isr -c isr:doSaveInterpPixels=True --register-dataset-types >& ../interp_test. 20210707 .log followed by grabbing the two flavors of exposure and differencing them yields the attached image.
            Hide
            czw Christopher Waters added a comment - - edited
            Show
            czw Christopher Waters added a comment - - edited Jenkins run checking that I didn't make a mistake with the six characters I removed:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34581/pipeline ETA: version that actually succeeded:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34583/pipeline

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Arun Kannawadi
              Watchers:
              Arun Kannawadi, Christopher Waters, Colin Slater, Jim Bosch, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.