Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30738

Backport gen3 imSim BF-kernels for gen2

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production

      Attachments

        Issue Links

          Activity

          Hide
          czw Christopher Waters added a comment -

          The new-style kernels need to be copied from /project/czw/DM-30738/newBfKernels to /datasets/DC2/repoRun2.2i/CALIB/bfk.  This will match the new mapper definition for bfKernel, which will now look with a template: bfk/bfKernel-%(raftName)s_%(detectorName)s.fits.

          The ip_isr change was necessary to get the updates done on DM-30730 in runQuantum propagated to readIsrData.

          Finally, this was tested with a template of ../rerun/private/czw/bfk/bfKernel-%(raftName)s_%(detectorName)s.fits to avoid various mapper issues.  This successfully found and used the kernel

          processCcd.isr INFO: Applying brighter-fatter correction using kernel type <class 'numpy.ndarray'> / gains <class 'NoneType'>.
          processCcd.isr INFO: Finished brighter-fatter correction in 2 iterations.

          but then crashed with a segfault at

          processCcd.charImage.measurePsf.psfDeterminer WARN: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
          processCcd.charImage.measurePsf INFO: PSF determination using 63/63 stars.
          Caught signal 11, backtrace follows:
          /home/czw/dev/DM-30738/utils/lib/libutils.so(+0x26389) [0x7f51f0eae389]
          /lib64/libc.so.6(+0x36450) [0x7f51fb61d450]
          /home/czw/dev/DM-30738/afw/python/lsst/afw/detection/_detection.so(+0x3b2c4) [0x7f51eaeaa2c4]

          As I needed to rebuild five extra packages to get obs_lsst to build, I'm hoping this is just something in my environment being broken.  An earlier segfault (in CR detection) was fixed with a rebuild of meas_algorithms, afw, and utils from the current master.

          Jenkins build to check that HSC BF correction is unaffected here: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34458/pipeline

          Show
          czw Christopher Waters added a comment - The new-style kernels need to be copied from /project/czw/ DM-30738 /newBfKernels to /datasets/DC2/repoRun2.2i/CALIB/bfk .  This will match the new mapper definition for bfKernel , which will now look with a template: bfk/bfKernel-%(raftName)s_%(detectorName)s.fits . The ip_isr change was necessary to get the updates done on DM-30730 in runQuantum propagated to readIsrData . Finally, this was tested with a template of ../rerun/private/czw/bfk/bfKernel-%(raftName)s_%(detectorName)s.fits to avoid various mapper issues.  This successfully found and used the kernel processCcd.isr INFO: Applying brighter-fatter correction using kernel type < class 'numpy.ndarray' > / gains < class 'NoneType' >. processCcd.isr INFO: Finished brighter-fatter correction in 2 iterations. but then crashed with a segfault at processCcd.charImage.measurePsf.psfDeterminer WARN: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value processCcd.charImage.measurePsf INFO: PSF determination using 63 / 63 stars. Caught signal 11 , backtrace follows: /home/czw/dev/DM- 30738 /utils/lib/libutils.so(+ 0x26389 ) [ 0x7f51f0eae389 ] /lib64/libc.so. 6 (+ 0x36450 ) [ 0x7f51fb61d450 ] /home/czw/dev/DM- 30738 /afw/python/lsst/afw/detection/_detection.so(+ 0x3b2c4 ) [ 0x7f51eaeaa2c4 ] As I needed to rebuild five extra packages to get obs_lsst to build, I'm hoping this is just something in my environment being broken.  An earlier segfault (in CR detection) was fixed with a rebuild of meas_algorithms , afw , and utils from the current master. Jenkins build to check that HSC BF correction is unaffected here:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34458/pipeline
          Hide
          lauren Lauren MacArthur added a comment -

          I just did a quick test of processCcd on a single HSC visit/detector and the output calexps look identical on master vs. your branch.

          Show
          lauren Lauren MacArthur added a comment - I just did a quick test of processCcd  on a single HSC visit/detector and the output calexps look identical on master vs. your branch.
          Hide
          lauren Lauren MacArthur added a comment -

          Ok, I may have taken a bit of a "liberty" here, but I went ahead and copied over your new kernels into /datasets/DC2/repoRun2.2i/CALIB/bfk and ran a visit with your branches setup. Here's what things looked like on the w_2021_16 gen2 DC2 run:

          and...with your branches and new kernels in the right place:

          Booyah!

          Show
          lauren Lauren MacArthur added a comment - Ok, I may have taken a bit of a "liberty" here, but I went ahead and copied over your new kernels into  /datasets/DC2/repoRun2.2i/CALIB/bfk and ran a visit with your branches setup. Here's what things looked like on the w_2021_16 gen2 DC2 run: and...with your branches and new kernels in the right place: Booyah!
          Hide
          lauren Lauren MacArthur added a comment -

          LGTM!

          Show
          lauren Lauren MacArthur added a comment - LGTM!
          Hide
          czw Christopher Waters added a comment -

          Great!  I'll get this merged then.

          Show
          czw Christopher Waters added a comment - Great!  I'll get this merged then.

            People

            Assignee:
            czw Christopher Waters
            Reporter:
            czw Christopher Waters
            Reviewers:
            Lauren MacArthur
            Watchers:
            Christopher Waters, Lauren MacArthur
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.