Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2795

Various fixes for broken code within display=True clauses and/or using --debug

    Details

      Description

      Running with the display and/or debug options turned on is revealing many instances of code that is now suffering from bit rot. This ticket will be used to track those encountered while trying to debug issues arising while porting HSC code and running processing tasks on real data.

        Attachments

          Activity

          Hide
          lauren Lauren MacArthur added a comment -

          Must update afw/ds9.py to also import "setDefaultFrame", "getDefaultFrame", "incrDefaultFrame" functions from afw/interface.py. This is so that ds9.functionName will still work with the much used import lsst.afw.display.ds9 as ds9.

          Show
          lauren Lauren MacArthur added a comment - Must update afw/ds9.py to also import "setDefaultFrame", "getDefaultFrame", "incrDefaultFrame" functions from afw/interface.py . This is so that ds9.functionName will still work with the much used import lsst.afw.display.ds9 as ds9 .
          Hide
          lauren Lauren MacArthur added a comment -

          Updates in meas_algorithms:

          In python/lsst/meas/algorithms/pcaPsfDeterminer.py:

          @@ -504,7 +504,7 @@ class PcaPsfDeterminer(object):
          -                    ds9.ds9Cmd(ds9.selectFrame(frame=7) + " ;scale limits 0 1")
          +                    ds9.scaleLimits(0, 1, frame=7)
          

          In python/lsst/meas/algorithms/utils.py:

          @@ -35,6 +35,7 @@ import lsst.afw.math as afwMath
          +import lsst.meas.base as measBase
           
          @@ -74,7 +75,7 @@ def showPsfSpatialCells(exposure, psfCellSet, nMaxPerCell=-1, 
          -            displayUtils.drawBBox(cell.getBBox(), origin=origin, frame=frame)
          +            displayUtils.drawBBox(cell.getBBox(), origin=origin, frame=frame, display=ds9)
           
          @@ -511,6 +512,10 @@ def plotPsfSpatialModel(exposure, psf, psfCellSet, 
          +        if vmin == vmax:
          +            vmin = fRange.min() - 0.05*numpy.fabs(fRange.min())
          +            vmax = fRange.max() + 0.05*numpy.fabs(fRange.max())
           
          @@ -634,16 +639,16 @@ def showPsfMosaic(exposure, psf=None, nx=7, ny=None,
          -    control = algorithmsLib.GaussianCentroidControl()
          -    centroider = algorithmsLib.MeasureSourcesBuilder().addAlgorithm(control).build(schema)
          +    control = measBase.GaussianCentroidControl()
          +    centroider = measBase.GaussianCentroidAlgorithm(control,"base_GaussianCentroid",schema)
           
          -    sdssShape = algorithmsLib.SdssShapeControl()
          -    shaper = algorithmsLib.MeasureSourcesBuilder().addAlgorithm(sdssShape).build(schema)
          -    
          -    table = afwTable.SourceTable.make(schema)
          +    sdssShape = measBase.SdssShapeControl()
          +    shaper = measBase.SdssShapeAlgorithm(sdssShape,"base_sdssShape",schema)
          +
          +    table = afwTable.SourceCatalog(schema)
           
          -    table.defineCentroid(control.name)
          -    table.defineShape(sdssShape.name)
          +    table.defineCentroid("base_GaussianCentroid")
          +    table.defineShape("base_sdssShape")
           
          @@ -669,15 +674,18 @@ def showPsfMosaic(exposure, psf=None, nx=7, ny=None,
          -            cen = afwGeom.PointD(im.getX0() + w//2, im.getY0() + h//2)
          -            src = table.makeRecord()
          +            x = im.getX0() + w//2
          +            y = im.getY0() + h//2
          +            src = table.addNew()
          +            src.set("base_GaussianCentroid_x", x)
          +            src.set("base_GaussianCentroid_y", y)
                       foot = afwDet.Footprint(exp.getBBox())
                       src.setFootprint(foot)
           
          -            centroider.apply(src, exp, cen)
          +            centroider.measure(src, exp)
                       centers.append((src.getX() - im.getX0(), src.getY() - im.getY0()))
           
          -            shaper.apply(src, exp, cen)
          +            shaper.measure(src, exp)
          

          Show
          lauren Lauren MacArthur added a comment - Updates in meas_algorithms : In python/lsst/meas/algorithms/pcaPsfDeterminer.py : @@ -504,7 +504,7 @@ class PcaPsfDeterminer(object): - ds9.ds9Cmd(ds9.selectFrame(frame=7) + " ;scale limits 0 1") + ds9.scaleLimits(0, 1, frame=7) In python/lsst/meas/algorithms/utils.py : @@ -35,6 +35,7 @@ import lsst.afw.math as afwMath +import lsst.meas.base as measBase   @@ -74,7 +75,7 @@ def showPsfSpatialCells(exposure, psfCellSet, nMaxPerCell=-1, - displayUtils.drawBBox(cell.getBBox(), origin=origin, frame=frame) + displayUtils.drawBBox(cell.getBBox(), origin=origin, frame=frame, display=ds9)   @@ -511,6 +512,10 @@ def plotPsfSpatialModel(exposure, psf, psfCellSet, + if vmin == vmax: + vmin = fRange.min() - 0.05*numpy.fabs(fRange.min()) + vmax = fRange.max() + 0.05*numpy.fabs(fRange.max())   @@ -634,16 +639,16 @@ def showPsfMosaic(exposure, psf=None, nx=7, ny=None, - control = algorithmsLib.GaussianCentroidControl() - centroider = algorithmsLib.MeasureSourcesBuilder().addAlgorithm(control).build(schema) + control = measBase.GaussianCentroidControl() + centroider = measBase.GaussianCentroidAlgorithm(control,"base_GaussianCentroid",schema) - sdssShape = algorithmsLib.SdssShapeControl() - shaper = algorithmsLib.MeasureSourcesBuilder().addAlgorithm(sdssShape).build(schema) - - table = afwTable.SourceTable.make(schema) + sdssShape = measBase.SdssShapeControl() + shaper = measBase.SdssShapeAlgorithm(sdssShape,"base_sdssShape",schema) + + table = afwTable.SourceCatalog(schema) - table.defineCentroid(control.name) - table.defineShape(sdssShape.name) + table.defineCentroid("base_GaussianCentroid") + table.defineShape("base_sdssShape")   @@ -669,15 +674,18 @@ def showPsfMosaic(exposure, psf=None, nx=7, ny=None, - cen = afwGeom.PointD(im.getX0() + w//2, im.getY0() + h//2) - src = table.makeRecord() + x = im.getX0() + w//2 + y = im.getY0() + h//2 + src = table.addNew() + src.set("base_GaussianCentroid_x", x) + src.set("base_GaussianCentroid_y", y) foot = afwDet.Footprint(exp.getBBox()) src.setFootprint(foot) - centroider.apply(src, exp, cen) + centroider.measure(src, exp) centers.append((src.getX() - im.getX0(), src.getY() - im.getY0())) - shaper.apply(src, exp, cen) + shaper.measure(src, exp)
          Hide
          lauren Lauren MacArthur added a comment - - edited

          Updates in meas_extensions_psfex:

          In python/lsst/meas/extensions/psfex/psfexPsfDeterminer.py:

          @@ -431,7 +431,7 @@ class PsfexPsfDeterminer(object):
          -                ds9.ds9Cmd(ds9.selectFrame(frame=7) + " ;scale limits 0 1")
          +                ds9.scale('linear', 0, 1, frame=7)
          

          This has been implemented as part of DM-3678.

          Show
          lauren Lauren MacArthur added a comment - - edited Updates in meas_extensions_psfex : In python/lsst/meas/extensions/psfex/psfexPsfDeterminer.py : @@ -431,7 +431,7 @@ class PsfexPsfDeterminer(object): - ds9.ds9Cmd(ds9.selectFrame(frame=7) + " ;scale limits 0 1") + ds9.scale('linear', 0, 1, frame=7) This has been implemented as part of DM-3678 .
          Hide
          swinbank John Swinbank added a comment -

          Mandeep S.S. Gill writes:

          from what i can tell, from Lauren and Robert’s fixes in 2015:

          https://github.com/lsst/meas_algorithms/commit/0eee5d4008ce55fef3a202748e1de3671fc84903
          https://github.com/lsst/meas_algorithms/commit/601469051509bf3214b1d6f9e53a03016e20b75c

          the upshot is that https://jira.lsstcorp.org/browse/DM-2795 is now fixed and ought be marked resolved (i think?).

          --------- Details:

          With a recent local stack install (mid-April 2017), once i do:

          setup pipe_tasks
          

          and

          setup display_ds9
          

          From $PIPE_TASKS/examples, i’m able to run:

          ./exampleCmdLineTask.py ${OBS_TEST_DIR}/data/input --id visit=1 --debug --clobber-versions --output outdirexample/
          

          with the following in my debug.py file in the same dir (which i have added by hand to my $PYTHONPATH):

          import lsstDebug
          def DebugInfo(name):
              di = lsstDebug.getInfo(name)
              di.display = True
              return di
          lsstDebug.Info = DebugInfo
          

          and it runs fine, with the following screen output:

          CameraMapper INFO: Unable to locate registry registry in root: registry.sqlite3
          CameraMapper INFO: Unable to locate registry registry in current dir: ./registry.sqlite3
          root INFO: Running: ./exampleCmdLineTask.py /Users/m/fizzAndAstro/lsst/lsstsw/stack/DarwinX86/obs_test/13.0-6-g7b63e3f/data/input --id visit=1 --debug --clobber-versions --output outdirexample/
          exampleTask INFO: Processing data ID {u'filter': 'g', 'visit': 1}
          exampleTask.stats INFO: clipped mean=1184.70; meanErr=0.02; stdDev=33.64; stdDevErr=1.04
          

          and then also pops up a ds9 window with 4 frames in it.

          Then as a further test, i cloned https://github.com/lsst/lsst_dm_stack_demo (because downloading it from the pipelines page brings up an old version looks like it has a bug in it

          [“KeyError: “Field with name ‘flags_negative’ not found”]
          

          , which i’ve pointed out to Jonathan but not heard back on yet), and then run from this dir:

          ./bin/demo.sh`
          

          with the following in my debug.py file in the same dir (which again i added by hand to my $PYTHONPATH):

          import lsstDebug
          def DebugInfo(name):
              di = lsstDebug.getInfo(name)
              if name == "lsst.ip.isrFunctions.isrTask":
                  di.display = {'postISRCCD':2}
          #        di.display = True
          #    if name == "lsst.pipe.tasks.characterizeImage":
          #          di.display = dict(
          #          repair = True,
          #     )
          #    if name == "lsst.pipe.tasks.calibrate":
          #          di.display = dict(
          #          calibrate = 1,
          #     )
              
              return di
          lsstDebug.Info = DebugInfo
          

          and where i in turn commented out the various sections (to see debugging work for Isr, CharImg, and CalibImg). It runs through without a hitch (though doesn’t pop up a ds9 window in these cases).

          Show
          swinbank John Swinbank added a comment - Mandeep S.S. Gill writes: from what i can tell, from Lauren and Robert’s fixes in 2015: https://github.com/lsst/meas_algorithms/commit/0eee5d4008ce55fef3a202748e1de3671fc84903 https://github.com/lsst/meas_algorithms/commit/601469051509bf3214b1d6f9e53a03016e20b75c the upshot is that https://jira.lsstcorp.org/browse/DM-2795 is now fixed and ought be marked resolved (i think?). --------- Details: With a recent local stack install (mid-April 2017), once i do: setup pipe_tasks and setup display_ds9 From $PIPE_TASKS/examples , i’m able to run: ./exampleCmdLineTask.py ${OBS_TEST_DIR}/data/input --id visit=1 --debug --clobber-versions --output outdirexample/ with the following in my debug.py file in the same dir (which i have added by hand to my $PYTHONPATH ): import lsstDebug def DebugInfo(name): di = lsstDebug.getInfo(name) di.display = True return di lsstDebug.Info = DebugInfo and it runs fine, with the following screen output: CameraMapper INFO: Unable to locate registry registry in root: registry.sqlite3 CameraMapper INFO: Unable to locate registry registry in current dir: ./registry.sqlite3 root INFO: Running: ./exampleCmdLineTask.py /Users/m/fizzAndAstro/lsst/lsstsw/stack/DarwinX86/obs_test/13.0-6-g7b63e3f/data/input --id visit=1 --debug --clobber-versions --output outdirexample/ exampleTask INFO: Processing data ID {u'filter': 'g', 'visit': 1} exampleTask.stats INFO: clipped mean=1184.70; meanErr=0.02; stdDev=33.64; stdDevErr=1.04 and then also pops up a ds9 window with 4 frames in it. Then as a further test, i cloned https://github.com/lsst/lsst_dm_stack_demo (because downloading it from the pipelines page brings up an old version looks like it has a bug in it [“KeyError: “Field with name ‘flags_negative’ not found”] , which i’ve pointed out to Jonathan but not heard back on yet), and then run from this dir: ./bin/demo.sh` with the following in my debug.py file in the same dir (which again i added by hand to my $PYTHONPATH): import lsstDebug def DebugInfo(name): di = lsstDebug.getInfo(name) if name == "lsst.ip.isrFunctions.isrTask": di.display = {'postISRCCD':2} # di.display = True # if name == "lsst.pipe.tasks.characterizeImage": # di.display = dict( # repair = True, # ) # if name == "lsst.pipe.tasks.calibrate": # di.display = dict( # calibrate = 1, # ) return di lsstDebug.Info = DebugInfo and where i in turn commented out the various sections (to see debugging work for Isr, CharImg, and CalibImg). It runs through without a hitch (though doesn’t pop up a ds9 window in these cases).
          Hide
          swinbank John Swinbank added a comment -

          Thanks Mandeep; I agree that this is done.

          Show
          swinbank John Swinbank added a comment - Thanks Mandeep; I agree that this is done.

            People

            • Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              John Swinbank
              Watchers:
              John Swinbank, Lauren MacArthur
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel