Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11356

Confusion of refObjLoader.pixelMargin and matcher.maxOffsetPix

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      External

      Description

      A recent change in AstrometryTask.solve increases the exposure bounding box by the matcher's maximum offset (matcher.maxOffsetPix. However, a parameter already exists to provide this growth in the reference catalog loader (refObjLoader.pixelMargin). Because the exposure bounding box is increased artificially after recording the catalog loading radius, loading the normalised persisted matches results in many missed matches.

      To fix this:

      • Remove the additional grow
      • Set the default refObjLoader.pixelMargin to match the default matcher.maxOffsetPix.
      • Add notes advising the user that the two parameters should be linked.
      • Stop AstrometryTask from reaching into the responsibility of the reference catalog loader (it shouldn't need to know about the details of the refObjLoader configuration).

        Attachments

          Issue Links

            Activity

            Hide
            danielsf Scott Daniel added a comment -

            ============================= test session starts ==============================
            platform linux -- Python 3.5.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0
            rootdir: /astro/users/danielsf/lsstsw3/build/meas_algorithms, inifile:
            plugins: session2file-0.1.9, xdist-1.19.2.dev0+g459d52e.d20170831, forked-0.3.dev0+g1dd93f6.d20170831, flake8-0.8.1
            gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
            gw0 [162] / gw1 [162] / gw2 [162] / gw3 [162] / gw4 [162] / gw5 [162] / gw6 [162] / gw7 [162] / gw8 [162] / gw9 [162] / gw10 [162] / gw11 [162] / gw12 [162] / gw13 [162] / gw14 [162] / gw15 [162] / gw16 [162] / gw17 [162] / gw18 [162] / gw19 [162] / gw20 [162] / gw21 [162] / gw22 [162] / gw23 [162] / gw24 [162] / gw25 [162] / gw26 [162] / gw27 [162] / gw28 [162] / gw29 [162] / gw30 [162] / gw31 [162]
             
            scheduling tests via LoadScheduling
            .....................................................................................................................F............................................
             generated xml file: /astro/users/danielsf/lsstsw3/build/meas_algorithms/tests/.tests/pytest-meas_algorithms.xml 
            =================================== FAILURES ===================================
            __________________ SpatialModelPsfTestCase.testCandidateList ___________________
            [gw5] linux -- Python 3.5.2 /astro/users/danielsf/lsstsw3/miniconda/bin/python
             
            self = <test_psfIO.SpatialModelPsfTestCase testMethod=testCandidateList>
             
                def setUp(self):
                    width, height = 100, 300
                    self.mi = afwImage.MaskedImageF(afwGeom.ExtentI(width, height))
                    self.mi.set(0)
                    self.mi.getVariance().set(10)
                    self.mi.getMask().addMaskPlane("DETECTED")
                
                    self.FWHM = 5
                    self.ksize = 25                      # size of desired kernel
                
                    self.exposure = afwImage.makeExposure(self.mi)
                
                    psf = roundTripPsf(2, algorithms.DoubleGaussianPsf(self.ksize, self.ksize,
                                                                       self.FWHM/(2*math.sqrt(2*math.log(2))), 1, 0.1))
                    self.exposure.setPsf(psf)
                
                    for x, y in [(20, 20),
                                 # (30, 35), (50, 50),
                                 (60, 20), (60, 210), (20, 210)]:
                
                        flux = 10000 - 0*x - 10*y
                
                        sigma = 3 + 0.01*(y - self.mi.getHeight()/2)
            >           psf = roundTripPsf(3, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, sigma, 1, 0.1))
             
            tests/test_psfIO.py:111: 
            _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
             
            key = 3
            psf = <lsst.meas.algorithms.doubleGaussianPsf.DoubleGaussianPsf object at 0x7fca627e56f8>
             
                def roundTripPsf(key, psf):
                    global psfFileNum
                    pol = policy.Policy()
                    additionalData = dafBase.PropertySet()
                
                    if psfFileNum % 2 == 1:
                        storageType = "Boost"
                    else:
                        storageType = "Xml"
                    loc = dafPersist.LogicalLocation(
                        "tests/data/psf%d-%d.%s" % (psfFileNum, key, storageType))
                    psfFileNum += 1
                    persistence = dafPersist.Persistence.getPersistence(pol)
                
                    storageList = dafPersist.StorageList()
                    storage = persistence.getPersistStorage("%sStorage" % (storageType), loc)
                    storageList.append(storage)
                    persistence.persist(psf, storageList, additionalData)
                
                    storageList2 = dafPersist.StorageList()
            >       storage2 = persistence.getRetrieveStorage("%sStorage" % (storageType), loc)
            E       RuntimeError: unrecognized XML syntax
             
            tests/test_psfIO.py:78: RuntimeError
            ==================== 1 failed, 161 passed in 40.90 seconds =====================
            

            Show
            danielsf Scott Daniel added a comment - ============================= test session starts ============================== platform linux -- Python 3.5.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0 rootdir: /astro/users/danielsf/lsstsw3/build/meas_algorithms, inifile: plugins: session2file-0.1.9, xdist-1.19.2.dev0+g459d52e.d20170831, forked-0.3.dev0+g1dd93f6.d20170831, flake8-0.8.1 gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I gw0 [162] / gw1 [162] / gw2 [162] / gw3 [162] / gw4 [162] / gw5 [162] / gw6 [162] / gw7 [162] / gw8 [162] / gw9 [162] / gw10 [162] / gw11 [162] / gw12 [162] / gw13 [162] / gw14 [162] / gw15 [162] / gw16 [162] / gw17 [162] / gw18 [162] / gw19 [162] / gw20 [162] / gw21 [162] / gw22 [162] / gw23 [162] / gw24 [162] / gw25 [162] / gw26 [162] / gw27 [162] / gw28 [162] / gw29 [162] / gw30 [162] / gw31 [162]   scheduling tests via LoadScheduling .....................................................................................................................F............................................ generated xml file: /astro/users/danielsf/lsstsw3/build/meas_algorithms/tests/.tests/pytest-meas_algorithms.xml =================================== FAILURES =================================== __________________ SpatialModelPsfTestCase.testCandidateList ___________________ [gw5] linux -- Python 3.5.2 /astro/users/danielsf/lsstsw3/miniconda/bin/python   self = <test_psfIO.SpatialModelPsfTestCase testMethod=testCandidateList>   def setUp(self): width, height = 100, 300 self.mi = afwImage.MaskedImageF(afwGeom.ExtentI(width, height)) self.mi.set(0) self.mi.getVariance().set(10) self.mi.getMask().addMaskPlane("DETECTED") self.FWHM = 5 self.ksize = 25 # size of desired kernel self.exposure = afwImage.makeExposure(self.mi) psf = roundTripPsf(2, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, self.FWHM/(2*math.sqrt(2*math.log(2))), 1, 0.1)) self.exposure.setPsf(psf) for x, y in [(20, 20), # (30, 35), (50, 50), (60, 20), (60, 210), (20, 210)]: flux = 10000 - 0*x - 10*y sigma = 3 + 0.01*(y - self.mi.getHeight()/2) > psf = roundTripPsf(3, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, sigma, 1, 0.1))   tests/test_psfIO.py:111: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _   key = 3 psf = <lsst.meas.algorithms.doubleGaussianPsf.DoubleGaussianPsf object at 0x7fca627e56f8>   def roundTripPsf(key, psf): global psfFileNum pol = policy.Policy() additionalData = dafBase.PropertySet() if psfFileNum % 2 == 1: storageType = "Boost" else: storageType = "Xml" loc = dafPersist.LogicalLocation( "tests/data/psf%d-%d.%s" % (psfFileNum, key, storageType)) psfFileNum += 1 persistence = dafPersist.Persistence.getPersistence(pol) storageList = dafPersist.StorageList() storage = persistence.getPersistStorage("%sStorage" % (storageType), loc) storageList.append(storage) persistence.persist(psf, storageList, additionalData) storageList2 = dafPersist.StorageList() > storage2 = persistence.getRetrieveStorage("%sStorage" % (storageType), loc) E RuntimeError: unrecognized XML syntax   tests/test_psfIO.py:78: RuntimeError ==================== 1 failed, 161 passed in 40.90 seconds =====================
            Hide
            danielsf Scott Daniel added a comment -

            (It's going to be a while; my non-NFS machine has to rebuild afw)

            Show
            danielsf Scott Daniel added a comment - (It's going to be a while; my non-NFS machine has to rebuild afw)
            Hide
            price Paul Price added a comment -

            Scott Daniel: The error in tests/test_psfIO.py has nothing to do with this work. I suggest you file a new ticket and/or ask on Slack.

            Show
            price Paul Price added a comment - Scott Daniel : The error in tests/test_psfIO.py has nothing to do with this work. I suggest you file a new ticket and/or ask on Slack.
            Hide
            danielsf Scott Daniel added a comment -

            Okay. I opened DM-11810 (as feared, everything worked fine without NFS)

            Show
            danielsf Scott Daniel added a comment - Okay. I opened DM-11810 (as feared, everything worked fine without NFS)
            Hide
            tjenness Tim Jenness added a comment -

            Paul Price regarding your ci_hsc py3 failure, if you want you can do a build with tickets/DM-8560-scons and that will allow you to test everything on py3.

            Show
            tjenness Tim Jenness added a comment - Paul Price regarding your ci_hsc py3 failure, if you want you can do a build with tickets/ DM-8560 -scons and that will allow you to test everything on py3.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Chris Morrison
                Watchers:
                Chris Morrison, Hsin-Fang Chiang, John Swinbank, Paul Price, Scott Daniel, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: