Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11356

Confusion of refObjLoader.pixelMargin and matcher.maxOffsetPix

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • meas_astrom
    • None
    • External

    Description

      A recent change in AstrometryTask.solve increases the exposure bounding box by the matcher's maximum offset (matcher.maxOffsetPix. However, a parameter already exists to provide this growth in the reference catalog loader (refObjLoader.pixelMargin). Because the exposure bounding box is increased artificially after recording the catalog loading radius, loading the normalised persisted matches results in many missed matches.

      To fix this:

      • Remove the additional grow
      • Set the default refObjLoader.pixelMargin to match the default matcher.maxOffsetPix.
      • Add notes advising the user that the two parameters should be linked.
      • Stop AstrometryTask from reaching into the responsibility of the reference catalog loader (it shouldn't need to know about the details of the refObjLoader configuration).

      Attachments

        Issue Links

          Activity

            danielsf Scott Daniel added a comment -

            ============================= test session starts ==============================
            platform linux -- Python 3.5.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0
            rootdir: /astro/users/danielsf/lsstsw3/build/meas_algorithms, inifile:
            plugins: session2file-0.1.9, xdist-1.19.2.dev0+g459d52e.d20170831, forked-0.3.dev0+g1dd93f6.d20170831, flake8-0.8.1
            gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
            gw0 [162] / gw1 [162] / gw2 [162] / gw3 [162] / gw4 [162] / gw5 [162] / gw6 [162] / gw7 [162] / gw8 [162] / gw9 [162] / gw10 [162] / gw11 [162] / gw12 [162] / gw13 [162] / gw14 [162] / gw15 [162] / gw16 [162] / gw17 [162] / gw18 [162] / gw19 [162] / gw20 [162] / gw21 [162] / gw22 [162] / gw23 [162] / gw24 [162] / gw25 [162] / gw26 [162] / gw27 [162] / gw28 [162] / gw29 [162] / gw30 [162] / gw31 [162]
             
            scheduling tests via LoadScheduling
            .....................................................................................................................F............................................
             generated xml file: /astro/users/danielsf/lsstsw3/build/meas_algorithms/tests/.tests/pytest-meas_algorithms.xml 
            =================================== FAILURES ===================================
            __________________ SpatialModelPsfTestCase.testCandidateList ___________________
            [gw5] linux -- Python 3.5.2 /astro/users/danielsf/lsstsw3/miniconda/bin/python
             
            self = <test_psfIO.SpatialModelPsfTestCase testMethod=testCandidateList>
             
                def setUp(self):
                    width, height = 100, 300
                    self.mi = afwImage.MaskedImageF(afwGeom.ExtentI(width, height))
                    self.mi.set(0)
                    self.mi.getVariance().set(10)
                    self.mi.getMask().addMaskPlane("DETECTED")
                
                    self.FWHM = 5
                    self.ksize = 25                      # size of desired kernel
                
                    self.exposure = afwImage.makeExposure(self.mi)
                
                    psf = roundTripPsf(2, algorithms.DoubleGaussianPsf(self.ksize, self.ksize,
                                                                       self.FWHM/(2*math.sqrt(2*math.log(2))), 1, 0.1))
                    self.exposure.setPsf(psf)
                
                    for x, y in [(20, 20),
                                 # (30, 35), (50, 50),
                                 (60, 20), (60, 210), (20, 210)]:
                
                        flux = 10000 - 0*x - 10*y
                
                        sigma = 3 + 0.01*(y - self.mi.getHeight()/2)
            >           psf = roundTripPsf(3, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, sigma, 1, 0.1))
             
            tests/test_psfIO.py:111: 
            _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
             
            key = 3
            psf = <lsst.meas.algorithms.doubleGaussianPsf.DoubleGaussianPsf object at 0x7fca627e56f8>
             
                def roundTripPsf(key, psf):
                    global psfFileNum
                    pol = policy.Policy()
                    additionalData = dafBase.PropertySet()
                
                    if psfFileNum % 2 == 1:
                        storageType = "Boost"
                    else:
                        storageType = "Xml"
                    loc = dafPersist.LogicalLocation(
                        "tests/data/psf%d-%d.%s" % (psfFileNum, key, storageType))
                    psfFileNum += 1
                    persistence = dafPersist.Persistence.getPersistence(pol)
                
                    storageList = dafPersist.StorageList()
                    storage = persistence.getPersistStorage("%sStorage" % (storageType), loc)
                    storageList.append(storage)
                    persistence.persist(psf, storageList, additionalData)
                
                    storageList2 = dafPersist.StorageList()
            >       storage2 = persistence.getRetrieveStorage("%sStorage" % (storageType), loc)
            E       RuntimeError: unrecognized XML syntax
             
            tests/test_psfIO.py:78: RuntimeError
            ==================== 1 failed, 161 passed in 40.90 seconds =====================
            

            danielsf Scott Daniel added a comment - ============================= test session starts ============================== platform linux -- Python 3.5.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0 rootdir: /astro/users/danielsf/lsstsw3/build/meas_algorithms, inifile: plugins: session2file-0.1.9, xdist-1.19.2.dev0+g459d52e.d20170831, forked-0.3.dev0+g1dd93f6.d20170831, flake8-0.8.1 gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I gw0 [162] / gw1 [162] / gw2 [162] / gw3 [162] / gw4 [162] / gw5 [162] / gw6 [162] / gw7 [162] / gw8 [162] / gw9 [162] / gw10 [162] / gw11 [162] / gw12 [162] / gw13 [162] / gw14 [162] / gw15 [162] / gw16 [162] / gw17 [162] / gw18 [162] / gw19 [162] / gw20 [162] / gw21 [162] / gw22 [162] / gw23 [162] / gw24 [162] / gw25 [162] / gw26 [162] / gw27 [162] / gw28 [162] / gw29 [162] / gw30 [162] / gw31 [162]   scheduling tests via LoadScheduling .....................................................................................................................F............................................ generated xml file: /astro/users/danielsf/lsstsw3/build/meas_algorithms/tests/.tests/pytest-meas_algorithms.xml =================================== FAILURES =================================== __________________ SpatialModelPsfTestCase.testCandidateList ___________________ [gw5] linux -- Python 3.5.2 /astro/users/danielsf/lsstsw3/miniconda/bin/python   self = <test_psfIO.SpatialModelPsfTestCase testMethod=testCandidateList>   def setUp(self): width, height = 100, 300 self.mi = afwImage.MaskedImageF(afwGeom.ExtentI(width, height)) self.mi.set(0) self.mi.getVariance().set(10) self.mi.getMask().addMaskPlane("DETECTED") self.FWHM = 5 self.ksize = 25 # size of desired kernel self.exposure = afwImage.makeExposure(self.mi) psf = roundTripPsf(2, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, self.FWHM/(2*math.sqrt(2*math.log(2))), 1, 0.1)) self.exposure.setPsf(psf) for x, y in [(20, 20), # (30, 35), (50, 50), (60, 20), (60, 210), (20, 210)]: flux = 10000 - 0*x - 10*y sigma = 3 + 0.01*(y - self.mi.getHeight()/2) > psf = roundTripPsf(3, algorithms.DoubleGaussianPsf(self.ksize, self.ksize, sigma, 1, 0.1))   tests/test_psfIO.py:111: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _   key = 3 psf = <lsst.meas.algorithms.doubleGaussianPsf.DoubleGaussianPsf object at 0x7fca627e56f8>   def roundTripPsf(key, psf): global psfFileNum pol = policy.Policy() additionalData = dafBase.PropertySet() if psfFileNum % 2 == 1: storageType = "Boost" else: storageType = "Xml" loc = dafPersist.LogicalLocation( "tests/data/psf%d-%d.%s" % (psfFileNum, key, storageType)) psfFileNum += 1 persistence = dafPersist.Persistence.getPersistence(pol) storageList = dafPersist.StorageList() storage = persistence.getPersistStorage("%sStorage" % (storageType), loc) storageList.append(storage) persistence.persist(psf, storageList, additionalData) storageList2 = dafPersist.StorageList() > storage2 = persistence.getRetrieveStorage("%sStorage" % (storageType), loc) E RuntimeError: unrecognized XML syntax   tests/test_psfIO.py:78: RuntimeError ==================== 1 failed, 161 passed in 40.90 seconds =====================
            danielsf Scott Daniel added a comment -

            (It's going to be a while; my non-NFS machine has to rebuild afw)

            danielsf Scott Daniel added a comment - (It's going to be a while; my non-NFS machine has to rebuild afw)
            price Paul Price added a comment -

            danielsf: The error in tests/test_psfIO.py has nothing to do with this work. I suggest you file a new ticket and/or ask on Slack.

            price Paul Price added a comment - danielsf : The error in tests/test_psfIO.py has nothing to do with this work. I suggest you file a new ticket and/or ask on Slack.
            danielsf Scott Daniel added a comment -

            Okay. I opened DM-11810 (as feared, everything worked fine without NFS)

            danielsf Scott Daniel added a comment - Okay. I opened DM-11810 (as feared, everything worked fine without NFS)
            tjenness Tim Jenness added a comment -

            price regarding your ci_hsc py3 failure, if you want you can do a build with tickets/DM-8560-scons and that will allow you to test everything on py3.

            tjenness Tim Jenness added a comment - price regarding your ci_hsc py3 failure, if you want you can do a build with tickets/ DM-8560 -scons and that will allow you to test everything on py3.

            People

              price Paul Price
              price Paul Price
              Chris Morrison [X] (Inactive)
              Chris Morrison [X] (Inactive), Hsin-Fang Chiang, John Swinbank, Paul Price, Scott Daniel, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.