Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18314

Namespace confusion prevents use of MatchOptimisticB with singleFrameDriver

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      External

      Description

      Hiroyuki Ikeda reported problems running singleFrameDriver.py when using matchOptimisticB, reproduced here:

      pprice@tiger2-sumire:~/temp $ cat config.py 
      from lsst.meas.astrom import MatchOptimisticBTask
      config.processCcd.calibrate.astrometry.matcher.retarget(MatchOptimisticBTask)
      pprice@tiger2-sumire:~/temp $ singleFrameDriver.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 --job test --cores 20 --batch-type=smp --mpiexec='-bind-to socket' --time 600 -C config.py
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      root INFO: Running: -c /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 -C config.py --noExit
      singleFrameDriver FATAL: Failed in task initialization: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config
      root WARN: Unable to retrieve exit status ('NoneType' object is not iterable); assuming success
      Traceback (most recent call last):
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 673, in writeConfig
          oldConfig = butler.get(configName, immediate=True)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1396, in get
          return callback()
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1389, in callback
          return self._read(location)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1590, in _read
          results = location.repository.read(location)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/repository.py", line 197, in read
          return butlerLocationStorage.read(butlerLocation)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 280, in read
          return readFormatter(butlerLocation)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 534, in readConfigStorage
          finalItem.load(logLoc.locString())
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 952, in load
          self.loadFromStream(stream=code, root=root)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 990, in loadFromStream
          exec(stream, {}, local)
        File "/tigress/HSC/HSC/rerun/price/test/config/singleFrameDriver.py", line 4797, in <module>
          config.processCcd.calibrate.astrometry.matcher.retarget(target=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask, ConfigClass=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBConfig)
      AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'
       
      During handling of the above exception, another exception occurred:
       
      Traceback (most recent call last):
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 333, in precall
          self._precallImpl(task, parsedCmd)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 309, in _precallImpl
          task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig, doBackup=self.doBackup)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 676, in writeConfig
          (configName, exc))
      AttributeError: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config
      

      On investigation, this is due to namespace confusion: there is a module named matchOptimisticB (the directory named matchOptimisticB that contains a __init__.py), but that module gets overwritten by a pybinded function named matchOptimisticB when the main lsst.meas.astrom module's __init__.py does a from matchOptimisticB import *.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Thanks for volunteering to review this, Russell Owen. There's a single commit in meas_astrom:

            price@MacBook:~/LSST/meas_astrom (tickets/DM-18314=) $ git sub
            commit 98e6dfcdb902f4de28ee3d863b3b8baa072dd0a7 (HEAD -> tickets/DM-18314, origin/tickets/DM-18314)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Mar 6 14:24:07 2019 -0500
             
                reduce namespace confusion for matchOptimisticB
                
                The 'matchOptimisticB' namespace is overloaded, which makes it
                impossible to do:
                    import lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued
                    lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask
                results in:
                    AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'
                
                The 'matchOptimisticB' module contains a function called
                'matchOptimisticB' which is being imported over the top of the
                module, so rename that function. Move the MatchOptimisticBTask
                from the 'matchOptimisticB' module so it can be accessed
                without problems. Added a test that detects the problem.
             
             python/lsst/meas/astrom/__init__.py                    |  2 +-
             python/lsst/meas/astrom/matchOptimisticB/__init__.py   |  1 -
             .../meas/astrom/matchOptimisticB/matchOptimisticB.cc   |  2 +-
             ...OptimisticBContinued.py => matchOptimisticBTask.py} |  8 ++++----
             python/lsst/meas/astrom/matchPessimisticB.py           |  2 +-
             tests/test_matchOptimisticB.py                         | 18 ++++++++++++++----
             6 files changed, 21 insertions(+), 12 deletions(-)
            

            Show
            price Paul Price added a comment - Thanks for volunteering to review this, Russell Owen . There's a single commit in meas_astrom: price@MacBook:~/LSST/meas_astrom (tickets/DM-18314=) $ git sub commit 98e6dfcdb902f4de28ee3d863b3b8baa072dd0a7 (HEAD -> tickets/DM-18314, origin/tickets/DM-18314) Author: Paul Price <price@astro.princeton.edu> Date: Wed Mar 6 14:24:07 2019 -0500   reduce namespace confusion for matchOptimisticB The 'matchOptimisticB' namespace is overloaded, which makes it impossible to do: import lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask results in: AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued' The 'matchOptimisticB' module contains a function called 'matchOptimisticB' which is being imported over the top of the module, so rename that function. Move the MatchOptimisticBTask from the 'matchOptimisticB' module so it can be accessed without problems. Added a test that detects the problem.   python/lsst/meas/astrom/__init__.py | 2 +- python/lsst/meas/astrom/matchOptimisticB/__init__.py | 1 - .../meas/astrom/matchOptimisticB/matchOptimisticB.cc | 2 +- ...OptimisticBContinued.py => matchOptimisticBTask.py} | 8 ++++---- python/lsst/meas/astrom/matchPessimisticB.py | 2 +- tests/test_matchOptimisticB.py | 18 ++++++++++++++---- 6 files changed, 21 insertions(+), 12 deletions(-)
            Hide
            price Paul Price added a comment - - edited

            Jenkins is running...

            Show
            price Paul Price added a comment - - edited Jenkins is running...
            Hide
            rowen Russell Owen added a comment -

            Thank you for the fix. This looks good.

            I confess I don't understand why this only now came up. I would have thought it would be a problem from the beginning or not at all. But it doesn't really matter, since it clearly needs to be fixed.

            Show
            rowen Russell Owen added a comment - Thank you for the fix. This looks good. I confess I don't understand why this only now came up. I would have thought it would be a problem from the beginning or not at all. But it doesn't really matter, since it clearly needs to be fixed.
            Hide
            price Paul Price added a comment -

            Thanks Russell Owen.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks Russell Owen . Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Russell Owen
                Watchers:
                Chris Morrison, Hiroyuki Ikeda, Paul Price, Russell Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: