Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18314

Namespace confusion prevents use of MatchOptimisticB with singleFrameDriver

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      External

      Description

      Hiroyuki Ikeda reported problems running singleFrameDriver.py when using matchOptimisticB, reproduced here:

      pprice@tiger2-sumire:~/temp $ cat config.py 
      from lsst.meas.astrom import MatchOptimisticBTask
      		 
      config.processCcd.calibrate.astrometry.matcher.retarget(MatchOptimisticBTask)
      		 
      pprice@tiger2-sumire:~/temp $ singleFrameDriver.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 --job test --cores 20 --batch-type=smp --mpiexec='-bind-to socket' --time 600 -C config.py
      		 
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      		 
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      		 
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      		 
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      		 
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      		 
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      		 
      root INFO: Running: -c /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 -C config.py --noExit
      		 
      singleFrameDriver FATAL: Failed in task initialization: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config
      		 
      root WARN: Unable to retrieve exit status ('NoneType' object is not iterable); assuming success
      		 
      Traceback (most recent call last):
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 673, in writeConfig
      		 
          oldConfig = butler.get(configName, immediate=True)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1396, in get
      		 
          return callback()
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1389, in callback
      		 
          return self._read(location)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1590, in _read
      		 
          results = location.repository.read(location)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/repository.py", line 197, in read
      		 
          return butlerLocationStorage.read(butlerLocation)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 280, in read
      		 
          return readFormatter(butlerLocation)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 534, in readConfigStorage
      		 
          finalItem.load(logLoc.locString())
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 952, in load
      		 
          self.loadFromStream(stream=code, root=root)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 990, in loadFromStream
      		 
          exec(stream, {}, local)
      		 
        File "/tigress/HSC/HSC/rerun/price/test/config/singleFrameDriver.py", line 4797, in <module>
      		 
          config.processCcd.calibrate.astrometry.matcher.retarget(target=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask, ConfigClass=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBConfig)
      		 
      AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'
      		 
       
      		 
      During handling of the above exception, another exception occurred:
      		 
       
      		 
      Traceback (most recent call last):
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 333, in precall
      		 
          self._precallImpl(task, parsedCmd)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 309, in _precallImpl
      		 
          task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig, doBackup=self.doBackup)
      		 
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 676, in writeConfig
      		 
          (configName, exc))
      		 
      AttributeError: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config

      On investigation, this is due to namespace confusion: there is a module named matchOptimisticB (the directory named matchOptimisticB that contains a __init__.py), but that module gets overwritten by a pybinded function named matchOptimisticB when the main lsst.meas.astrom module's __init__.py does a from matchOptimisticB import *.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17743 Shepherd tickets for HSC release candidate

          • Done
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. DM-18372 w_2019_10 pipelines.lsst.io build failure (matchOptimisticB task refactoring)

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            price Paul Price added a comment -

            Thanks for volunteering to review this, Russell Owen. There's a single commit in meas_astrom:

            price@MacBook:~/LSST/meas_astrom (tickets/DM-18314=) $ git sub
            		 
            commit 98e6dfcdb902f4de28ee3d863b3b8baa072dd0a7 (HEAD -> tickets/DM-18314, origin/tickets/DM-18314)
            		 
            Author: Paul Price <price@astro.princeton.edu>
            		 
            Date:   Wed Mar 6 14:24:07 2019 -0500
            		 
             
            		 
                reduce namespace confusion for matchOptimisticB
            		 
                
                The 'matchOptimisticB' namespace is overloaded, which makes it
            		 
                impossible to do:
            		 
                    import lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued
            		 
                    lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask
            		 
                results in:
            		 
                    AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'
            		 
                
                The 'matchOptimisticB' module contains a function called
            		 
                'matchOptimisticB' which is being imported over the top of the
            		 
                module, so rename that function. Move the MatchOptimisticBTask
            		 
                from the 'matchOptimisticB' module so it can be accessed
            		 
                without problems. Added a test that detects the problem.
            		 
             
            		 
             python/lsst/meas/astrom/__init__.py                    |  2 +-
            		 
             python/lsst/meas/astrom/matchOptimisticB/__init__.py   |  1 -
            		 
             .../meas/astrom/matchOptimisticB/matchOptimisticB.cc   |  2 +-
            		 
             ...OptimisticBContinued.py => matchOptimisticBTask.py} |  8 ++++----
            		 
             python/lsst/meas/astrom/matchPessimisticB.py           |  2 +-
            		 
             tests/test_matchOptimisticB.py                         | 18 ++++++++++++++----
            		 
             6 files changed, 21 insertions(+), 12 deletions(-)

            Show
            price Paul Price added a comment - Thanks for volunteering to review this, Russell Owen . There's a single commit in meas_astrom: price@MacBook:~/LSST/meas_astrom (tickets/DM-18314=) $ git sub commit 98e6dfcdb902f4de28ee3d863b3b8baa072dd0a7 (HEAD -> tickets/DM-18314, origin/tickets/DM-18314) Author: Paul Price <price@astro.princeton.edu> Date: Wed Mar 6 14:24:07 2019 -0500   reduce namespace confusion for matchOptimisticB The 'matchOptimisticB' namespace is overloaded, which makes it impossible to do: import lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask results in: AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued' The 'matchOptimisticB' module contains a function called 'matchOptimisticB' which is being imported over the top of the module, so rename that function. Move the MatchOptimisticBTask from the 'matchOptimisticB' module so it can be accessed without problems. Added a test that detects the problem.   python/lsst/meas/astrom/__init__.py | 2 +- python/lsst/meas/astrom/matchOptimisticB/__init__.py | 1 - .../meas/astrom/matchOptimisticB/matchOptimisticB.cc | 2 +- ...OptimisticBContinued.py => matchOptimisticBTask.py} | 8 ++++---- python/lsst/meas/astrom/matchPessimisticB.py | 2 +- tests/test_matchOptimisticB.py | 18 ++++++++++++++---- 6 files changed, 21 insertions(+), 12 deletions(-)
            Hide
            Permalink
            price Paul Price added a comment - - edited

            Jenkins is running...

            Show
            price Paul Price added a comment - - edited Jenkins is running...
            Hide
            Permalink
            rowen Russell Owen added a comment -

            Thank you for the fix. This looks good.

            I confess I don't understand why this only now came up. I would have thought it would be a problem from the beginning or not at all. But it doesn't really matter, since it clearly needs to be fixed.

            Show
            rowen Russell Owen added a comment - Thank you for the fix. This looks good. I confess I don't understand why this only now came up. I would have thought it would be a problem from the beginning or not at all. But it doesn't really matter, since it clearly needs to be fixed.
            Hide
            Permalink
            price Paul Price added a comment -

            Thanks Russell Owen.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks Russell Owen . Merged to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Russell Owen
              Watchers:
              Chris Morrison, Hiroyuki Ikeda, Paul Price, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.