Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18314

Namespace confusion prevents use of MatchOptimisticB with singleFrameDriver

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Templates:
    • Team:
      External

      Description

      Hiroyuki Ikeda reported problems running singleFrameDriver.py when using matchOptimisticB, reproduced here:

      pprice@tiger2-sumire:~/temp $ cat config.py 
      from lsst.meas.astrom import MatchOptimisticBTask
      config.processCcd.calibrate.astrometry.matcher.retarget(MatchOptimisticBTask)
      pprice@tiger2-sumire:~/temp $ singleFrameDriver.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 --job test --cores 20 --batch-type=smp --mpiexec='-bind-to socket' --time 600 -C config.py
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/singleFrameDriver.py'
      root INFO: Loading config overrride file '/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/16.0-51-g0aa6a961/config/hsc/singleFrameDriver.py'
      CameraMapper INFO: Loading exposure registry from /tigress/HSC/HSC/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /tigress/HSC/HSC/CALIB_s18a/calibRegistry.sqlite3
      root INFO: Running: -c /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB_s18a --rerun price/test --id visit=1248 -C config.py --noExit
      singleFrameDriver FATAL: Failed in task initialization: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config
      root WARN: Unable to retrieve exit status ('NoneType' object is not iterable); assuming success
      Traceback (most recent call last):
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 673, in writeConfig
          oldConfig = butler.get(configName, immediate=True)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1396, in get
          return callback()
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1389, in callback
          return self._read(location)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/butler.py", line 1590, in _read
          results = location.repository.read(location)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/repository.py", line 197, in read
          return butlerLocationStorage.read(butlerLocation)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 280, in read
          return readFormatter(butlerLocation)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/daf_persistence/16.0-17-g7c01f5c+3/python/lsst/daf/persistence/posixStorage.py", line 534, in readConfigStorage
          finalItem.load(logLoc.locString())
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 952, in load
          self.loadFromStream(stream=code, root=root)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-14-g703d48b/python/lsst/pex/config/config.py", line 990, in loadFromStream
          exec(stream, {}, local)
        File "/tigress/HSC/HSC/rerun/price/test/config/singleFrameDriver.py", line 4797, in <module>
          config.processCcd.calibrate.astrometry.matcher.retarget(target=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBTask, ConfigClass=lsst.meas.astrom.matchOptimisticB.matchOptimisticBContinued.MatchOptimisticBConfig)
      AttributeError: 'builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'
       
      During handling of the above exception, another exception occurred:
       
      Traceback (most recent call last):
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 333, in precall
          self._precallImpl(task, parsedCmd)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 309, in _precallImpl
          task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig, doBackup=self.doBackup)
        File "/tigress/HSC/LSST/stack_tiger2-sumire.princeton.edu_20181028/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/16.0-37-gc7333cb+8/python/lsst/pipe/base/cmdLineTask.py", line 676, in writeConfig
          (configName, exc))
      AttributeError: Unable to read stored config file singleFrameDriver_config ('builtin_function_or_method' object has no attribute 'matchOptimisticBContinued'); consider using --clobber-config
      

      On investigation, this is due to namespace confusion: there is a module named matchOptimisticB (the directory named matchOptimisticB that contains a __init__.py), but that module gets overwritten by a pybinded function named matchOptimisticB when the main lsst.meas.astrom module's __init__.py does a from matchOptimisticB import *.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  price Paul Price
                  Reporter:
                  price Paul Price
                  Reviewers:
                  Russell Owen
                  Watchers:
                  Chris Morrison, Hiroyuki Ikeda, Paul Price, Russell Owen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel