Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12492

Fix ci_hsc breakage from DM-10443

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc, obs_subaru
    • Labels:
    • Team:
      External

      Description

      DM-10443 broke ci_hsc because falling back to a PosixRegistry enabled things to work, but that option was removed.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Passed Jenkins as part of DM-6965 (with changes on u/price/20171102). Re-running independently with this ticket branch, just to be sure.

            Show
            price Paul Price added a comment - Passed Jenkins as part of DM-6965 (with changes on u/price/20171102 ). Re-running independently with this ticket branch, just to be sure.
            Hide
            price Paul Price added a comment -

            Jenkins is all green.

            Show
            price Paul Price added a comment - Jenkins is all green.
            Hide
            price Paul Price added a comment -

            Kian-Tat Lim reviewed the changes to obs_subaru when it was on a different branch.
            Jim Bosch reviewed the changes to ci_hsc as part of DM-6965 when it was living on that branch.
            I therefore take this work to be reviewed, and with Jenkins green am going ahead with the merge.

            Show
            price Paul Price added a comment - Kian-Tat Lim reviewed the changes to obs_subaru when it was on a different branch . Jim Bosch reviewed the changes to ci_hsc as part of DM-6965 when it was living on that branch. I therefore take this work to be reviewed, and with Jenkins green am going ahead with the merge.
            Hide
            price Paul Price added a comment -

            DM-12493 is the ticket for moving features from HscMapper into the butler (or maybe the CameraMapper) requested by Kian-Tat Lim in his review.

            Merged to master.

            Show
            price Paul Price added a comment - DM-12493 is the ticket for moving features from HscMapper into the butler (or maybe the CameraMapper ) requested by Kian-Tat Lim in his review. Merged to master.
            Hide
            yusra Yusra AlSayyad added a comment -

            Note that I've been reverting this commit to reprocess the RC data starting with Hsin-Fang's calexps, or else I get the following complaint. Let me know if I should work around another way.

            [yusra@lsst-dev01 w_2017_44]$  source  /software/lsstsw/stack2/loadLSST.bash
            Note: use devtoolset-6 in conjunction with this stack.
            [yusra@lsst-dev01 w_2017_44]$ setup pipe_drivers -t w_2017_45
            [yusra@lsst-dev01 w_2017_44]$ setup -r  ~/lsst_devel/LSST/DMS/obs_subaru -t w_2017_45
            [yusra@lsst-dev01 w_2017_44]$ setup  -r  ~/lsst_devel/LSST/DMS/meas_mosaic -t w_2017_45
            [yusra@lsst-dev01 w_2017_44]$ setup -r  ~/lsst_devel/LSST/DMS/pipe_tasks -t  w_2017_45
            [yusra@lsst-dev01 w_2017_44]$ coaddDriver.py  /datasets/hsc/repo --rerun RC/w_2017_44/DM-12388:private/yusra/psfMatching/DM-12445  --batch-type=slurm --mpiexec='-bind-to socket' --job z_psfMatch --time 1600 --cores 23  --id tract=9813 filter=HSC-Z patch=0,0 --selectId ccd=0..8^10..103 visit=$cosmosVisitsZ   --no-versions  --clobber-config
            root INFO: Loading config overrride file u'/home/yusra/lsst_devel/LSST/DMS/obs_subaru/config/coaddDriver.py'
            root INFO: Loading config overrride file u'/home/yusra/lsst_devel/LSST/DMS/obs_subaru/config/hsc/coaddDriver.py'
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/RC/w_2017_44/DM-12388
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            Traceback (most recent call last):  File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/pipe_drivers/14.0-5-g782f885+6/bin/coaddDriver.py", line 3, in <module>
                CoaddDriverTask.parseAndSubmit()
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0+8/python/lsst/ctrl/pool/parallel.py", line 424, in parseAndSubmit
                **kwargs)
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0+8/python/lsst/ctrl/pool/parallel.py", line 333, in parse_args
                args.parent = self._parent.parse_args(config, args=leftover, **kwargs)
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/pipe_base/14.0-4-gd190390+6/python/lsst/pipe/base/argumentParser.py", line 614, in parse_args
                namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/butler.py", line 537, in __init__
                repoData.repo = Repository(repoData)
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/repository.py", line 143, in __init__
                self._initMapper(repoData)
              File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/repository.py", line 172, in _initMapper
                **mapperArgs)
              File "/home/yusra/lsst_devel/LSST/DMS/obs_subaru/python/lsst/obs/hsc/hscMapper.py", line 38, in __init__
                calibSearch += [os.path.join(cfg.root, 'CALIB') for cfg in kwargs["repositoryCfg"].parents]
            AttributeError: 'str' object has no attribute 'root'
            

            Show
            yusra Yusra AlSayyad added a comment - Note that I've been reverting this commit to reprocess the RC data starting with Hsin-Fang's calexps, or else I get the following complaint. Let me know if I should work around another way. [yusra@lsst-dev01 w_2017_44]$ source /software/lsstsw/stack2/loadLSST.bash Note: use devtoolset-6 in conjunction with this stack. [yusra@lsst-dev01 w_2017_44]$ setup pipe_drivers -t w_2017_45 [yusra@lsst-dev01 w_2017_44]$ setup -r ~/lsst_devel/LSST/DMS/obs_subaru -t w_2017_45 [yusra@lsst-dev01 w_2017_44]$ setup -r ~/lsst_devel/LSST/DMS/meas_mosaic -t w_2017_45 [yusra@lsst-dev01 w_2017_44]$ setup -r ~/lsst_devel/LSST/DMS/pipe_tasks -t w_2017_45 [yusra@lsst-dev01 w_2017_44]$ coaddDriver.py /datasets/hsc/repo --rerun RC/w_2017_44/DM-12388:private/yusra/psfMatching/DM-12445 --batch-type=slurm --mpiexec='-bind-to socket' --job z_psfMatch --time 1600 --cores 23 --id tract=9813 filter=HSC-Z patch=0,0 --selectId ccd=0..8^10..103 visit=$cosmosVisitsZ --no-versions --clobber-config root INFO: Loading config overrride file u'/home/yusra/lsst_devel/LSST/DMS/obs_subaru/config/coaddDriver.py' root INFO: Loading config overrride file u'/home/yusra/lsst_devel/LSST/DMS/obs_subaru/config/hsc/coaddDriver.py' CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/RC/w_2017_44/DM-12388 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 Traceback (most recent call last): File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/pipe_drivers/14.0-5-g782f885+6/bin/coaddDriver.py", line 3, in <module> CoaddDriverTask.parseAndSubmit() File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0+8/python/lsst/ctrl/pool/parallel.py", line 424, in parseAndSubmit **kwargs) File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0+8/python/lsst/ctrl/pool/parallel.py", line 333, in parse_args args.parent = self._parent.parse_args(config, args=leftover, **kwargs) File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/pipe_base/14.0-4-gd190390+6/python/lsst/pipe/base/argumentParser.py", line 614, in parse_args namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/butler.py", line 537, in __init__ repoData.repo = Repository(repoData) File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/repository.py", line 143, in __init__ self._initMapper(repoData) File "/software/lsstsw/stack2_20171022/stack/miniconda2-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-11-g0362164+2/python/lsst/daf/persistence/repository.py", line 172, in _initMapper **mapperArgs) File "/home/yusra/lsst_devel/LSST/DMS/obs_subaru/python/lsst/obs/hsc/hscMapper.py", line 38, in __init__ calibSearch += [os.path.join(cfg.root, 'CALIB') for cfg in kwargs["repositoryCfg"].parents] AttributeError: 'str' object has no attribute 'root'
            Hide
            price Paul Price added a comment -

            See DM-12595.

            Show
            price Paul Price added a comment - See DM-12595 .

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                Paul Price, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel