Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8841

ci_hsc is broken

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None

      Description

      John Parejko in DM-8825 points out that ci_hsc is broken. It appears to have broken between build #932 (Jan 3, 2017 5:57:59 PM) and #933 (Jan 4, 2017 1:57:00 AM). Diagnose and fix.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            The log of the failure contains:

            [2017-01-04T09:57:32.643720Z] Traceback (most recent call last):
            [2017-01-04T09:57:32.643826Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_tasks/12.1-15-g5789d98/bin/processCcd.py", line 25, in <module>
            [2017-01-04T09:57:32.643856Z]     ProcessCcdTask.parseAndRun()
            [2017-01-04T09:57:32.643930Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/cmdLineTask.py", line 477, in parseAndRun
            [2017-01-04T09:57:32.643978Z]     parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
            [2017-01-04T09:57:32.644053Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/argumentParser.py", line 464, in parse_args
            [2017-01-04T09:57:32.644080Z]     self._applyInitialOverrides(namespace)
            [2017-01-04T09:57:32.644159Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/argumentParser.py", line 634, in _applyInitialOverrides
            [2017-01-04T09:57:32.644182Z]     namespace.config.load(filePath)
            [2017-01-04T09:57:32.644252Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/config.py", line 557, in load
            [2017-01-04T09:57:32.644283Z]     self.loadFromStream(stream=code, root=root)
            [2017-01-04T09:57:32.644358Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/config.py", line 577, in loadFromStream
            [2017-01-04T09:57:32.644386Z]     exec(stream, {}, local)
            [2017-01-04T09:57:32.644461Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/obs_subaru/12.1-11-geffe4d0+11/config/processCcd.py", line 31, in <module>
            [2017-01-04T09:57:32.644506Z]     config.calibrate.refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config",
            [2017-01-04T09:57:32.644613Z]   File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/configurableField.py", line 109, in __getattr__
            [2017-01-04T09:57:32.644644Z]     return getattr(self._value, name)
            [2017-01-04T09:57:32.644677Z] AttributeError: 'CalibrateConfig' object has no attribute 'refObjLoader'
            

            I'm going to suggest that this is fallout from DM-8232.

            Show
            price Paul Price added a comment - The log of the failure contains: [2017-01-04T09:57:32.643720Z] Traceback (most recent call last): [2017-01-04T09:57:32.643826Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_tasks/12.1-15-g5789d98/bin/processCcd.py", line 25, in <module> [2017-01-04T09:57:32.643856Z] ProcessCcdTask.parseAndRun() [2017-01-04T09:57:32.643930Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/cmdLineTask.py", line 477, in parseAndRun [2017-01-04T09:57:32.643978Z] parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) [2017-01-04T09:57:32.644053Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/argumentParser.py", line 464, in parse_args [2017-01-04T09:57:32.644080Z] self._applyInitialOverrides(namespace) [2017-01-04T09:57:32.644159Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pipe_base/12.1-4-gaefc628+14/python/lsst/pipe/base/argumentParser.py", line 634, in _applyInitialOverrides [2017-01-04T09:57:32.644182Z] namespace.config.load(filePath) [2017-01-04T09:57:32.644252Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/config.py", line 557, in load [2017-01-04T09:57:32.644283Z] self.loadFromStream(stream=code, root=root) [2017-01-04T09:57:32.644358Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/config.py", line 577, in loadFromStream [2017-01-04T09:57:32.644386Z] exec(stream, {}, local) [2017-01-04T09:57:32.644461Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/obs_subaru/12.1-11-geffe4d0+11/config/processCcd.py", line 31, in <module> [2017-01-04T09:57:32.644506Z] config.calibrate.refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config", [2017-01-04T09:57:32.644613Z] File "/home/jenkins-slave/workspace/stack-os-matrix/label/centos-6/python/py2/lsstsw/stack/Linux64/pex_config/12.1-4-gf46015e/python/lsst/pex/config/configurableField.py", line 109, in __getattr__ [2017-01-04T09:57:32.644644Z] return getattr(self._value, name) [2017-01-04T09:57:32.644677Z] AttributeError: 'CalibrateConfig' object has no attribute 'refObjLoader' I'm going to suggest that this is fallout from DM-8232 .
            Hide
            price Paul Price added a comment - - edited

            After tweaking the config in obs_subaru, I've had a successful ci_hsc run.

            John Parejko, would you please review these changes?

            price@price-laptop:~/LSST/obs/subaru (tickets/DM-8841=) $ git sub-patch
            commit 7a9d0623ce13d575b0df02f924f7bd669137f27c
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Jan 4 22:27:38 2017 -0500
             
                config: adapt to split of reference catalog loader
                
                The reference catalog loader has been split into separate
                astrometry and photometry versions.
             
            diff --git a/config/hsc/processCcd.py b/config/hsc/processCcd.py
            index c89ccf8..8cee42e 100644
            --- a/config/hsc/processCcd.py
            +++ b/config/hsc/processCcd.py
            @@ -11,16 +11,18 @@ config.load(os.path.join(hscConfigDir, 'isr.py'))
             config.calibrate.photoCal.colorterms.load(os.path.join(hscConfigDir, 'colorterms.py'))
             config.charImage.measurePsf.starSelector["objectSize"].widthMin = 0.9
             config.charImage.measurePsf.starSelector["objectSize"].fluxMin = 4000
            -config.calibrate.refObjLoader.load(os.path.join(hscConfigDir, "filterMap.py"))
            +for refObjLoader in (config.calibrate.astromRefObjLoader,
            +                     config.calibrate.photoRefObjLoader,
            +                     config.charImage.refObjLoader,
            +                     ):
            +    refObjLoader.load(os.path.join(hscConfigDir, "filterMap.py"))
             
             # Set to match defaults curretnly used in HSC production runs (e.g. S15B)
             config.calibrate.astrometry.matcher.sourceFluxType = 'Psf'
             config.calibrate.astrometry.matcher.allowedNonperpDeg = 0.2
             config.calibrate.astrometry.matcher.maxRotationDeg = 1.145916
             config.calibrate.astrometry.wcsFitter.numRejIter = 3
            -
            -for subTask in (config.charImage, config.calibrate):
            -    subTask.astrometry.wcsFitter.order = 3
            +config.calibrate.astrometry.wcsFitter.order = 3
             config.calibrate.astrometry.matcher.maxMatchDistArcSec = 2.0
             config.calibrate.astrometry.matcher.maxOffsetPix = 750
             
            diff --git a/config/processCcd.py b/config/processCcd.py
            index efb0d59..a1a7d2e 100644
            --- a/config/processCcd.py
            +++ b/config/processCcd.py
            @@ -28,8 +28,11 @@ except ImportError as e:
                 config.charImage.measurePsf.psfDeterminer.name = "pca"
             
             # Astrometry
            -config.calibrate.refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config",
            -                                                "filterMap.py"))
            +for refObjLoader in (config.calibrate.astromRefObjLoader,
            +                     config.calibrate.photoRefObjLoader,
            +                     config.charImage.refObjLoader,
            +                     ):
            +    refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config", "filterMap.py"))
             
             # Set to match defaults curretnly used in HSC production runs (e.g. S15B)
             config.charImage.catalogCalculation.plugins['base_ClassificationExtendedness'].fluxRatio = 0.95
            @@ -39,8 +42,10 @@ config.calibrate.photoCal.applyColorTerms = True
             
             from lsst.pipe.tasks.setConfigFromEups import setConfigFromEups
             menu = {"ps1*": {}, # Defaults are fine
            -        "sdss*": {"refObjLoader.filterMap": {"y": "z"}}, # No y-band, use z instead
            -        "2mass*": {"refObjLoader.filterMap": {ff: "J" for ff in "grizy"}}, # No optical bands, use J instead
            +        "sdss*": {"photoRefObjLoader.filterMap": {"y": "z"},
            +                  "astromRefObjLoader.filterMap": {"y": "z"}}, # No y-band, use z instead
            +        "2mass*": {"photoRefObjLoader.filterMap": {ff: "J" for ff in "grizy"},
            +                   "astromRefObjLoader.filterMap": {ff: "J" for ff in "grizy"}}, # No optical, use J instead
                     "10*": {}, # Match the empty astrometry_net_data version for use without a ref catalog
                     }
             setConfigFromEups(config.calibrate.photoCal, config.calibrate, menu)
            

            Show
            price Paul Price added a comment - - edited After tweaking the config in obs_subaru, I've had a successful ci_hsc run . John Parejko , would you please review these changes? price@price-laptop:~/LSST/obs/subaru (tickets/DM-8841=) $ git sub-patch commit 7a9d0623ce13d575b0df02f924f7bd669137f27c Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 22:27:38 2017 -0500   config: adapt to split of reference catalog loader The reference catalog loader has been split into separate astrometry and photometry versions.   diff --git a/config/hsc/processCcd.py b/config/hsc/processCcd.py index c89ccf8..8cee42e 100644 --- a/config/hsc/processCcd.py +++ b/config/hsc/processCcd.py @@ -11,16 +11,18 @@ config.load(os.path.join(hscConfigDir, 'isr.py')) config.calibrate.photoCal.colorterms.load(os.path.join(hscConfigDir, 'colorterms.py')) config.charImage.measurePsf.starSelector["objectSize"].widthMin = 0.9 config.charImage.measurePsf.starSelector["objectSize"].fluxMin = 4000 -config.calibrate.refObjLoader.load(os.path.join(hscConfigDir, "filterMap.py")) +for refObjLoader in (config.calibrate.astromRefObjLoader, + config.calibrate.photoRefObjLoader, + config.charImage.refObjLoader, + ): + refObjLoader.load(os.path.join(hscConfigDir, "filterMap.py")) # Set to match defaults curretnly used in HSC production runs (e.g. S15B) config.calibrate.astrometry.matcher.sourceFluxType = 'Psf' config.calibrate.astrometry.matcher.allowedNonperpDeg = 0.2 config.calibrate.astrometry.matcher.maxRotationDeg = 1.145916 config.calibrate.astrometry.wcsFitter.numRejIter = 3 - -for subTask in (config.charImage, config.calibrate): - subTask.astrometry.wcsFitter.order = 3 +config.calibrate.astrometry.wcsFitter.order = 3 config.calibrate.astrometry.matcher.maxMatchDistArcSec = 2.0 config.calibrate.astrometry.matcher.maxOffsetPix = 750 diff --git a/config/processCcd.py b/config/processCcd.py index efb0d59..a1a7d2e 100644 --- a/config/processCcd.py +++ b/config/processCcd.py @@ -28,8 +28,11 @@ except ImportError as e: config.charImage.measurePsf.psfDeterminer.name = "pca" # Astrometry -config.calibrate.refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config", - "filterMap.py")) +for refObjLoader in (config.calibrate.astromRefObjLoader, + config.calibrate.photoRefObjLoader, + config.charImage.refObjLoader, + ): + refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config", "filterMap.py")) # Set to match defaults curretnly used in HSC production runs (e.g. S15B) config.charImage.catalogCalculation.plugins['base_ClassificationExtendedness'].fluxRatio = 0.95 @@ -39,8 +42,10 @@ config.calibrate.photoCal.applyColorTerms = True from lsst.pipe.tasks.setConfigFromEups import setConfigFromEups menu = {"ps1*": {}, # Defaults are fine - "sdss*": {"refObjLoader.filterMap": {"y": "z"}}, # No y-band, use z instead - "2mass*": {"refObjLoader.filterMap": {ff: "J" for ff in "grizy"}}, # No optical bands, use J instead + "sdss*": {"photoRefObjLoader.filterMap": {"y": "z"}, + "astromRefObjLoader.filterMap": {"y": "z"}}, # No y-band, use z instead + "2mass*": {"photoRefObjLoader.filterMap": {ff: "J" for ff in "grizy"}, + "astromRefObjLoader.filterMap": {ff: "J" for ff in "grizy"}}, # No optical, use J instead "10*": {}, # Match the empty astrometry_net_data version for use without a ref catalog } setConfigFromEups(config.calibrate.photoCal, config.calibrate, menu)
            Hide
            Parejkoj John Parejko added a comment -

            Looks fine (I made a PR). I'm not really keen on the for-loop over 3 refObjLoaders, but I guess its fine.

            Show
            Parejkoj John Parejko added a comment - Looks fine (I made a PR). I'm not really keen on the for-loop over 3 refObjLoaders, but I guess its fine.
            Hide
            price Paul Price added a comment -

            Thanks John.

            The for loop isn't the prettiest, but it's better than setting the three independently because then something will get forgotten.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks John. The for loop isn't the prettiest, but it's better than setting the three independently because then something will get forgotten. Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                John Parejko, Paul Price, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel