Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11500

validate_drp hsc dataset butler failure

    Details

      Description

      After resolution of DM-11410, the hsc dataset has a new failure mode:

      https://ci.lsst.codes/job/sqre/job/validate_drp/1018/dataset=hsc,label=centos-7,python=py2/console

      root INFO: Loading config overrride file u'/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/obs_subaru/13.0-41-g9300a79+5/config/hsc/makeDiscreteSkyMap.py'
      Traceback (most recent call last):
        File "/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_tasks/13.0-50-gf3eeffc+5/bin/makeDiscreteSkyMap.py", line 25, in <module>
          MakeDiscreteSkyMapTask.parseAndRun()
        File "/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-11-gdf6a56c+9/python/lsst/pipe/base/cmdLineTask.py", line 526, in parseAndRun
          parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
        File "/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-11-gdf6a56c+9/python/lsst/pipe/base/argumentParser.py", line 514, in parse_args
          namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
        File "/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/daf_persistence/13.0-29-g32a3f02/python/lsst/daf/persistence/butler.py", line 531, in __init__
          self._setAndVerifyParentsLists(repoDataList)
        File "/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/daf_persistence/13.0-29-g32a3f02/python/lsst/daf/persistence/butler.py", line 914, in _setAndVerifyParentsLists
          parents, repoData.cfg.parents, e))
      RuntimeError: Inputs of this Butler:[RepositoryCfg(root='../..', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={'calibRoot': '/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validation_data_hsc/CALIB'}, parents=[], policy=None), RepositoryCfg(root='../..', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={}, parents=[], policy=None)] do not match parents of existing writable cfg:[RepositoryCfg(root='/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={'calibRoot': '/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validation_data_hsc/CALIB'}, parents=[], policy=None)] (ParentMismatch exception: The beginning of the passed-in parents list: [RepositoryCfg(root='../..', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={'calibRoot': '/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validation_data_hsc/CALIB'}, parents=[], policy=None), RepositoryCfg(root='../..', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={}, parents=[], policy=None)] does not match the existing parents list in this RepositoryCfg: [RepositoryCfg(root='/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs={'calibRoot': '/home/jenkins-slave/workspace/sqre/validate_drp/dataset/hsc/label/centos-7/python/py2/validation_data_hsc/CALIB'}, parents=[], policy=None)]
      

        Attachments

          Issue Links

            Activity

            Hide
            krughoff Simon Krughoff added a comment -

            O.K. I'm going to merge.

            Show
            krughoff Simon Krughoff added a comment - O.K. I'm going to merge.
            Hide
            krughoff Simon Krughoff added a comment -

            Merged to master.

            Show
            krughoff Simon Krughoff added a comment - Merged to master.
            Hide
            ktl Kian-Tat Lim added a comment -

            It should be possible to change the Butler initialization logic to accept steps that have a subset of input repos in the current output repo config (without changing the latter) in addition to steps that add new input repos to the output repo config (while changing it). That would be a better long-term fix.

            Show
            ktl Kian-Tat Lim added a comment - It should be possible to change the Butler initialization logic to accept steps that have a subset of input repos in the current output repo config (without changing the latter) in addition to steps that add new input repos to the output repo config (while changing it). That would be a better long-term fix.
            Hide
            frossie Frossie Economou added a comment -

            Fritz Mueller or Nate Pease since Simon Krughoff did this do you mind if we pull this ticket into a SQuaRE epic to get credit? Otherwise we'll just file a duplicate

            Show
            frossie Frossie Economou added a comment - Fritz Mueller or Nate Pease since Simon Krughoff did this do you mind if we pull this ticket into a SQuaRE epic to get credit? Otherwise we'll just file a duplicate
            Hide
            fritzm Fritz Mueller added a comment -

            Frossie Economou: sure – bouncing this over to you. I have created a downstream story to capture the "should have been done this way instead" future work.

            Show
            fritzm Fritz Mueller added a comment - Frossie Economou : sure – bouncing this over to you. I have created a downstream story to capture the "should have been done this way instead" future work.

              People

              • Assignee:
                krughoff Simon Krughoff
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Fritz Mueller, Frossie Economou, Joshua Hoblitt, Kian-Tat Lim, Michael Wood-Vasey, Nate Pease, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel