Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9909

hsc cat format change breaks processing of validation_data_hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Migrated from DM-9829 per https://jira.lsstcorp.org/browse/DM-9829?focusedCommentId=75005&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-75005

      Yesterday's hsc dataset run failed:

      https://ci.lsst.codes/job/validate_drp/dataset=hsc,label=centos-7,python=py2/869/

      CameraMapper INFO: Loading Posix registry from /home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc/rerun/20170105
      Traceback (most recent call last):
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_drivers/13.0-3-gfcfef02+2/bin/singleFrameDriver.py", line 3, in <module>
          SingleFrameDriverTask.parseAndSubmit()
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/ctrl_pool/13.0-1-g0a48345+1/python/lsst/ctrl/pool/parallel.py", line 422, in parseAndSubmit
          if not cls.RunnerClass(cls, batchArgs.parent).precall(batchArgs.parent):  # Write config, schema
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+4/python/lsst/pipe/base/cmdLineTask.py", line 300, in precall
          task = self.makeTask(parsedCmd=parsedCmd)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+4/python/lsst/pipe/base/cmdLineTask.py", line 391, in makeTask
          return self.TaskClass(config=self.config, log=self.log, butler=butler)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_drivers/13.0-3-gfcfef02+2/python/lsst/pipe/drivers/singleFrameDriver.py", line 50, in __init__
          astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+4/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_tasks/13.0-7-g4569002/python/lsst/pipe/tasks/processCcd.py", line 159, in __init__
          astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+4/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_tasks/13.0-7-g4569002/python/lsst/pipe/tasks/calibrate.py", line 315, in __init__
          self.makeSubtask('astromRefObjLoader', butler=butler)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+4/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/meas_algorithms/13.0-6-g6f8a914+1/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py", line 50, in __init__
          dataset_config = butler.get("ref_cat_config", name=self.config.ref_dataset_name, immediate=True)
        File "/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/lsstsw/stack/Linux64/daf_persistence/13.0-5-g87674b4/python/lsst/daf/persistence/butler.py", line 945, in get
          raise NoResults("No locations for get:", datasetType, dataId)
      lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'ps1_pv3_3pi_20170110'}, tag=set([]))
      cp: cannot stat ‘/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc_rerun_20170105_HSC-I.json’: No such file or directory
      cp: cannot stat ‘/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc_rerun_20170105_HSC-R.json’: No such file or directory
      cp: cannot stat ‘/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/data_hsc_rerun_20170105_HSC-Y.json’: No such file or directory
      cp: cannot stat ‘/home/jenkins-slave/workspace/validate_drp/dataset/hsc/label/centos-7/python/py2/validate_drp/job_singleFrame.log’: No such file or directory
      

        Attachments

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            This looks good, Tim Morton.
            Go ahead and merge.

            My sincere apologies for the long delay in reviewing this. Thank you again for digging in to this.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - This looks good, Tim Morton . Go ahead and merge. My sincere apologies for the long delay in reviewing this. Thank you again for digging in to this.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Tim Morton A gentle ping to merge this.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Tim Morton A gentle ping to merge this.
            Hide
            swinbank John Swinbank added a comment -

            Adding this to the DRP sprint so that we can more easily track its progress.

            Show
            swinbank John Swinbank added a comment - Adding this to the DRP sprint so that we can more easily track its progress.
            Hide
            tmorton Tim Morton added a comment - - edited

            Sorry for letting this slip... Just wanted to confirm that I should merge this as is, even though things have pretty solidly moved away from using astrometry_net

            Show
            tmorton Tim Morton added a comment - - edited Sorry for letting this slip... Just wanted to confirm that I should merge this as is, even though things have pretty solidly moved away from using astrometry_net
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Yes. Merge.

            Someday someone could spend the effort to generate the new reference catalogs, but that's a low priority.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Yes. Merge. Someday someone could spend the effort to generate the new reference catalogs, but that's a low priority.

              People

              • Assignee:
                tmorton Tim Morton
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Michael Wood-Vasey
                Watchers:
                Angelo Fausti, John Swinbank, Joshua Hoblitt, Michael Wood-Vasey, Paul Price, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel