Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23703

ap_association fails if a new visit has 0 new diaObjects

    Details

      Description

      When association.py is called on a new visit, new_dia_objects is initialized as an empty list. That list only receives properly formatted entries if there are in fact any new DIA objects, but it is used to create the new_dia_objects pandas dataframe regardless. This results in an error on the next line when diaObjectId is set as the index: https://github.com/lsst/ap_association/blob/744ec977e52da689ac3639ffcbb519abeedabcd2/python/lsst/ap/association/association.py#L416

       

      Example error message:

      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 180, in runDataRef
          diaPipeResults = self.runAssociation(calexpRef)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 267, in runAssociation
          ccdExposureIdBits=sensorRef.get("ccdExposureId_bits"))
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/diaPipe.py", line 212, in run
          loaderResult.diaSources)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 128, in run
          matchResult = self.associate_sources(diaObjects, diaSources)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 222, in associate_sources
          match_result = self.match(dia_objects, dia_sources, scores)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 416, in match
          new_dia_objects.set_index("diaObjectId", inplace=True, drop=False)
        File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/pandas/core/frame.py", line 4303, in set_index
          raise KeyError(f"None of {missing} are in the columns")
      KeyError: "None of ['diaObjectId'] are in the columns"
      
      

        Attachments

          Activity

          Hide
          cmorrison Chris Morrison added a comment -

          Added a work around to the problem and a test to make sure the bug is caught. Feel free to test for yourself.

          Additionally removed a deprecation warning.

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31303/pipeline

          Show
          cmorrison Chris Morrison added a comment - Added a work around to the problem and a test to make sure the bug is caught. Feel free to test for yourself. Additionally removed a deprecation warning. Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31303/pipeline
          Hide
          sullivan Ian Sullivan added a comment -

          Looks great! This appears to fix the errors I was seeing.

          Show
          sullivan Ian Sullivan added a comment - Looks great! This appears to fix the errors I was seeing.

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Ian Sullivan
              Watchers:
              Chris Morrison, Ian Sullivan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel