Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23703

ap_association fails if a new visit has 0 new diaObjects

    Details

      Description

      When association.py is called on a new visit, new_dia_objects is initialized as an empty list. That list only receives properly formatted entries if there are in fact any new DIA objects, but it is used to create the new_dia_objects pandas dataframe regardless. This results in an error on the next line when diaObjectId is set as the index: https://github.com/lsst/ap_association/blob/744ec977e52da689ac3639ffcbb519abeedabcd2/python/lsst/ap/association/association.py#L416

       

      Example error message:

      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 180, in runDataRef
          diaPipeResults = self.runAssociation(calexpRef)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 267, in runAssociation
          ccdExposureIdBits=sensorRef.get("ccdExposureId_bits"))
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/diaPipe.py", line 212, in run
          loaderResult.diaSources)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 128, in run
          matchResult = self.associate_sources(diaObjects, diaSources)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 222, in associate_sources
          match_result = self.match(dia_objects, dia_sources, scores)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 416, in match
          new_dia_objects.set_index("diaObjectId", inplace=True, drop=False)
        File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/pandas/core/frame.py", line 4303, in set_index
          raise KeyError(f"None of {missing} are in the columns")
      KeyError: "None of ['diaObjectId'] are in the columns"
      
      

        Attachments

          Activity

          sullivan Ian Sullivan created issue -
          sullivan Ian Sullivan made changes -
          Field Original Value New Value
          Epic Link DM-22484 [ 427311 ]
          swinbank John Swinbank made changes -
          Assignee John Swinbank [ swinbank ]
          swinbank John Swinbank made changes -
          Assignee John Swinbank [ swinbank ] Chris Morrison [ cmorrison ]
          swinbank John Swinbank made changes -
          Epic Link DM-22484 [ 427311 ] DM-22636 [ 427745 ]
          cmorrison Chris Morrison made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          cmorrison Chris Morrison added a comment -

          Added a work around to the problem and a test to make sure the bug is caught. Feel free to test for yourself.

          Additionally removed a deprecation warning.

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31303/pipeline

          Show
          cmorrison Chris Morrison added a comment - Added a work around to the problem and a test to make sure the bug is caught. Feel free to test for yourself. Additionally removed a deprecation warning. Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31303/pipeline
          cmorrison Chris Morrison made changes -
          Reviewers Ian Sullivan [ sullivan ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          sullivan Ian Sullivan added a comment -

          Looks great! This appears to fix the errors I was seeing.

          Show
          sullivan Ian Sullivan added a comment - Looks great! This appears to fix the errors I was seeing.
          sullivan Ian Sullivan made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          cmorrison Chris Morrison made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Ian Sullivan
              Watchers:
              Chris Morrison, Ian Sullivan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel