Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_association
-
Labels:
-
Story Points:1
-
Epic Link:
-
Team:Alert Production
-
Urgent?:No
Description
When association.py is called on a new visit, new_dia_objects is initialized as an empty list. That list only receives properly formatted entries if there are in fact any new DIA objects, but it is used to create the new_dia_objects pandas dataframe regardless. This results in an error on the next line when diaObjectId is set as the index: https://github.com/lsst/ap_association/blob/744ec977e52da689ac3639ffcbb519abeedabcd2/python/lsst/ap/association/association.py#L416
Example error message:
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__ |
result = self.runTask(task, dataRef, kwargs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask |
return task.runDataRef(dataRef, **kwargs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 180, in runDataRef |
diaPipeResults = self.runAssociation(calexpRef) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_pipe/19.0.0-6-g8dfed85+6/python/lsst/ap/pipe/ap_pipe.py", line 267, in runAssociation |
ccdExposureIdBits=sensorRef.get("ccdExposureId_bits")) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/diaPipe.py", line 212, in run |
loaderResult.diaSources)
|
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 128, in run |
matchResult = self.associate_sources(diaObjects, diaSources) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 222, in associate_sources |
match_result = self.match(dia_objects, dia_sources, scores) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper |
res = func(self, *args, **keyArgs) |
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/ap_association/19.0.0-13-g744ec97+5/python/lsst/ap/association/association.py", line 416, in match |
new_dia_objects.set_index("diaObjectId", inplace=True, drop=False) |
File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/pandas/core/frame.py", line 4303, in set_index |
raise KeyError(f"None of {missing} are in the columns") |
KeyError: "None of ['diaObjectId'] are in the columns" |
|
Attachments
Activity
Looks great! This appears to fix the errors I was seeing.
Added a work around to the problem and a test to make sure the bug is caught. Feel free to test for yourself.
Additionally removed a deprecation warning.
Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31303/pipeline