Details
-
Type:
Bug
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Middleware
-
Epic Link:
-
Team:Telescope and Site
Description
While trying to collect data from SAL with python I uncovered an issue with the numpy arrays. If you cache the array directly from the data structure (for example, to get an average of the data) you will get bad numbers. The issue appears to relate to garbage collection and the SAL data structure being collected while something else still holds a reference to the numpy arrays contained by that data structure.
Attached is an example python file that has 3 ways of caching the data.
I also tried a change in which I print the min/max of the first data cached while adding more data to the cache. What I found is the min / max of the first set of data cached changes. I have attached an image of it.
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
I did, even if you do a 0.1 second sleep it does not change the behavior.
Christopher Contaxis [X] (Inactive)
added a comment - I did, even if you do a 0.1 second sleep it does not change the behavior. And what happens if you just create the myData before entering the loop and keep reusing the same object ?
Dave Mills
added a comment - And what happens if you just create the myData before entering the loop and keep reusing the same object ? If you keep reusing the same data structure then the old data you are attempting to cache is overwritten by new data.
This is expected behavior given that you aren't instantiating new memory for the new data to be placed into.
Christopher Contaxis [X] (Inactive)
added a comment - - edited If you keep reusing the same data structure then the old data you are attempting to cache is overwritten by new data.
This is expected behavior given that you aren't instantiating new memory for the new data to be placed into. I think this is caused by the fact that the SALPY pybind11 is clever about how it wraps arrays and shares memory instead of copying the elements. If we can afford the small performance hit I think copying the data (as pybind11 does by default) would be much better because it would avoid issues like this.
Multiple other methods are available to avoid this
Dave Mills
added a comment - Multiple other methods are available to avoid this 
Did you try adding a small sleep so that you are not in a python cpu-locked loop.
You should never call SAL methods in a python cpu bound loop as the GIL may not
be able to schedule things correctly.