Details
-
Type:
Story
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: utils
-
Labels:None
-
Story Points:0
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
Pandas started giving me this warning:
(Pdb) type(res)
|
<class 'pandas.core.series.Series'>
|
(Pdb) res *= 1
|
numexpr.utils INFO: Note: NumExpr detected 24 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
numexpr.utils INFO: NumExpr defaulting to 8 threads.
|
This is going to get annoying. Seen in transformObjectCatalogTask. Let's do something intentional.
Attachments
Issue Links
- duplicates
-
- Done
Activity
Right place to do this looks like https://github.com/lsst/base/blob/master/src/threads.cc#L132, which already covers OMP_NUM_THREADS (and a few others), but not NUMEXPR_MAX_THREADS.
I have no idea why that's implemented in C++ rather than Python, but it's already being called by both CmdLineTask and pipetask when -j N is used with N > 1. And where there might be other contexts where we want to disable implicit threading, they should probably do it by calling lsst.base.disableImplicitThreading(), too.
import pandas is enough to trigger the warning now.
This was fixed in DM-33622.
By fixed I mean that you do need to use lsst.utils.threads.disable_implicit_threading. There is no way to stop the log message appearing if you have globally enabled INFO log level because this triggers as soon as pandas is imported – you would need to call disable_implicit_threading before all other imports or set the NUMEXPR envvar in your shell.
We no longer set the python root logger to INFO so you will no longer see the log message by default.
I’m also now seeing this in the `pipe_analysis` scripts as I move to using pandas dataframes, +1 from me!