XMLWordPrintable

#### Details

• Type: Story
• Status: Invalid
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
0
• Team:
Data Release Production
• Urgent?:
No

#### Description

Pandas started giving me this warning:

 (Pdb) type(res)  (Pdb) res *= 1 numexpr.utils INFO: Note: NumExpr detected 24 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. numexpr.utils INFO: NumExpr defaulting to 8 threads. 

This is going to get annoying. Seen in transformObjectCatalogTask. Let's do something intentional.

#### Activity

Hide
Lauren MacArthur added a comment -

I’m also now seeing this in the pipe_analysis scripts as I move to using pandas dataframes, +1 from me!

Show
Lauren MacArthur added a comment - I’m also now seeing this in the pipe_analysis scripts as I move to using pandas dataframes, +1 from me!
Hide
Jim Bosch added a comment -

I have no idea why that's implemented in C++ rather than Python, but it's already being called by both CmdLineTask and pipetask when -j N is used with N > 1. And where there might be other contexts where we want to disable implicit threading, they should probably do it by calling lsst.base.disableImplicitThreading(), too.

Show
Jim Bosch added a comment - Right place to do this looks like https://github.com/lsst/base/blob/master/src/threads.cc#L132 , which already covers OMP_NUM_THREADS (and a few others), but not NUMEXPR_MAX_THREADS. I have no idea why that's implemented in C++ rather than Python, but it's already being called by both CmdLineTask and pipetask when -j N is used with N > 1 . And where there might be other contexts where we want to disable implicit threading, they should probably do it by calling lsst.base.disableImplicitThreading() , too.
Hide
Tim Jenness added a comment -

import pandas is enough to trigger the warning now.

Show
Tim Jenness added a comment - import pandas is enough to trigger the warning now.
Hide
Tim Jenness added a comment -

This was fixed in DM-33622.

Show
Tim Jenness added a comment - This was fixed in DM-33622 .
Hide
Tim Jenness added a comment -

By fixed I mean that you do need to use lsst.utils.threads.disable_implicit_threading. There is no way to stop the log message appearing if you have globally enabled INFO log level because this triggers as soon as pandas is imported – you would need to call disable_implicit_threading before all other imports or set the NUMEXPR envvar in your shell.

We no longer set the python root logger to INFO so you will no longer see the log message by default.

Show
Tim Jenness added a comment - By fixed I mean that you do need to use lsst.utils.threads.disable_implicit_threading . There is no way to stop the log message appearing if you have globally enabled INFO log level because this triggers as soon as pandas is imported – you would need to call disable_implicit_threading before all other imports or set the NUMEXPR envvar in your shell. We no longer set the python root logger to INFO so you will no longer see the log message by default.

#### People

Assignee:
Unassigned
Reporter:
Watchers:
Bryce Kalmbach, Jim Bosch, Krzysztof Suberlak, Lauren MacArthur, Tim Jenness, Yusra AlSayyad