Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-835

Add threadpoolctl to rubin-env

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      We currently control the number of implicit threads used by numpy/pandas etc by having custom C++ code in the base package that sets environment variables and tries to call the low-level APIs to force the number of implicit threads to 1.

      What I would like to do is to drop our custom code and to instead use the threadpoolctl package.

      threadpoolctl is already in the rubin-env since it is brought in by scikit-learn so this will not change the size of the env at all.

      The thread pool control code in base is only used by ctrl_mpexec (and pipe_base for gen2). In DM-33622 I am moving the ability to disable implicit threading to utils and would like to do so without using ctypes or the base approach. I propose to be allowed to use threadpoolctl in this code.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            Seems OK. Since this is forward and backward compatible, I think this would only be 2.0.1.

            Show
            ktl Kian-Tat Lim added a comment - Seems OK. Since this is forward and backward compatible, I think this would only be 2.0.1.
            Hide
            erykoff Eli Rykoff added a comment -

            I support this! And it definitely will be forward and back compatible (for now). The reason to do it is so that we wouldn't break if scikit-learn removes their dependency.

            Show
            erykoff Eli Rykoff added a comment - I support this! And it definitely will be forward and back compatible (for now). The reason to do it is so that we wouldn't break if scikit-learn removes their dependency.
            Hide
            jbosch Jim Bosch added a comment -

            Show
            jbosch Jim Bosch added a comment -

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Colin Slater, Eli Rykoff, Jim Bosch, Kian-Tat Lim, Leanne Guy, Lee Kelvin, Paul Price, Steve Pietrowicz, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  CI Builds

                  No builds found.