Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-835

Add threadpoolctl to rubin-env

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      We currently control the number of implicit threads used by numpy/pandas etc by having custom C++ code in the base package that sets environment variables and tries to call the low-level APIs to force the number of implicit threads to 1.

      What I would like to do is to drop our custom code and to instead use the threadpoolctl package.

      threadpoolctl is already in the rubin-env since it is brought in by scikit-learn so this will not change the size of the env at all.

      The thread pool control code in base is only used by ctrl_mpexec (and pipe_base for gen2). In DM-33622 I am moving the ability to disable implicit threading to utils and would like to do so without using ctypes or the base approach. I propose to be allowed to use threadpoolctl in this code.

        Attachments

          Issue Links

            Activity

            No builds found.
            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue is triggering DM-33622 [ DM-33622 ]
            Hide
            ktl Kian-Tat Lim added a comment -

            Seems OK. Since this is forward and backward compatible, I think this would only be 2.0.1.

            Show
            ktl Kian-Tat Lim added a comment - Seems OK. Since this is forward and backward compatible, I think this would only be 2.0.1.
            Hide
            erykoff Eli Rykoff added a comment -

            I support this! And it definitely will be forward and back compatible (for now). The reason to do it is so that we wouldn't break if scikit-learn removes their dependency.

            Show
            erykoff Eli Rykoff added a comment - I support this! And it definitely will be forward and back compatible (for now). The reason to do it is so that we wouldn't break if scikit-learn removes their dependency.
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-33951 [ DM-33951 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 32301 ]
            tjenness Tim Jenness made changes -
            Status Proposed [ 10805 ] Flagged [ 10606 ]
            Hide
            jbosch Jim Bosch added a comment -

            Show
            jbosch Jim Bosch added a comment -
            womullan Wil O'Mullane made changes -
            Status Flagged [ 10606 ] Board Recommended [ 11405 ]
            tjenness Tim Jenness made changes -
            Status Board Recommended [ 11405 ] Adopted [ 10806 ]
            ktl Kian-Tat Lim made changes -
            Link This issue is triggering DM-33994 [ DM-33994 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 32390 ]

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Colin Slater, Eli Rykoff, Jim Bosch, Kian-Tat Lim, Leanne Guy, Lee Kelvin, Paul Price, Steve Pietrowicz, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.