Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15419

utils cache tests fail in python 3.7 and boost 1.68

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: utils
    • Labels:
      None
    • Story Points:
      0.5
    • Team:
      Architecture

      Description

      Testing the stack with Python 3.7 fails when building utils.

      Python 3.7 requires boost 1.68 (DM-15385). To reproduce:

      • Install lsstsw using bin/deploy -b to get the "bleeding edge" conda packages.
      • Rebuild utils with boost 1.68: rebuild -r tickets/DM-15385 utils

      This will hang during the utils test_cache.py tests. The hang occurs even with optimization disabled in utils. I have not tried to disable boost optimization.

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15385 Update boost to v1.68

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15444 Investigate mysterious GIL management in pybind11 wrappers

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            6 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            All tests pass with this change. Does this mean it really is a bug in pybind11 or that I should apply this patch to utils?

            Show
            tjenness Tim Jenness added a comment - All tests pass with this change. Does this mean it really is a bug in pybind11 or that I should apply this patch to utils?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            The recent pybind11 upgrade was from 2.1, not 1.7; I don't think we ever used anything earlier than 2.0.

            The logic of Paul Price's fix seems sound to me, though I don't know what about pybind11 did actually change if it wasn't the commit found, or whether this would be considered a bug upstream.

            I assume the reason the callback is not being invoked by the thread in which it was defined is because pytest is creating multiple threads?  In any case, I think this is just good GIL citizenship, though I don't have a good sense for whether pybind11 could have made it unnecessary for us to handle it ourselves here.

            Show
            jbosch Jim Bosch added a comment - The recent pybind11 upgrade was from 2.1, not 1.7; I don't think we ever used anything earlier than 2.0. The logic of Paul Price 's fix seems sound to me, though I don't know what about pybind11 did actually change if it wasn't the commit found, or whether this would be considered a bug upstream. I assume the reason the callback is not being invoked by the thread in which it was defined is because pytest is creating multiple threads?  In any case, I think this is just good GIL citizenship, though I don't have a good sense for whether pybind11 could have made it unnecessary for us to handle it ourselves here.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            The test was hanging even if you run the test on its own with python (no pytest). I'll commit Paul Price's fix and do a Jenkins run.

            Show
            tjenness Tim Jenness added a comment - The test was hanging even if you run the test on its own with python (no pytest). I'll commit Paul Price 's fix and do a Jenkins run.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            The patch from Paul Price works on Jenkins and Python 3.7 so I think we should merge this. Jim Bosch can you please formally review?

            Show
            tjenness Tim Jenness added a comment - The patch from Paul Price works on Jenkins and Python 3.7 so I think we should merge this. Jim Bosch can you please formally review?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Looks good.

            I've created DM-15444 to try to record places where we've found a need to do GIL management that we don't completely understand; please link any other tickets that describe such workarounds to that ticket as well, so if (for example) pybind11 behavior changes again we have a record of all the places in the codebase we should revisit.

            Show
            jbosch Jim Bosch added a comment - Looks good. I've created DM-15444 to try to record places where we've found a need to do GIL management that we don't completely understand; please link any other tickets that describe such workarounds to that ticket as well, so if (for example) pybind11 behavior changes again we have a record of all the places in the codebase we should revisit.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.