Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15419

utils cache tests fail in python 3.7 and boost 1.68

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • utils
    • None
    • 0.5
    • Architecture

    Description

      Testing the stack with Python 3.7 fails when building utils.

      Python 3.7 requires boost 1.68 (DM-15385). To reproduce:

      • Install lsstsw using bin/deploy -b to get the "bleeding edge" conda packages.
      • Rebuild utils with boost 1.68: rebuild -r tickets/DM-15385 utils

      This will hang during the utils test_cache.py tests. The hang occurs even with optimization disabled in utils. I have not tried to disable boost optimization.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            All tests pass with this change. Does this mean it really is a bug in pybind11 or that I should apply this patch to utils?

            tjenness Tim Jenness added a comment - All tests pass with this change. Does this mean it really is a bug in pybind11 or that I should apply this patch to utils?
            jbosch Jim Bosch added a comment -

            The recent pybind11 upgrade was from 2.1, not 1.7; I don't think we ever used anything earlier than 2.0.

            The logic of price's fix seems sound to me, though I don't know what about pybind11 did actually change if it wasn't the commit found, or whether this would be considered a bug upstream.

            I assume the reason the callback is not being invoked by the thread in which it was defined is because pytest is creating multiple threads?  In any case, I think this is just good GIL citizenship, though I don't have a good sense for whether pybind11 could have made it unnecessary for us to handle it ourselves here.

            jbosch Jim Bosch added a comment - The recent pybind11 upgrade was from 2.1, not 1.7; I don't think we ever used anything earlier than 2.0. The logic of price 's fix seems sound to me, though I don't know what about pybind11 did actually change if it wasn't the commit found, or whether this would be considered a bug upstream. I assume the reason the callback is not being invoked by the thread in which it was defined is because pytest is creating multiple threads?  In any case, I think this is just good GIL citizenship, though I don't have a good sense for whether pybind11 could have made it unnecessary for us to handle it ourselves here.
            tjenness Tim Jenness added a comment -

            The test was hanging even if you run the test on its own with python (no pytest). I'll commit price's fix and do a Jenkins run.

            tjenness Tim Jenness added a comment - The test was hanging even if you run the test on its own with python (no pytest). I'll commit price 's fix and do a Jenkins run.
            tjenness Tim Jenness added a comment -

            The patch from price works on Jenkins and Python 3.7 so I think we should merge this. jbosch can you please formally review?

            tjenness Tim Jenness added a comment - The patch from price works on Jenkins and Python 3.7 so I think we should merge this. jbosch can you please formally review?
            jbosch Jim Bosch added a comment -

            Looks good.

            I've created DM-15444 to try to record places where we've found a need to do GIL management that we don't completely understand; please link any other tickets that describe such workarounds to that ticket as well, so if (for example) pybind11 behavior changes again we have a record of all the places in the codebase we should revisit.

            jbosch Jim Bosch added a comment - Looks good. I've created DM-15444 to try to record places where we've found a need to do GIL management that we don't completely understand; please link any other tickets that describe such workarounds to that ticket as well, so if (for example) pybind11 behavior changes again we have a record of all the places in the codebase we should revisit.

            People

              tjenness Tim Jenness
              tjenness Tim Jenness
              Jim Bosch
              Jim Bosch, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.