Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11375

ci_hsc intermittently fails with "skymap in packages" failed

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None
    • Team:
      External

      Description

      This appears to be due to a race condition in writing the packages. lsst.skymap is only import-ed before Task execution once: in makeSkyMap.py. If that runs at the same time as something else that's updating the packages, the skymap package version could be overwritten.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          This has passed Jenkins. Well, once anyway. We'd need to run it many times before a race condition reveals itself, but at least we know the changes haven't broken things.

          Show
          price Paul Price added a comment - This has passed Jenkins . Well, once anyway. We'd need to run it many times before a race condition reveals itself, but at least we know the changes haven't broken things.
          Hide
          price Paul Price added a comment -

          Nate Lust, since you're working in ci_hsc at the moment, would you mind reviewing this?

          price@pap-laptop:~/LSST/ci_hsc (tickets/DM-11375=) $ git sub
          commit c17b3a9f3fe6716221486f6d7be0d8bde17deb7d
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Jul 25 15:52:21 2017 -0400
           
              protect against race conditions in package versions
              
              We're getting rare, intermittent failures:
                 AssertionError: Failed test: skymap in packages
              This appears to be due to a race condition in writing the package verisons.
              lsst.skymap is only imported before Task execution (and therefore subject to
              having its version recorded) once: in makeSkyMap.py. If that runs at the same
              time as something else that's updating the package versions, the skymap
              package version could be overwritten. This patch forces skymap generation to
              be done early, and in serial to avoid race conditions.
           
           SConstruct | 16 +++++++++-------
           1 file changed, 9 insertions(+), 7 deletions(-)
          

          Show
          price Paul Price added a comment - Nate Lust , since you're working in ci_hsc at the moment, would you mind reviewing this? price@pap-laptop:~/LSST/ci_hsc (tickets/DM-11375=) $ git sub commit c17b3a9f3fe6716221486f6d7be0d8bde17deb7d Author: Paul Price <price@astro.princeton.edu> Date: Tue Jul 25 15:52:21 2017 -0400   protect against race conditions in package versions We're getting rare, intermittent failures: AssertionError: Failed test: skymap in packages This appears to be due to a race condition in writing the package verisons. lsst.skymap is only imported before Task execution (and therefore subject to having its version recorded) once: in makeSkyMap.py. If that runs at the same time as something else that's updating the package versions, the skymap package version could be overwritten. This patch forces skymap generation to be done early, and in serial to avoid race conditions.   SConstruct | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
          Hide
          nlust Nate Lust added a comment -

          logical flow looks fine to me

          Show
          nlust Nate Lust added a comment - logical flow looks fine to me
          Hide
          price Paul Price added a comment -

          Thanks Nate.

          Merged to master.

          Show
          price Paul Price added a comment - Thanks Nate. Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Nate Lust
              Watchers:
              Nate Lust, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel