Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14064

lsst_dm_stack_demo failing with: ./bin/demo.sh: No such file or directory

    Details

      Description

      There are multiple reports on #dm-square this morning of the stack demo failing on same stack-os-matrix configurations with errors similar to:

      /home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.py3/ci-scripts/runManifestDemo.sh: line 210: ./bin/demo.sh: No such file or directory
      

      The clean build of of lsst_distrib failed all configurations but had been passing for several days prior:

      https://ci.lsst.codes/blue/organizations/jenkins/science-pipelines%2Flsst_distrib/detail/lsst_distrib/211/pipeline

      There have been no changes merged to either https://github.com/lsst/lsst_dm_stack_demo or https://github.com/lsst-sqre/ci-scripts/blob/master/runManifestDemo.sh in the last couple of days.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I'm merging a small PR to fiddle with the order of operations to try to ensure there's no change fo the PWD to become silently changed (already seems highly unlikely) and some additional status/debug output.

          Show
          jhoblitt Joshua Hoblitt added a comment - I'm merging a small PR to fiddle with the order of operations to try to ensure there's no change fo the PWD to become silently changed (already seems highly unlikely) and some additional status/debug output.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I've started a stack-os-matrix build of lsst_apps:master as this is likely that fastest way to [re]test for the error.

          https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27807/pipeline

          Show
          jhoblitt Joshua Hoblitt added a comment - I've started a stack-os-matrix build of lsst_apps:master as this is likely that fastest way to [re] test for the error. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27807/pipeline
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The nightly-release build appears to be unaffected, which is odd, as it runs the demo 7 times:

          https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141

          Show
          jhoblitt Joshua Hoblitt added a comment - The nightly-release build appears to be unaffected, which is odd, as it runs the demo 7 times: https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The tarball as downloaded by runManifestDemo.sh when run by hand also seems fine:

          $ tree lsst_dm_stack_demo-master/bin/
          lsst_dm_stack_demo-master/bin/
          ├── check_astrometry
          ├── compare
          ├── demo.sh
          └── export-results
          

          Show
          jhoblitt Joshua Hoblitt added a comment - The tarball as downloaded by runManifestDemo.sh when run by hand also seems fine: $ tree lsst_dm_stack_demo-master/bin/ lsst_dm_stack_demo-master/bin/ ├── check_astrometry ├── compare ├── demo.sh └── export-results
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141 succeeded. Either a side effect of my status/debugging PR fixed the problem or its transitory. The later would most likely mean that the github tarball download was somehow bad.

          Show
          jhoblitt Joshua Hoblitt added a comment - https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141 succeeded. Either a side effect of my status/debugging PR fixed the problem or its transitory. The later would most likely mean that the github tarball download was somehow bad.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I found a smoking gun and, rather regrettably, github is to blame. I've attached lsst_dm_stack_demo-b3542.tar.gz which was taken from the workspace for the science-pipelines/lsst_distrib clean build last night for centos-6.py3 running on el7-1.

          $ tree lsst_dm_stack_demo-b3542
          lsst_dm_stack_demo-b3542
          ├── demo.sh
          ├── demo_small.sh
          └── export-results
           
          0 directories, 3 files
          

          Show
          jhoblitt Joshua Hoblitt added a comment - I found a smoking gun and, rather regrettably, github is to blame. I've attached lsst_dm_stack_demo-b3542.tar.gz which was taken from the workspace for the science-pipelines/lsst_distrib clean build last night for centos-6.py3 running on el7-1 . $ tree lsst_dm_stack_demo-b3542 lsst_dm_stack_demo-b3542 ├── demo.sh ├── demo_small.sh └── export-results   0 directories, 3 files
          Hide
          jhoblitt Joshua Hoblitt added a comment - - edited

          I've opened a PR that adds some sanity checking that the demo scripts at least exist, a --debug flag that prints the xtrace (only) for commands being executed, and some assorted cleanups.

          If this problem reoccurs, the demo retrival mechanism should probably be changed to be a git clone or, better yet, the demo converted into an eups product.

          Show
          jhoblitt Joshua Hoblitt added a comment - - edited I've opened a PR that adds some sanity checking that the demo scripts at least exist, a --debug flag that prints the xtrace (only) for commands being executed, and some assorted cleanups. If this problem reoccurs, the demo retrival mechanism should probably be changed to be a git clone or, better yet, the demo converted into an eups product.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The status/debug PR has been merged.  I'm closing this PR and crossing my fingers.

          Show
          jhoblitt Joshua Hoblitt added a comment - The status/debug PR has been merged.  I'm closing this PR and crossing my fingers.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Tim Bond
              Watchers:
              Fred Moolekamp, Joshua Hoblitt, Paul Price, Tim Bond, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel