# lsst_dm_stack_demo failing with: ./bin/demo.sh: No such file or directory

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
SQuaRE

## Description

There are multiple reports on #dm-square this morning of the stack demo failing on same stack-os-matrix configurations with errors similar to:

 /home/jenkins-slave/workspace/science-pipelines/lsst_distrib/centos-6.py3/ci-scripts/runManifestDemo.sh: line 210: ./bin/demo.sh: No such file or directory 

The clean build of of lsst_distrib failed all configurations but had been passing for several days prior:

https://ci.lsst.codes/blue/organizations/jenkins/science-pipelines%2Flsst_distrib/detail/lsst_distrib/211/pipeline

There have been no changes merged to either https://github.com/lsst/lsst_dm_stack_demo or https://github.com/lsst-sqre/ci-scripts/blob/master/runManifestDemo.sh in the last couple of days.

## Attachments

1. lsst_dm_stack_demo-b3542.tar.gz
2 kB

## Activity

Hide
Joshua Hoblitt added a comment -

I'm merging a small PR to fiddle with the order of operations to try to ensure there's no change fo the PWD to become silently changed (already seems highly unlikely) and some additional status/debug output.

Show
Joshua Hoblitt added a comment - I'm merging a small PR to fiddle with the order of operations to try to ensure there's no change fo the PWD to become silently changed (already seems highly unlikely) and some additional status/debug output.
Hide
Joshua Hoblitt added a comment -

I've started a stack-os-matrix build of lsst_apps:master as this is likely that fastest way to [re]test for the error.

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27807/pipeline

Show
Joshua Hoblitt added a comment - I've started a stack-os-matrix build of lsst_apps:master as this is likely that fastest way to [re] test for the error. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27807/pipeline
Hide
Joshua Hoblitt added a comment -

The nightly-release build appears to be unaffected, which is odd, as it runs the demo 7 times:

https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141

Show
Joshua Hoblitt added a comment - The nightly-release build appears to be unaffected, which is odd, as it runs the demo 7 times: https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141
Hide
Joshua Hoblitt added a comment -

The tarball as downloaded by runManifestDemo.sh when run by hand also seems fine:

 $tree lsst_dm_stack_demo-master/bin/ lsst_dm_stack_demo-master/bin/ ├── check_astrometry ├── compare ├── demo.sh └── export-results  Show Joshua Hoblitt added a comment - The tarball as downloaded by runManifestDemo.sh when run by hand also seems fine:$ tree lsst_dm_stack_demo-master/bin/ lsst_dm_stack_demo-master/bin/ ├── check_astrometry ├── compare ├── demo.sh └── export-results
Hide
Joshua Hoblitt added a comment -

https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141 succeeded. Either a side effect of my status/debugging PR fixed the problem or its transitory. The later would most likely mean that the github tarball download was somehow bad.

Show
Joshua Hoblitt added a comment - https://ci.lsst.codes/blue/organizations/jenkins/release%2Fnightly-release/detail/nightly-release/257/pipeline/141 succeeded. Either a side effect of my status/debugging PR fixed the problem or its transitory. The later would most likely mean that the github tarball download was somehow bad.
Hide
Joshua Hoblitt added a comment -

I found a smoking gun and, rather regrettably, github is to blame. I've attached lsst_dm_stack_demo-b3542.tar.gz which was taken from the workspace for the science-pipelines/lsst_distrib clean build last night for centos-6.py3 running on el7-1.

 $tree lsst_dm_stack_demo-b3542 lsst_dm_stack_demo-b3542 ├── demo.sh ├── demo_small.sh └── export-results   0 directories, 3 files  Show Joshua Hoblitt added a comment - I found a smoking gun and, rather regrettably, github is to blame. I've attached lsst_dm_stack_demo-b3542.tar.gz which was taken from the workspace for the science-pipelines/lsst_distrib clean build last night for centos-6.py3 running on el7-1 .$ tree lsst_dm_stack_demo-b3542 lsst_dm_stack_demo-b3542 ├── demo.sh ├── demo_small.sh └── export-results   0 directories, 3 files
Hide
Joshua Hoblitt added a comment - - edited

I've opened a PR that adds some sanity checking that the demo scripts at least exist, a --debug flag that prints the xtrace (only) for commands being executed, and some assorted cleanups.

If this problem reoccurs, the demo retrival mechanism should probably be changed to be a git clone or, better yet, the demo converted into an eups product.

Show
Joshua Hoblitt added a comment - - edited I've opened a PR that adds some sanity checking that the demo scripts at least exist, a --debug flag that prints the xtrace (only) for commands being executed, and some assorted cleanups. If this problem reoccurs, the demo retrival mechanism should probably be changed to be a git clone or, better yet, the demo converted into an eups product.
Hide
Joshua Hoblitt added a comment -

The status/debug PR has been merged.  I'm closing this PR and crossing my fingers.

Show
Joshua Hoblitt added a comment - The status/debug PR has been merged.  I'm closing this PR and crossing my fingers.

## People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Reviewers:
Tim Bond
Watchers:
Fred Moolekamp, Joshua Hoblitt, Paul Price, Tim Bond, Tim Jenness