Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1357

CI use of demo should use git branch preferences

    Details

      Description

      When we need to make modifications to the demo script and/or its expected results to account for a change in the pipeline, it'd be nice to be able to test that with buildbot on a branch before merging to master. Currently, however, buildbot always uses the master branch of the demo package, rather than the branches provided when starting the build.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Implemented as part of DM-14631

            Show
            jhoblitt Joshua Hoblitt added a comment - Implemented as part of DM-14631
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            The previously mentioned reworking of runManifestDemo.sh wasn't helpful when running under CI as it was being passed the "manifest id", aka, the bXXXX, number which lsst_build creates as an eups tag. I've taken another stab at it and renamed the exist -tag > -eups-tag and added a new git-ref, which can be specified multiple times. lsstswBuild.sh is splitting up the BRANCH list and passing each entry as a -git-ref.

            https://github.com/lsst-sqre/ci-scripts/pull/52

            This was needed to try to sort out breakage of the RC candidates by demo changes but may also resolve this issue.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited The previously mentioned reworking of runManifestDemo.sh wasn't helpful when running under CI as it was being passed the "manifest id", aka, the bXXXX , number which lsst_build creates as an eups tag. I've taken another stab at it and renamed the exist - tag > -eups-tag and added a new git-ref , which can be specified multiple times. lsstswBuild.sh is splitting up the BRANCH list and passing each entry as a -git-ref . https://github.com/lsst-sqre/ci-scripts/pull/52 This was needed to try to sort out breakage of the RC candidates by demo changes but may also resolve this issue.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            runManifestDemo.sh was reworked in April of 2017 to accept a single --tag parameter that it attempts to match against a git ref specifically to enable end-users to be able to run the demo against official releases. This would need to be extended to support multiple refs to be really useful under the current CI workflow. However, this would not solve the problem of being able to select the correct version of the demo to run with arbitrary EUPS tags. Eg., w_2018_13 There have been several discussion on slack about this problem but I don't believe any of them were captured in jira. The current state of affairs means that past nightly/weekly releases can not be reliably rebuilt under CI with the demo enabled. Gabriele Comoretto was bitten overnight by changes to the stack+demo 2 days ago breaking the demo master branch against the EUPS v15_0_rc3 tag.

            My position is that the demo should be turned into an eups product, and added to lsst_apps, so that the correct version will be recorded under an eups tag. This would also allow end users that have performed an eups distrib install -t v42 lsst_apps to run the demo as something like setup lsst_demo; lsst_demo rather than needing to document a per release incantation.

            Show
            jhoblitt Joshua Hoblitt added a comment - runManifestDemo.sh was reworked in April of 2017 to accept a single --tag parameter that it attempts to match against a git ref specifically to enable end-users to be able to run the demo against official releases. This would need to be extended to support multiple refs to be really useful under the current CI workflow. However, this would not solve the problem of being able to select the correct version of the demo to run with arbitrary EUPS tags. Eg., w_2018_13 There have been several discussion on slack about this problem but I don't believe any of them were captured in jira. The current state of affairs means that past nightly/weekly releases can not be reliably rebuilt under CI with the demo enabled. Gabriele Comoretto was bitten overnight by changes to the stack+demo 2 days ago breaking the demo master branch against the EUPS v15_0_rc3 tag. My position is that the demo should be turned into an eups product, and added to lsst_apps , so that the correct version will be recorded under an eups tag. This would also allow end users that have performed an eups distrib install -t v42 lsst_apps to run the demo as something like setup lsst_demo; lsst_demo rather than needing to document a per release incantation.
            Hide
            tjenness Tim Jenness added a comment -

            Yes. although you'd need to change the way the shell script is triggered. That would put it more in line with how lsst_ci is triggered (and at that point you would make demo depend on lsst_ci or lsst_ci depend on demo). Then we could remove the special demo logic from jenkins.

            Show
            tjenness Tim Jenness added a comment - Yes. although you'd need to change the way the shell script is triggered. That would put it more in line with how lsst_ci is triggered (and at that point you would make demo depend on lsst_ci or lsst_ci depend on demo). Then we could remove the special demo logic from jenkins.
            Hide
            jbosch Jim Bosch added a comment -

            Couldn't we just make the demo a regular eupspkg package?

            Show
            jbosch Jim Bosch added a comment - Couldn't we just make the demo a regular eupspkg package?
            Hide
            tjenness Tim Jenness added a comment -

            Don't think so. This is the thing that needs to support git refs: https://github.com/lsst-sqre/buildbot-scripts/blob/master/runManifestDemo.sh (and it would have to do the lsst-build thing of trying multiple ones until it gets one that works).

            Show
            tjenness Tim Jenness added a comment - Don't think so. This is the thing that needs to support git refs: https://github.com/lsst-sqre/buildbot-scripts/blob/master/runManifestDemo.sh (and it would have to do the lsst-build thing of trying multiple ones until it gets one that works).
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            I just hit this issue again on DM-10902. Any update?

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - I just hit this issue again on DM-10902 . Any update?

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Frossie Economou, Gabriele Comoretto, Jim Bosch, John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Perry Gee, Pim Schellart [X] (Inactive), Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel