Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1357

CI use of demo should use git branch preferences

    Details

      Description

      When we need to make modifications to the demo script and/or its expected results to account for a change in the pipeline, it'd be nice to be able to test that with buildbot on a branch before merging to master. Currently, however, buildbot always uses the master branch of the demo package, rather than the branches provided when starting the build.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue is duplicated by DM-1399 [ DM-1399 ]
            tjenness Tim Jenness made changes -
            Component/s Continuous Integration [ 10711 ]
            Component/s Developer Infrastructure [ 10712 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-3862 [ DM-3862 ]
            frossie Frossie Economou made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Invalid [ 11005 ]
            jhoblitt Joshua Hoblitt made changes -
            Resolution Done [ 10000 ]
            Status Invalid [ 11005 ] To Do [ 10001 ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-5417 [ 23241 ]
            jhoblitt Joshua Hoblitt made changes -
            Component/s buildbot [ 11304 ]
            Summary buildbot use of demo should use git branch preferences CI use of demo should use git branch preferences
            jhoblitt Joshua Hoblitt made changes -
            Assignee Frossie Economou [ frossie ] Joshua Hoblitt [ jhoblitt ]
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            I just hit this issue again on DM-10902. Any update?

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - I just hit this issue again on DM-10902 . Any update?
            Hide
            tjenness Tim Jenness added a comment -

            Don't think so. This is the thing that needs to support git refs: https://github.com/lsst-sqre/buildbot-scripts/blob/master/runManifestDemo.sh (and it would have to do the lsst-build thing of trying multiple ones until it gets one that works).

            Show
            tjenness Tim Jenness added a comment - Don't think so. This is the thing that needs to support git refs: https://github.com/lsst-sqre/buildbot-scripts/blob/master/runManifestDemo.sh (and it would have to do the lsst-build thing of trying multiple ones until it gets one that works).
            Hide
            jbosch Jim Bosch added a comment -

            Couldn't we just make the demo a regular eupspkg package?

            Show
            jbosch Jim Bosch added a comment - Couldn't we just make the demo a regular eupspkg package?
            Hide
            tjenness Tim Jenness added a comment -

            Yes. although you'd need to change the way the shell script is triggered. That would put it more in line with how lsst_ci is triggered (and at that point you would make demo depend on lsst_ci or lsst_ci depend on demo). Then we could remove the special demo logic from jenkins.

            Show
            tjenness Tim Jenness added a comment - Yes. although you'd need to change the way the shell script is triggered. That would put it more in line with how lsst_ci is triggered (and at that point you would make demo depend on lsst_ci or lsst_ci depend on demo). Then we could remove the special demo logic from jenkins.
            rowen Russell Owen made changes -
            Labels PairCoding
            jhoblitt Joshua Hoblitt made changes -
            Watchers Jim Bosch, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness [ Jim Bosch, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness ] Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness [ Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness ]
            jhoblitt Joshua Hoblitt made changes -
            Watchers Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness [ Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness ] Frossie Economou, Gabriele Comoretto, Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness [ Frossie Economou, Gabriele Comoretto, Jim Bosch, John Swinbank, Jonathan Sick, Kian-Tat Lim, Perry Gee, Pim Schellart, Tim Jenness ]
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            runManifestDemo.sh was reworked in April of 2017 to accept a single --tag parameter that it attempts to match against a git ref specifically to enable end-users to be able to run the demo against official releases. This would need to be extended to support multiple refs to be really useful under the current CI workflow. However, this would not solve the problem of being able to select the correct version of the demo to run with arbitrary EUPS tags. Eg., w_2018_13 There have been several discussion on slack about this problem but I don't believe any of them were captured in jira. The current state of affairs means that past nightly/weekly releases can not be reliably rebuilt under CI with the demo enabled. Gabriele Comoretto was bitten overnight by changes to the stack+demo 2 days ago breaking the demo master branch against the EUPS v15_0_rc3 tag.

            My position is that the demo should be turned into an eups product, and added to lsst_apps, so that the correct version will be recorded under an eups tag. This would also allow end users that have performed an eups distrib install -t v42 lsst_apps to run the demo as something like setup lsst_demo; lsst_demo rather than needing to document a per release incantation.

            Show
            jhoblitt Joshua Hoblitt added a comment - runManifestDemo.sh was reworked in April of 2017 to accept a single --tag parameter that it attempts to match against a git ref specifically to enable end-users to be able to run the demo against official releases. This would need to be extended to support multiple refs to be really useful under the current CI workflow. However, this would not solve the problem of being able to select the correct version of the demo to run with arbitrary EUPS tags. Eg., w_2018_13 There have been several discussion on slack about this problem but I don't believe any of them were captured in jira. The current state of affairs means that past nightly/weekly releases can not be reliably rebuilt under CI with the demo enabled. Gabriele Comoretto was bitten overnight by changes to the stack+demo 2 days ago breaking the demo master branch against the EUPS v15_0_rc3 tag. My position is that the demo should be turned into an eups product, and added to lsst_apps , so that the correct version will be recorded under an eups tag. This would also allow end users that have performed an eups distrib install -t v42 lsst_apps to run the demo as something like setup lsst_demo; lsst_demo rather than needing to document a per release incantation.
            jhoblitt Joshua Hoblitt made changes -
            Link This issue relates to DM-14631 [ DM-14631 ]
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            The previously mentioned reworking of runManifestDemo.sh wasn't helpful when running under CI as it was being passed the "manifest id", aka, the bXXXX, number which lsst_build creates as an eups tag. I've taken another stab at it and renamed the exist -tag > -eups-tag and added a new git-ref, which can be specified multiple times. lsstswBuild.sh is splitting up the BRANCH list and passing each entry as a -git-ref.

            https://github.com/lsst-sqre/ci-scripts/pull/52

            This was needed to try to sort out breakage of the RC candidates by demo changes but may also resolve this issue.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited The previously mentioned reworking of runManifestDemo.sh wasn't helpful when running under CI as it was being passed the "manifest id", aka, the bXXXX , number which lsst_build creates as an eups tag. I've taken another stab at it and renamed the exist - tag > -eups-tag and added a new git-ref , which can be specified multiple times. lsstswBuild.sh is splitting up the BRANCH list and passing each entry as a -git-ref . https://github.com/lsst-sqre/ci-scripts/pull/52 This was needed to try to sort out breakage of the RC candidates by demo changes but may also resolve this issue.
            jhoblitt Joshua Hoblitt made changes -
            Risk Score 0
            jhoblitt Joshua Hoblitt made changes -
            Link This issue relates to DM-14806 [ DM-14806 ]
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Implemented as part of DM-14631

            Show
            jhoblitt Joshua Hoblitt added a comment - Implemented as part of DM-14631
            jhoblitt Joshua Hoblitt made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Done [ 10002 ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-5417 [ 23241 ] DM-14567 [ 94448 ]
            tjenness Tim Jenness made changes -
            Component/s lsst_dm_stack_demo [ 12401 ]

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Frossie Economou, Gabriele Comoretto, Jim Bosch, John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Perry Gee, Pim Schellart [X] (Inactive), Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel