Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7556

Remove lsst_ci from demo in Jenkins and add to default product list

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Please add lsst_ci to the default product list (along with lsst_distrib and lsst_sims) to make it clear that lsst_ci is going to be built by default. Revert the "Skip Demo" behavior to the old behavior prior to DM-5433, of running the stack demo or not running the stack demo (which only requires lsst_apps to be built). The current situation has caused much confusion.

      This ticket was initially a request to make it clear in the text associated with "Skip Demo" that choosing the demo will cause lsst_ci to be built.

        Attachments

          Issue Links

            Activity

            No builds found.
            frossie Frossie Economou created issue -
            frossie Frossie Economou made changes -
            Field Original Value New Value
            Epic Link DM-7555 [ 26708 ]
            tjenness Tim Jenness made changes -
            Description
            'Skip Demo' wording in Jenkins should change to make it clear that no matter what you want built, ticking the box will build the kitchen sink.

            See:

            https://jira.lsstcorp.org/browse/DM-5433

            (I would also argue that adding lsst_ci is is not desired behaviour at all for run-rebuild but never mind that for now. )

            'Skip Demo' wording in Jenkins should change to make it clear that no matter what you want built, ticking the box will build the kitchen sink.

            See: DM-5433.

            (I would also argue that adding lsst_ci is is not desired behaviour at all for run-rebuild but never mind that for now. )

            tjenness Tim Jenness made changes -
            Link This issue is triggered by DM-5433 [ DM-5433 ]
            tjenness Tim Jenness made changes -
            Watchers Frossie Economou [ Frossie Economou ] Frossie Economou, Michael Wood-Vasey [ Frossie Economou, Michael Wood-Vasey ]
            tjenness Tim Jenness made changes -
            Description 'Skip Demo' wording in Jenkins should change to make it clear that no matter what you want built, ticking the box will build the kitchen sink.

            See: DM-5433.

            (I would also argue that adding lsst_ci is is not desired behaviour at all for run-rebuild but never mind that for now. )

            'Skip Demo' wording in Jenkins should change to make it clear that no matter what you want built, leaving this box unchecked will build the kitchen sink.

            See: DM-5433.

            (I would also argue that adding lsst_ci is is not desired behaviour at all for run-rebuild but never mind that for now. )

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            What would you like the description to read? The deps pulled in are only what is needed to run said tests.

            Show
            jhoblitt Joshua Hoblitt added a comment - What would you like the description to read? The deps pulled in are only what is needed to run said tests.
            Hide
            tjenness Tim Jenness added a comment -

            See for example Michael Wood-Vasey's comment on DM-5433: https://jira.lsstcorp.org/browse/DM-5433?focusedCommentId=46294&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-46294

            It currently says:

            Do not run the demo after all packages have completed building.

            but people have no real idea that your definition of demo now includes lsst_ci. We had huge confusion over this during the Python 3 porting extravaganza where jobs were failing all over the place because they forgot to skip the demo. Changing it to:

            Do not run the demo after all packages have completed building and do not include lsst_ci in the product list.

            or something, would help mental triggers.

            Historically, the demo did not pull extra packages into the build, it just failed if lsst_apps was not available, but that was at least after all the requested packages built. Conceptually, DM-5433 made a huge difference to the list of products being built. Before that ticket: the system built what you asked for regardless of the demo flag. After that ticket you got many extra packages built. It would have been clearer to add lsst_ci to the list of default packages in the PRODUCT entry along with lsst_sims and lsst_distrib.

            I agree with Frossie that run-rebuild should only ever build the packages you are requesting.

            Show
            tjenness Tim Jenness added a comment - See for example Michael Wood-Vasey 's comment on DM-5433 : https://jira.lsstcorp.org/browse/DM-5433?focusedCommentId=46294&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-46294 It currently says: Do not run the demo after all packages have completed building. but people have no real idea that your definition of demo now includes lsst_ci . We had huge confusion over this during the Python 3 porting extravaganza where jobs were failing all over the place because they forgot to skip the demo. Changing it to: Do not run the demo after all packages have completed building and do not include lsst_ci in the product list. or something, would help mental triggers. Historically, the demo did not pull extra packages into the build, it just failed if lsst_apps was not available, but that was at least after all the requested packages built. Conceptually, DM-5433 made a huge difference to the list of products being built. Before that ticket: the system built what you asked for regardless of the demo flag. After that ticket you got many extra packages built. It would have been clearer to add lsst_ci to the list of default packages in the PRODUCT entry along with lsst_sims and lsst_distrib . I agree with Frossie that run-rebuild should only ever build the packages you are requesting.
            Hide
            ctslater Colin Slater added a comment - - edited

            "It would have been clearer to add lsst_ci to the list of default packages in the PRODUCT entry along with lsst_sims and lsst_distrib."

            Seconding this. Much better solution than having some complicated text to describe the demo checkbox.

            Show
            ctslater Colin Slater added a comment - - edited "It would have been clearer to add lsst_ci to the list of default packages in the PRODUCT entry along with lsst_sims and lsst_distrib." Seconding this. Much better solution than having some complicated text to describe the demo checkbox.
            Hide
            rowen Russell Owen added a comment -

            I agree with Tim Jenness and Colin Slater (as you have heard me say before). It would be a huge help to all developers if the product list showed the actual list of products passed to lsstsw's "rebuild" command.

            Show
            rowen Russell Owen added a comment - I agree with Tim Jenness and Colin Slater (as you have heard me say before). It would be a huge help to all developers if the product list showed the actual list of products passed to lsstsw's "rebuild" command.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            +1

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - +1
            Hide
            tjenness Tim Jenness added a comment -

            Michael Wood-Vasey is that +1 to change the wording or +1 to add lsst_ci to the product list?

            Show
            tjenness Tim Jenness added a comment - Michael Wood-Vasey is that +1 to change the wording or +1 to add lsst_ci to the product list?
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            +1 to add `lsst_ci` to the product list and not have separate logic in the demo.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - +1 to add `lsst_ci` to the product list and not have separate logic in the demo.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            We need to in general re-think the demo and its invocation. We also need to think and plan out how to run more extensive tests as part of lsst_ci. But I suggest that at this point there are significant practical gains for clarity to devs to defer those large discussions, and implement lsst_ci as an explicit package to be built.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - We need to in general re-think the demo and its invocation. We also need to think and plan out how to run more extensive tests as part of lsst_ci . But I suggest that at this point there are significant practical gains for clarity to devs to defer those large discussions, and implement lsst_ci as an explicit package to be built.
            jsick Jonathan Sick made changes -
            Assignee Joshua Hoblitt [ jhoblitt ] Jonathan Sick [ jsick ]
            jsick Jonathan Sick made changes -
            Link This issue relates to DM-8158 [ DM-8158 ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-7555 [ 26708 ] DM-7491 [ 26620 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-8360 [ DM-8360 ]
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-8360 [ DM-8360 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-8360 [ DM-8360 ]
            Hide
            tjenness Tim Jenness added a comment -

            Why is it taking so long to sort this out? All we want is lsst_ci to be added as an explicit package in the default list and for the demo to be just the demo.

            Show
            tjenness Tim Jenness added a comment - Why is it taking so long to sort this out? All we want is lsst_ci to be added as an explicit package in the default list and for the demo to be just the demo.
            Hide
            jsick Jonathan Sick added a comment -

            Joshua Hoblitt assigned it to me as a learning opportunity but I've been completely loaded this cycle and haven't gotten to this.

            Show
            jsick Jonathan Sick added a comment - Joshua Hoblitt assigned it to me as a learning opportunity but I've been completely loaded this cycle and haven't gotten to this.
            Hide
            tjenness Tim Jenness added a comment - - edited

            Ah, I was wondering why Joshua Hoblitt was no longer the assignee...

            Show
            tjenness Tim Jenness added a comment - - edited Ah, I was wondering why Joshua Hoblitt was no longer the assignee...
            frossie Frossie Economou made changes -
            Epic Link DM-7491 [ 26620 ] DM-8481 [ 28113 ]
            Hide
            tjenness Tim Jenness added a comment -

            I would like to update the description and title of this ticket to reflect the strong feeling in the comments that the real fix here is to remove lsst_ci from the default demo. When this ticket was going to be a five minute tweak to the UI it made sense to keep it simple but it's not clear we are going to get the wording change and we would all rather get the real fix done. I won't make the change immediately, but give time for people to jump in and tell me to create a new ticket.

            Show
            tjenness Tim Jenness added a comment - I would like to update the description and title of this ticket to reflect the strong feeling in the comments that the real fix here is to remove lsst_ci from the default demo. When this ticket was going to be a five minute tweak to the UI it made sense to keep it simple but it's not clear we are going to get the wording change and we would all rather get the real fix done. I won't make the change immediately, but give time for people to jump in and tell me to create a new ticket.
            tjenness Tim Jenness made changes -
            Summary Skip Demo wording in Jenkins Remove lsst_ci from demo in Jenkins
            tjenness Tim Jenness made changes -
            Description 'Skip Demo' wording in Jenkins should change to make it clear that no matter what you want built, leaving this box unchecked will build the kitchen sink.

            See: DM-5433.

            (I would also argue that adding lsst_ci is is not desired behaviour at all for run-rebuild but never mind that for now. )

            Please add {{lsst_ci}} to the default product list (along with {{lsst_distrib}} and {{lsst_sims}}) to make it clear that {{lsst_ci}} is going to be built by default. Revert the "Skip Demo" behavior to the old behavior prior to DM-5433, of running the stack demo or not running the stack demo (which only requires {{lsst_apps}} to be built). The current situation has caused much confusion.

            This ticket was initially a request to make it clear in the text associated with "Skip Demo" that choosing the demo will cause {{lsst_ci}} to be built.

            tjenness Tim Jenness made changes -
            Summary Remove lsst_ci from demo in Jenkins Remove lsst_ci from demo in Jenkins and add to default product list
            Hide
            tjenness Tim Jenness added a comment - - edited

            As threatened last week I have updated this ticket to reflect the consensus that we wish lsst_ci to be added to the default product list along with lsst_distrib and lsst_sims and for DM-5433 to be reverted so that demo means demo. I imagine you may want to change the Assignee.

            Show
            tjenness Tim Jenness added a comment - - edited As threatened last week I have updated this ticket to reflect the consensus that we wish lsst_ci to be added to the default product list along with lsst_distrib and lsst_sims and for DM-5433 to be reverted so that demo means demo. I imagine you may want to change the Assignee.
            Hide
            tjenness Tim Jenness added a comment -

            This was the commit that added lsst_ci to the demo: https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/23/commits/2356e4f95210e23f2911816552e587b28c232f22

            It looks like that commit can't be reverted because it has unrelated changes included.

            Show
            tjenness Tim Jenness added a comment - This was the commit that added lsst_ci to the demo: https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/23/commits/2356e4f95210e23f2911816552e587b28c232f22 It looks like that commit can't be reverted because it has unrelated changes included.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Note that the files touched in DM-5433 don't even exist anymore.

            Show
            jhoblitt Joshua Hoblitt added a comment - Note that the files touched in DM-5433 don't even exist anymore.
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-8481 [ 28113 ] DM-8445 [ 28061 ]
            Hide
            frossie Frossie Economou added a comment -

            Sorry, I am confused as to why this is assigned to Jonathan, assigning back to Josh.

            I agree the current behaviour is very confusing.

            We're about to do some major overhaul of both the CI user interface and the jobs list, but in the meanwhile Josh, just add some text to the UI so instead of "Skip demo?" it says "Skip demo and lsst_ci?"

            Show
            frossie Frossie Economou added a comment - Sorry, I am confused as to why this is assigned to Jonathan, assigning back to Josh. I agree the current behaviour is very confusing. We're about to do some major overhaul of both the CI user interface and the jobs list, but in the meanwhile Josh, just add some text to the UI so instead of "Skip demo?" it says "Skip demo and lsst_ci?"
            frossie Frossie Economou made changes -
            Assignee Jonathan Sick [ jsick ] Joshua Hoblitt [ jhoblitt ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-8445 [ 28061 ] DM-10099 [ 31683 ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-10099 [ 31683 ] DM-5417 [ 23241 ]
            jhoblitt Joshua Hoblitt made changes -
            Epic Link DM-5417 [ 23241 ] DM-14566 [ 94441 ]
            jhoblitt Joshua Hoblitt made changes -
            Watchers Colin Slater, Frossie Economou, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness [ Colin Slater, Frossie Economou, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness ] Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness [ Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness ]
            jhoblitt Joshua Hoblitt made changes -
            Risk Score 0
            jhoblitt Joshua Hoblitt made changes -
            Link This issue is blocked by DM-14806 [ DM-14806 ]
            jhoblitt Joshua Hoblitt made changes -
            Watchers Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness [ Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Tim Jenness ] Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Simon Krughoff, Tim Jenness [ Colin Slater, Frossie Economou, Gabriele Comoretto, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Simon Krughoff, Tim Jenness ]
            jhoblitt Joshua Hoblitt made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Per a few discussions on slack over the last couple of days, the way forward seems to be unlocked now in that the demo, which is now (also) a proper eups product, be added to lsst_ci, that lsst_ci be added to the default products list, and the SKIP_DEMO button be removed completely.

            Show
            jhoblitt Joshua Hoblitt added a comment - Per a few discussions on slack over the last couple of days, the way forward seems to be unlocked now in that the demo, which is now (also) a proper eups product, be added to lsst_ci , that lsst_ci be added to the default products list, and the SKIP_DEMO button be removed completely.
            jhoblitt Joshua Hoblitt made changes -
            Story Points 0.5
            jhoblitt Joshua Hoblitt made changes -
            Story Points 0.5 1.5
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Summary of changes implimented:

            • The SKIP_DEMO checkbox has been removed from various jenkins jobs
            • the runManifestDemo.sh demo driver script has been removed from the ci-scripts
            • the env vars used by jenkins/ci-scripts to configure a build were overhauled, including adding a prefix of LSST_ to most env vars.
            • the hard-coded product list (previously) of lsst_distrib, that was in a surprising number of jenkins jobs has been centralized to jenkins-dm-jobs:/etc/scipipe/build_matrix.yaml
            • miscellaneous cleanups
            Show
            jhoblitt Joshua Hoblitt added a comment - Summary of changes implimented: The SKIP_DEMO checkbox has been removed from various jenkins jobs the runManifestDemo.sh demo driver script has been removed from the ci-scripts the env vars used by jenkins/ ci-scripts to configure a build were overhauled, including adding a prefix of LSST_ to most env vars. the hard-coded product list (previously) of lsst_distrib , that was in a surprising number of jenkins jobs has been centralized to jenkins-dm-jobs:/etc/scipipe/build_matrix.yaml miscellaneous cleanups
            jhoblitt Joshua Hoblitt made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              frossie Frossie Economou
              Watchers:
              Colin Slater, Frossie Economou, Gabriele Comoretto [X] (Inactive), Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Russell Owen, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.