Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12047

run nightly "clean" builds of lsst_distrib & qserv_distrib

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Run 'clean', ie, starting from a sanitized build env with no existing state, builds of lsst_distrib, qserv_distrib, and dax_webserv on a nightly basis.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          John Swinbank Do you have any objection to removing the jenkins root level lsst_apps, lsst_distrib, & lsst_obs jobs?

          Show
          jhoblitt Joshua Hoblitt added a comment - John Swinbank Do you have any objection to removing the jenkins root level lsst_apps, lsst_distrib, & lsst_obs jobs?
          Hide
          swinbank John Swinbank added a comment -

          I'm not sure I really understand the question.

          I think we regularly need to exercise building all of the codebase (at least, all of the Pipelines code; I assume similar applies to Qserv, but I'll let Fritz speak to that).

          I think we need to be able to see the logs of those builds, ie not have them wiped out overnight.

          I think it needs to be easy for developers to trigger new builds on demand.

          I'm not attached to those particular labels appearing in their current place on the UI.

          I don't really know exactly what you're asking here.

          Show
          swinbank John Swinbank added a comment - I'm not sure I really understand the question. I think we regularly need to exercise building all of the codebase (at least, all of the Pipelines code; I assume similar applies to Qserv, but I'll let Fritz speak to that). I think we need to be able to see the logs of those builds, ie not have them wiped out overnight. I think it needs to be easy for developers to trigger new builds on demand. I'm not attached to those particular labels appearing in their current place on the UI. I don't really know exactly what you're asking here.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The scope of this task is to run builds once a day that builds all of the packages from source. This differs from stack-os-matrix that preserves state so that it doesn't always take ~3 hours for a CI run. At present, there are several jobs {lsst_apps, lsst_distrib, & lsst_obs}} (and qserv/qserv_distrib & qserv/dax_webserv) that run on a timer and trigger builds of stack-os-matrix. As it appears that these jobs are virtually unused by developers, and lsst_apps/lsst_obs are part of lsst_distrib, I'm proposing that they be removed completely and replaced by a new "clean" job with a name along the lines of science_pipelines/lsst_distrib. Similarly, the existing qserv/qserv_distrib & qserv/dax_webserv jobs be converted from triggering stack-os-matrix to stand alone "clean" jobs.

          Show
          jhoblitt Joshua Hoblitt added a comment - The scope of this task is to run builds once a day that builds all of the packages from source. This differs from stack-os-matrix that preserves state so that it doesn't always take ~3 hours for a CI run. At present, there are several jobs {lsst_apps, lsst_distrib, & lsst_obs}} (and qserv/qserv_distrib & qserv/dax_webserv ) that run on a timer and trigger builds of stack-os-matrix . As it appears that these jobs are virtually unused by developers, and lsst_apps/lsst_obs are part of lsst_distrib , I'm proposing that they be removed completely and replaced by a new "clean" job with a name along the lines of science_pipelines/lsst_distrib . Similarly, the existing qserv/qserv_distrib & qserv/dax_webserv jobs be converted from triggering stack-os-matrix to stand alone "clean" jobs.
          Hide
          swinbank John Swinbank added a comment -

          I think that all sounds fine. Specifically, I'm happy that:

          • The existing lsst_apps, lsst_distrib and lsst_obs jobs will disappear;
          • Developers will still be able to trigger builds of those top level packages using the stack-os-matrix job;
          • A nightly build of all of the Pipelines code, starting from scratch, will be introduced.

          Will the output of that nightly job then be used to seed future, developer-requested, runs of stack-os-matrix?

          Show
          swinbank John Swinbank added a comment - I think that all sounds fine. Specifically, I'm happy that: The existing lsst_apps , lsst_distrib and lsst_obs jobs will disappear; Developers will still be able to trigger builds of those top level packages using the stack-os-matrix job; A nightly build of all of the Pipelines code, starting from scratch, will be introduced. Will the output of that nightly job then be used to seed future, developer-requested, runs of stack-os-matrix ?
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The proposed branch has been merged to master / deployed. I ended up having to manually remove the prior qserv jobs with identical jobs names to the new implementation. I'm placing the ticket into self review until all of the new jobs have successfully run.

          Show
          jhoblitt Joshua Hoblitt added a comment - The proposed branch has been merged to master / deployed. I ended up having to manually remove the prior qserv jobs with identical jobs names to the new implementation. I'm placing the ticket into self review until all of the new jobs have successfully run.
          Show
          jhoblitt Joshua Hoblitt added a comment - All of the new "clean" build jobs ran successfully overnight. Note that ci_hsci has been confirmed as run as part of science-pipelines/lsst_distrib . https://ci.lsst.codes/blue/organizations/jenkins/science-pipelines%2Flsst_distrib/detail/lsst_distrib/1/pipeline https://ci.lsst.codes/blue/organizations/jenkins/qserv%2Fqserv_distrib/activity https://ci.lsst.codes/blue/organizations/jenkins/qserv%2Fdax_webserv/detail/dax_webserv/1/pipeline

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Fritz Mueller, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel