Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-466

lsst-build updates based on feedback from 1st month of use

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      6
    • Sprint:
      DevOps Sprint 1, DevOps Sprint 2
    • Team:
      SQuaRE

      Description

      change lsst-build interface to:

      • rename 'prepare' to 'clone'
      • use current directory to clone to
      • remove the default REPOSITORY_PATTERN, as it isn't complete
      • use the current username as default build tag prefix
      • don't write the manifest by default; use --manifest=<filename>
        option instead
      • refuse to change/overwrite repositories if they're dirty, use
        --force to override
      • create a separate 'version' verb
      • ProductFetcher.fetch doesn't need to return ref, sha1
      • read config file within build directory, .btconfig
      • rename the binary to bt
      • have the versioner use git db by default, but fall back to generating versions from hash by default

      See the linked web page for a mock of the command line interface.

        Attachments

          Issue Links

            Activity

            No builds found.
            mjuric Mario Juric created issue -
            mjuric Mario Juric made changes -
            Field Original Value New Value
            Assignee Mario Juric [ mjuric ]
            Status To Do [ 10001 ] In Progress [ 3 ]
            mjuric Mario Juric made changes -
            Rank Ranked higher
            mjuric Mario Juric made changes -
            Story Points 6
            mjuric Mario Juric made changes -
            Rank Ranked higher
            mjuric Mario Juric made changes -
            Sprint DevOps Sprint 1 [ 41 ]
            mjuric Mario Juric made changes -
            Rank Ranked higher
            mjuric Mario Juric made changes -
            Sprint DevOps Sprint 1 [ 41 ]
            mjuric Mario Juric made changes -
            Sprint DevOps 1 [ 42 ]
            Hide
            ktl Kian-Tat Lim added a comment -

            What are you planning to use for a config file format? Python ConfigParser? json? YAML?

            Show
            ktl Kian-Tat Lim added a comment - What are you planning to use for a config file format? Python ConfigParser ? json? YAML?
            Hide
            mjuric Mario Juric added a comment -

            Just ConfigParser. Can't use pex_config since lsst-build needs to work before anything else has been installed.


            Sent from my Android device with K-9 Mail. Please excuse my brevity.

            Show
            mjuric Mario Juric added a comment - Just ConfigParser. Can't use pex_config since lsst-build needs to work before anything else has been installed. – Sent from my Android device with K-9 Mail. Please excuse my brevity.
            mjuric Mario Juric made changes -
            Remote Link This issue links to "Wiki Page (Confluence)" [ 10306 ]
            mjuric Mario Juric made changes -
            Remote Link This issue links to "Wiki Page (Confluence)" [ 10306 ]
            mjuric Mario Juric made changes -
            Remote Link This issue links to "Wiki Page (Confluence)" [ 10307 ]
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited

            Here's another quick item for this (or a new Jira Issue, your choice) wrap-up issue.

            'lsst-build build' does not return the exit status of the overall operation on its completion. Ie. in build.py method: 'def run (args)', the final line is: "b.build()". Since you trapped the originating exception in order to do clean-up processing, the exception doesn't bleed through to buildbot.

            For testing, I added:

            retcode = b.build()
                    print "Final retcode= %s" %retcode
                    if not retcode:
                        exit(1)

            The prints were for debugging only. A simple "exit(retcode)" would be sufficient.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited Here's another quick item for this (or a new Jira Issue, your choice) wrap-up issue. 'lsst-build build' does not return the exit status of the overall operation on its completion. Ie. in build.py method: 'def run (args)', the final line is: "b.build()". Since you trapped the originating exception in order to do clean-up processing, the exception doesn't bleed through to buildbot. For testing, I added: retcode = b.build() print "Final retcode= %s" % retcode if not retcode: exit( 1 ) The prints were for debugging only. A simple "exit(retcode)" would be sufficient.
            mjuric Mario Juric made changes -
            Link This issue is duplicated by DM-571 [ DM-571 ]
            mjuric Mario Juric made changes -
            Description change lsst-build interface to:
              * rename 'prepare' to 'clone'
              * use current directory to clone to
              * remove the default REPOSITORY_PATTERN, as it isn't complete
              * use the current username as default build tag prefix
              * don't write the manifest by default; use --manifest=<filename>
            option instead
              * refuse to change/overwrite repositories if they're dirty, use
            --force to override
              * create a separate 'version' verb
              * ProductFetcher.fetch doesn't need to return ref, sha1
              * read config file within build directory, .btconfig
              * rename the binary to bt
              * have the versioner use git db by default, but fall back to generating versions from hash by default

            A mock of typical use:
            {code}
            bt init

            bt config repository_pattern ....
            bt config versiondb.repository git@....
            bt config versiondb.write true
            bt config versiondb.pull false
            bt config versiondb.push false
            bt config exclusions <filename>

            bt clone lsst_distrib --ref tickets/xxxx
            bt build lsst_distrib [...]

            bt versiondb pull

            bt init --config http://....
            cat ~/.btconfig
            {code}
            change lsst-build interface to:
              * rename 'prepare' to 'clone'
              * use current directory to clone to
              * remove the default REPOSITORY_PATTERN, as it isn't complete
              * use the current username as default build tag prefix
              * don't write the manifest by default; use --manifest=<filename>
            option instead
              * refuse to change/overwrite repositories if they're dirty, use
            --force to override
              * create a separate 'version' verb
              * ProductFetcher.fetch doesn't need to return ref, sha1
              * read config file within build directory, .btconfig
              * rename the binary to bt
              * have the versioner use git db by default, but fall back to generating versions from hash by default

            See the linked web page for a mock of the command line interface.
            robyn Robyn Allsman [X] (Inactive) made changes -
            Sprint DevOps Sprint 1 [ 42 ] DevOps Sprint 1, DevOps Sprint 2 [ 42, 52 ]
            robyn Robyn Allsman [X] (Inactive) made changes -
            Rank Ranked higher
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            I used lsst-build.git tickets/DM-271 to house the presumptive fix for returning the error status on exit from 'lsst-build build'. It's been in use by lsstsw2 BB tester for a few weeks. and has exercised the error return. You may cherry pick as appropriate.

            diff --git a/python/lsst/ci/build.py b/python/lsst/ci/build.py
            index c0823a8..bf8a7ec 100644
            — a/python/lsst/ci/build.py
            +++ b/python/lsst/ci/build.py
            @@ -237,4 +237,5 @@ class Builder(object):
            manifest = Manifest.fromFile(fp)

            b = Builder(build_dir, manifest, progress, eupsObj)

            • b.build()
              + retcode = b.build()
              + exit(retcode == 0)

            It's enough so the calling code can determine if there was a failure or not in order to take the correct action.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - I used lsst-build.git tickets/ DM-271 to house the presumptive fix for returning the error status on exit from 'lsst-build build'. It's been in use by lsstsw2 BB tester for a few weeks. and has exercised the error return. You may cherry pick as appropriate. diff --git a/python/lsst/ci/build.py b/python/lsst/ci/build.py index c0823a8..bf8a7ec 100644 — a/python/lsst/ci/build.py +++ b/python/lsst/ci/build.py @@ -237,4 +237,5 @@ class Builder(object): manifest = Manifest.fromFile(fp) b = Builder(build_dir, manifest, progress, eupsObj) b.build() + retcode = b.build() + exit(retcode == 0) It's enough so the calling code can determine if there was a failure or not in order to take the correct action.
            Hide
            mjuric Mario Juric added a comment -

            OK, will cherry pick when I get back, thanks!

            Sent from my Android device with K-9 Mail. Please excuse my brevity.

            Show
            mjuric Mario Juric added a comment - OK, will cherry pick when I get back, thanks! – Sent from my Android device with K-9 Mail. Please excuse my brevity.
            robyn Robyn Allsman [X] (Inactive) made changes -
            Link This issue relates to DM-889 [ DM-889 ]
            robyn Robyn Allsman [X] (Inactive) made changes -
            Rank Ranked higher
            robyn Robyn Allsman [X] (Inactive) made changes -
            Rank Ranked lower
            mjuric Mario Juric made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Won't Fix [ 10405 ]
            tjenness Tim Jenness made changes -
            Team SQuaRE [ 10302 ]
            Watchers Kian-Tat Lim, Mario Juric, Robyn Allsman [ Kian-Tat Lim, Mario Juric, Robyn Allsman ] Kian-Tat Lim, Mario Juric [ Kian-Tat Lim, Mario Juric ]

              People

              Assignee:
              mjuric Mario Juric
              Reporter:
              mjuric Mario Juric
              Watchers:
              Kian-Tat Lim, Mario Juric
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.