Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11794

tag-git-repos failed due to eups distrib tag and versiondb being out of sync

    Details

      Description

      This was presumably caused by the the build/publish being repeated several times due to failures, and the most recently published distrib tag not being visible due to s3's eventually consistency when replacing an object, or a slow s3 -> gcloud sync. Adding a sleep into the release pipeline might largely mitigate this problem. It might also help to stop running the build/publish/git-tag as a single triggered job and trigger each of these operations separately from the release pipeline.

      https://raw.githubusercontent.com/lsst/versiondb/master/manifests/b3044.txt
      Traceback (most recent call last):
        File "/home/jenkins-slave/workspace/release/tag-git-repos/venv/bin/github-tag-version", line 9, in <module>
          load_entry_point('sqre-codekit==3.1.0', 'console_scripts', 'github-tag-version')()
        File "/home/jenkins-slave/workspace/release/tag-git-repos/venv/lib/python2.7/site-packages/codekit/cli/github_tag_version.py", line 258, in main
          debug=args.debug)
        File "/home/jenkins-slave/workspace/release/tag-git-repos/venv/lib/python2.7/site-packages/codekit/codetools.py", line 265, in eups2git_ref
          'not match manifest', eups_ref, eupsver)
      RuntimeError: ('Something has gone wrong, release file does not match manifest', '13.0-94-ga1c4440+6', '13.0-95-g793934f')
      Build step 'Execute shell' marked build as failure
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            A 15min sleep has been added to weekly-release and nightly-release after any operation that pushes to s3. This issue is being left open to monitor how these changes in production.

            Show
            jhoblitt Joshua Hoblitt added a comment - A 15min sleep has been added to weekly-release and nightly-release after any operation that pushes to s3. This issue is being left open to monitor how these changes in production.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The refactoring broke the parameter setup for triggering run-rebuild. This has been fixed on master and will be tested live since the nightly failed last night.

            Show
            jhoblitt Joshua Hoblitt added a comment - The refactoring broke the parameter setup for triggering run-rebuild . This has been fixed on master and will be tested live since the nightly failed last night.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            This problem has not been observed since the 15min sleeps were added so I'm going to close out this ticket.

            Show
            jhoblitt Joshua Hoblitt added a comment - This problem has not been observed since the 15min sleeps were added so I'm going to close out this ticket.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel