Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22511

Official-Release Jenkins job failure

    XMLWordPrintable

    Details

    • Team:
      Architecture

      Description

        Attachments

          Issue Links

            Activity

            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            Tim Jenness suggested tagging with 19.0.0 tag the new repository dax_apdb and replay the release job.

            Job 32 ended successfully.

            However, it remains to investigate the reasons for the failure.

             

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - Tim Jenness suggested tagging with 19.0.0 tag the new repository dax_apdb and replay the release job. Job 32 ended successfully. However, it remains to investigate the reasons for the failure.  
            Hide
            tjenness Tim Jenness added a comment - - edited

            The 19.0.0 tag ended up on the lsst-dm/dax_ppdb repository by mistake. The fix works because dax_apdb has all the history of the dax_ppdb package.

            My guess is that the tagging script assumes that every package being released lives in github in the lsst org with the same name. In this case the GitHub redirect triggered so that dax_ppdb was found in lsst-dm. The real fix I think is to use repos.yaml to map products to git repositories. If that was done then the correct new location would have been found. I have not verified this guess by looking at the code that does the tagging.

            Of course not renaming packages in the middle of a release cycle would help.

            Show
            tjenness Tim Jenness added a comment - - edited The 19.0.0 tag ended up on the lsst-dm/dax_ppdb repository by mistake. The fix works because dax_apdb has all the history of the dax_ppdb package. My guess is that the tagging script assumes that every package being released lives in github in the lsst org with the same name. In this case the GitHub redirect triggered so that dax_ppdb was found in lsst-dm. The real fix I think is to use repos.yaml to map products to git repositories. If that was done then the correct new location would have been found. I have not verified this guess by looking at the code that does the tagging. Of course not renaming packages in the middle of a release cycle would help.
            Hide
            tjenness Tim Jenness added a comment -

            Gabriele Comoretto [X] if this ticket is reporting the failure last night then I think we can close it because the manual tagging fixed the problem. If the intent is for this ticket to track an investigation into the tagging failure then the title and description should change.

            Show
            tjenness Tim Jenness added a comment - Gabriele Comoretto [X] if this ticket is reporting the failure last night then I think we can close it because the manual tagging fixed the problem. If the intent is for this ticket to track an investigation into the tagging failure then the title and description should change.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            As you mentioned, the problem has been work-around and the release is now available.

            I opened this ticket in order to give visibility to the problem, so it can be investigated and eventually fixed.

            We can close this issue when we agree on a way forward.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - As you mentioned, the problem has been work-around and the release is now available. I opened this ticket in order to give visibility to the problem, so it can be investigated and eventually fixed. We can close this issue when we agree on a way forward.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            The two repositories, lsst/dax_apdb, and lsst-dm/dax_ppdb, are supposed to have the same history, from what I have understood since the first is a copy of the second. But they don't.

            lsst/dax_apdb is missing w.2019.46 and v19.0.0.rc1 tags, that are instead visible in lsst-dm/dax_ppdb.

            This means, something when wrong during the copy, probably it has been done during the weekend when w.2019.46 has been tagged.

            It implies also, that the v19.0.0.rc1 has been done before dax_ppdb is moved from lsst to lsst-dm.

            We shall add the 2 tags to lsst/dax_apdb in order to make them consistent.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - The two repositories, lsst/dax_apdb, and lsst-dm/dax_ppdb, are supposed to have the same history, from what I have understood since the first is a copy of the second. But they don't. lsst/dax_apdb is missing w.2019.46 and v19.0.0.rc1 tags, that are instead visible in lsst-dm/dax_ppdb. This means, something when wrong during the copy, probably it has been done during the weekend when w.2019.46 has been tagged. It implies also, that the v19.0.0.rc1 has been done before dax_ppdb is moved from lsst to lsst-dm. We shall add the 2 tags to lsst/dax_apdb in order to make them consistent.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            Based on the above comment, it seems that DMTN-027 (Renaming an LSST git repository) may require an additional check:

            • ensure that no changes in the old package have been introduced after the fork.

            and for changes, I am not only referring to code changes, that are quite unlikely to happen but tags.

            In this case in fact, during the process, two tags have been added in the old one and have not been propagated to the new one.

            I can't be sure, but if this were the case, the official-release job 31 yesterday would not have failed, because of the correct v19.0.0.rc1 tag was found in the new lsst/dax_apdb, and the correct 19.0.0 would have been tagged in the right place.

            Said that I am quite reluctant to suggest a change in the tooling. If the renaming of the package is done properly, there should be no problems. In case of a missing tag, the workaround is trivial.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - Based on the above comment, it seems that DMTN-027 (Renaming an LSST git repository) may require an additional check: ensure that no changes in the old package have been introduced after the fork. and for changes, I am not only referring to code changes, that are quite unlikely to happen but tags. In this case in fact, during the process, two tags have been added in the old one and have not been propagated to the new one. I can't be sure, but if this were the case, the official-release job 31 yesterday would not have failed, because of the correct v19.0.0.rc1 tag was found in the new lsst/dax_apdb, and the correct 19.0.0 would have been tagged in the right place. Said that I am quite reluctant to suggest a change in the tooling. If the renaming of the package is done properly, there should be no problems. In case of a missing tag, the workaround is trivial.
            Hide
            tjenness Tim Jenness added a comment -

            I'm not entirely sure I agree that it would have worked fine if v19.0.0rc1 was present on the dax_apdb repo. How would the tagging script have known to look there when it was applying the 19.0.0 tag? It was looking for dax_ppdb. A 19.0.0 tag appeared (in a weird place) on the dax_ppdb backup repo (and I don't understand how the SHA1 was chosen since it was the SHA1 of HEAD of dax_apdb). I admit I haven't read the tagging code though.

            Show
            tjenness Tim Jenness added a comment - I'm not entirely sure I agree that it would have worked fine if v19.0.0rc1 was present on the dax_apdb repo. How would the tagging script have known to look there when it was applying the 19.0.0 tag? It was looking for dax_ppdb. A 19.0.0 tag appeared (in a weird place) on the dax_ppdb backup repo (and I don't understand how the SHA1 was chosen since it was the SHA1 of HEAD of dax_apdb). I admit I haven't read the tagging code though.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            That's thanks to GitHub redirect, which should be working for some time after the renaming.

            When the redirect will not work anymore, we will need to add manually the tag, and (re)run the release.

            However, old repo and new repo need to be in sync, and now they aren't. That the source of the problem.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - That's thanks to GitHub redirect, which should be working for some time after the renaming. When the redirect will not work anymore, we will need to add manually the tag, and (re)run the release. However, old repo and new repo need to be in sync, and now they aren't. That the source of the problem.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            After a short chat with Tim Jenness, we agreed that this issue can be closed since the 19.0.0 tag has been applied manually to the correct repository and this made possible to complete the release process.

            However, 2 updates are suggested to the renaming procedure (DMTN-027):

            • the old package-name should be renamed as legacy_package-name
            • an extra check shall be done at the end of the renaming process, in order to ensure that no new changes have been included in the old repository, including tags, especially if the renaming required more than few days.

            I open another Jira issue for this.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - After a short chat with Tim Jenness , we agreed that this issue can be closed since the 19.0.0 tag has been applied manually to the correct repository and this made possible to complete the release process. However, 2 updates are suggested to the renaming procedure (DMTN-027): the old package-name should be renamed as legacy_package-name an extra check shall be done at the end of the renaming process, in order to ensure that no new changes have been included in the old repository, including tags, especially if the renaming required more than few days. I open another Jira issue for this.

              People

              Assignee:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Reporter:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Watchers:
              Gabriele Comoretto [X] (Inactive), John Swinbank, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.