Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11566

automatic purging of daily eups distrib tags

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Short lived tags (daily, etc.) should be automatically cleaned up once they have "expired". This should include eupspkg and tarball repos.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            A new utility named tag-monger has been written which scans an s3 bucket for daily tag like objects and parses the filename into a date that can be compared for expiration. Tags are retired by moving them into a sub-directory named old_tags. This is designed to work for multiple EUPS_PKGROOTS hosted in a single s3 bucket without each repo being explicitly declared – the entire bucket is scanned for objects that look like daily tag files. A new iam account + policy is setup as part of the deploy-eups-pkgroot deployment to restrict copy/delete operations for this utility to only eups distrib tag files, and these credentials have been added to jenkins. A docker image is being published to lsstsqre/tag-monger and jenkins jobs have been setup to automatically rebuild the image, run the image, and to trigger a cleanup run once per day.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited A new utility named tag-monger has been written which scans an s3 bucket for daily tag like objects and parses the filename into a date that can be compared for expiration. Tags are retired by moving them into a sub-directory named old_tags . This is designed to work for multiple EUPS_PKGROOTS hosted in a single s3 bucket without each repo being explicitly declared – the entire bucket is scanned for objects that look like daily tag files. A new iam account + policy is setup as part of the deploy-eups-pkgroot deployment to restrict copy/delete operations for this utility to only eups distrib tag files, and these credentials have been added to jenkins. A docker image is being published to lsstsqre/tag-monger and jenkins jobs have been setup to automatically rebuild the image, run the image, and to trigger a cleanup run once per day.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The output of tag-monger claims to be retiring objects but they are still visible in the bucket. Either something is wrong with the program or the s3 bucket is taking days to remove objects.

            Show
            jhoblitt Joshua Hoblitt added a comment - The output of tag-monger claims to be retiring objects but they are still visible in the bucket. Either something is wrong with the program or the s3 bucket is taking days to remove objects.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            It looks like the early 2018 d tags were gone on Friday and have reappeared on Saturday. I suspect this isn't related to the program or s3 but rather the weekly/nightly release is re-pushing old tags.

            Fri: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/3/pipeline
            Sat: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/4/pipeline

            These are the tags that reappeared:

            stack/src/tags/d_2018_01_31.list
            stack/src/tags/d_2018_02_01.list
            stack/src/tags/d_2018_02_02.list
            stack/src/tags/d_2018_02_04.list
            stack/src/tags/d_2018_02_05.list
            stack/src/tags/d_2018_02_06.list
            stack/src/tags/d_2018_02_07.list
            stack/src/tags/d_2018_02_08.list
            stack/src/tags/d_2018_02_09.list
            stack/src/tags/d_2018_02_11.list
            stack/src/tags/d_2018_02_12.list
            stack/src/tags/d_2018_02_13.list
            stack/src/tags/d_2018_02_14.list
            stack/src/tags/d_2018_02_15.list
            stack/src/tags/d_2018_02_16.list
            stack/src/tags/d_2018_02_18.list
            stack/src/tags/d_2018_02_19.list
            stack/src/tags/d_2018_02_20.list
            stack/src/tags/d_2018_02_21.list
            stack/src/tags/d_2018_02_22.list
            stack/src/tags/d_2018_02_23.list
            stack/src/tags/d_2018_02_25.list
            stack/src/tags/d_2018_02_26.list
            stack/src/tags/d_2018_02_27.list
            stack/src/tags/d_2018_02_28.list
            stack/src/tags/d_2018_03_01.list
            stack/src/tags/d_2018_03_02.list
            stack/src/tags/d_2018_03_04.list
            stack/src/tags/d_2018_03_05.list
            stack/src/tags/d_2018_03_06.list
            stack/src/tags/d_2018_03_07.list
            stack/src/tags/d_2018_03_08.list
            stack/src/tags/d_2018_03_09.list
            stack/src/tags/d_2018_03_11.list
            stack/src/tags/d_2018_03_12.list
            stack/src/tags/d_2018_03_13.list
            stack/src/tags/d_2018_03_14.list
            stack/src/tags/d_2018_03_15.list
            stack/src/tags/d_2018_03_16.list
            stack/src/tags/d_2018_03_18.list
            stack/src/tags/d_2018_03_19.list
            stack/src/tags/d_2018_03_20.list
            stack/src/tags/d_2018_03_21.list
            stack/src/tags/d_2018_03_22.list
            stack/src/tags/d_2018_03_23.list
            stack/src/tags/d_2018_03_25.list
            stack/src/tags/d_2018_03_26.list
            stack/src/tags/d_2018_03_27.list
            stack/src/tags/d_2018_03_28.list
            stack/src/tags/d_2018_03_29.list
            stack/src/tags/d_2018_03_30.list
            

            Show
            jhoblitt Joshua Hoblitt added a comment - It looks like the early 2018 d tags were gone on Friday and have reappeared on Saturday. I suspect this isn't related to the program or s3 but rather the weekly/nightly release is re-pushing old tags. Fri: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/3/pipeline Sat: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/4/pipeline These are the tags that reappeared: stack/src/tags/d_2018_01_31.list stack/src/tags/d_2018_02_01.list stack/src/tags/d_2018_02_02.list stack/src/tags/d_2018_02_04.list stack/src/tags/d_2018_02_05.list stack/src/tags/d_2018_02_06.list stack/src/tags/d_2018_02_07.list stack/src/tags/d_2018_02_08.list stack/src/tags/d_2018_02_09.list stack/src/tags/d_2018_02_11.list stack/src/tags/d_2018_02_12.list stack/src/tags/d_2018_02_13.list stack/src/tags/d_2018_02_14.list stack/src/tags/d_2018_02_15.list stack/src/tags/d_2018_02_16.list stack/src/tags/d_2018_02_18.list stack/src/tags/d_2018_02_19.list stack/src/tags/d_2018_02_20.list stack/src/tags/d_2018_02_21.list stack/src/tags/d_2018_02_22.list stack/src/tags/d_2018_02_23.list stack/src/tags/d_2018_02_25.list stack/src/tags/d_2018_02_26.list stack/src/tags/d_2018_02_27.list stack/src/tags/d_2018_02_28.list stack/src/tags/d_2018_03_01.list stack/src/tags/d_2018_03_02.list stack/src/tags/d_2018_03_04.list stack/src/tags/d_2018_03_05.list stack/src/tags/d_2018_03_06.list stack/src/tags/d_2018_03_07.list stack/src/tags/d_2018_03_08.list stack/src/tags/d_2018_03_09.list stack/src/tags/d_2018_03_11.list stack/src/tags/d_2018_03_12.list stack/src/tags/d_2018_03_13.list stack/src/tags/d_2018_03_14.list stack/src/tags/d_2018_03_15.list stack/src/tags/d_2018_03_16.list stack/src/tags/d_2018_03_18.list stack/src/tags/d_2018_03_19.list stack/src/tags/d_2018_03_20.list stack/src/tags/d_2018_03_21.list stack/src/tags/d_2018_03_22.list stack/src/tags/d_2018_03_23.list stack/src/tags/d_2018_03_25.list stack/src/tags/d_2018_03_26.list stack/src/tags/d_2018_03_27.list stack/src/tags/d_2018_03_28.list stack/src/tags/d_2018_03_29.list stack/src/tags/d_2018_03_30.list
            Show
            jhoblitt Joshua Hoblitt added a comment - Re-running tag-monger twice confirms that the tag objects are properly being "renamed". https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/6/pipeline https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Ftag-monger/detail/tag-monger/7/pipeline
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Changes have been made to remove local eups distrib tags from release/run-rebuild and release/tarball to be prevent left over tags from a prior build ever being published.

            Show
            jhoblitt Joshua Hoblitt added a comment - Changes have been made to remove local eups distrib tags from release/run-rebuild and release/tarball to be prevent left over tags from a prior build ever being published.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: