Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14138

github repos with incorrect team membership for automated tagging

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: stack release
    • Labels:
      None
    • Templates:
    • Story Points:
      26
    • Epic Link:
    • Team:
      SQuaRE

      Description

      The github_tag_versions script, used to apply git tags from eups tags + versiondb manifests, had its error reporting overhauled in DM-14097. This includes better reporting of github repos that are associated with products in an eups tag but do not have the correct github team membership for a git tag to be applied.

      This is from verifying w_2018_15:

      WARNING:codekit:No action for lsst/astrometry_net_data
        has teams: ['Overlords']
        does not belong to any of: ['Data Management', 'DM Externals']                
      WARNING:codekit:No action for lsst/meas_mosaic
        has teams: []
        does not belong to any of: ['Data Management', 'DM Externals']                
      WARNING:codekit:No action for lsst/qserv_distrib
        has teams: ['Overlords', 'Database']
        does not belong to any of: ['Data Management', 'DM Externals']                
      WARNING:codekit:No action for lsst/qserv_testdata
        has teams: ['Overlords', 'Database']
        does not belong to any of: ['Data Management', 'DM Externals']                
      WARNING:codekit:No action for lsst/validation_data_decam
        has teams: ['DM Auxilliaries']
        does not belong to any of: ['Data Management', 'DM Externals']          
      

      This has been a long running problem due to both a lack of reporting (technical) and developer education. I suspect this will some day bite us if we ever try to rebuild past releases from git tags. I'm considering making this scenario fatal on the weekly release pipelines (nightly releases do no receive git tags) but I'm not very enthusiastic, in principle, about designing a process to always play Whac-A-Mole after the fact.

      One possibility would be to incorporate similar checking into stack-os-matric and make it the developers responsibility upfront. A concern I have about that approach is that a check for lsst_disrib is almost 600 github rest API calls (we might be able to cut this down with ETags but would require some investigation). The github API request limit is 5K/hr per user – so we'd probably be OK as its unlikely we'll be able to do 10 builds of lsst_distrib per hour any time soon. However, that feels like being a bad citizen with the gh API (we've already had the travis api get shut off on us due to heavy regular use) and we really should build a local validation service if we want to check every build. Another option would be have an automated and highly visible nag, such as notifications posted to #dm something like once a day.

      Do folks have any comments or suggestions?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Fritz Mueller, Frossie Economou, Gabriele Comoretto, John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel