Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-188

Committing to supported DM stack releases

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      This is a proposal for DM to commit to supporting (perhaps selected) DM stack releases with bugfixes and occasional backports of functionality. The proposal was triggered by a discussion at the Project Science Team meeting last week (incl. Chris Stubbs, Chuck Claver, Zeljko Ivezic and Robert Lupton), but I think the problem's been recognized for a while now. After a little discussion with Kian-Tat Lim and Robert Lupton, I thought that an RFC would be in order.

      The issue seems to be that our releases de-facto become outdated very quickly, as bugfixes and new functionality tend to land on master. This makes our downstream users gravitate towards using master (e.g., the sims team), but at a cost of issues with API breakage unrelated to their work and additional instability. This instability propagates far out, as sims codes are becoming used broadly within the LSST community (especially within DESC, and MAF is being used across all collaborations). It also makes the discoverable docs (which may refer to the APIs as they were at the time of the release) inconsistent with what people are actually using (whatever is on master). Finally, it makes it difficult to simultaneously use the DM release and the most recent sims release.

      The concrete straw-man proposal (just to begin the discussion) is for us (DM) to commit to:

      • Backporting bugfixes to some number of stable releases (likely only the most recent one?)
      • Occasional backports of new functionality that may be required by downstream users
      • Regularly (at least nightly) CI-ing the backports above
      • Making regular (~monthly?) point releases with the backports accumulated above (including binaries, when we get to releasing those)
      • Never breaking backwards compatibility in point releases.

      If we do this, it should enable our downstream clients (e.g., the sims team) always work off of a stable release, rather than a master. That will make it possible to have DM and sims releases setup-ed (in the EUPS (or conda) sense) at the same time. It will also make support and documentation easier – we can point everyone to the docs related to the release (which we then maintain), rather than have downstream people follow our development process to understand what's going on on master. Finally, if we commit to supporting stable versions it will be easier for other subsystems to use it (e.g., the camera team).

      This proposal will obviously require resources (K-T mentioned maintenance engineers), but I'd like to focus the discussion here on what would be a good process to establish here, both from DM's point of view as well as our downstream users (e.g., sims – Lynne Jones or Chris Walter or others, feel free to comment). And given the release of X16 in a few weeks, this discussion is timely!

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I suggest we defer a decision on this until Wil O'Mullane arrives. We did talk a little bit about "users" at the DMLT last week and we agreed that we would investigate using EUPS for binary distribution. Given that production of weeklies is already automated, turning that into a binary weekly should help enormously in terms of helping people keep up to date. There are issues with macOS in terms of SIP and shebang rewriting that need to be addressed. Once the system has been shown to work it should be possible for anyone to publish a binary build on demand.

            What this does not address is the issue of supporting old releases and decisions on when a patch is suitable for back-porting.

            Show
            tjenness Tim Jenness added a comment - I suggest we defer a decision on this until Wil O'Mullane arrives. We did talk a little bit about "users" at the DMLT last week and we agreed that we would investigate using EUPS for binary distribution. Given that production of weeklies is already automated, turning that into a binary weekly should help enormously in terms of helping people keep up to date. There are issues with macOS in terms of SIP and shebang rewriting that need to be addressed. Once the system has been shown to work it should be possible for anyone to publish a binary build on demand. What this does not address is the issue of supporting old releases and decisions on when a patch is suitable for back-porting.
            Hide
            tjenness Tim Jenness added a comment -

            Flagging this to make it more obvious that we need a formal discussion on how to proceed.

            Show
            tjenness Tim Jenness added a comment - Flagging this to make it more obvious that we need a formal discussion on how to proceed.
            Hide
            tjenness Tim Jenness added a comment -

            Wil O'Mullane Are we deferring this discussion to when the release manager turns up? Or are we defining the policy here for the release manager to implement?

            Show
            tjenness Tim Jenness added a comment - Wil O'Mullane Are we deferring this discussion to when the release manager turns up? Or are we defining the policy here for the release manager to implement?
            Hide
            womullan Wil O'Mullane added a comment -

            We must do this for commissioning anyway and should not wait too long to get more formality in place. Yes that mean we must start back porting fixes. The new Release Manger will have to take control of the process, also listing features and fixes which go in next release and making “users” aware of what is coming.

            Show
            womullan Wil O'Mullane added a comment - We must do this for commissioning anyway and should not wait too long to get more formality in place. Yes that mean we must start back porting fixes. The new Release Manger will have to take control of the process, also listing features and fixes which go in next release and making “users” aware of what is coming.
            Hide
            tjenness Tim Jenness added a comment -

            I have added DM-7771 as the triggering ticket for this RFC.

            Show
            tjenness Tim Jenness added a comment - I have added DM-7771 as the triggering ticket for this RFC.

              People

              Assignee:
              womullan Wil O'Mullane
              Reporter:
              mjuric Mario Juric
              Watchers:
              Chris Walter, Donald Petravick, Frossie Economou, Gabriele Comoretto [X] (Inactive), Gregory Dubois-Felsmann, John Parejko, John Swinbank, Jonathan Sick, Lynne Jones, Michael Wood-Vasey, Paul Price, Pim Schellart [X] (Inactive), Tim Jenness, Wil O'Mullane, Xiuqin Wu [X] (Inactive), Zeljko Ivezic
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.