Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35579

Remove pipeline redirects in pipe_tasks and obs_ packages

    XMLWordPrintable

Details

    Description

      When standing up drp_pipe for RFC-775, I (JFB) decided to leave redirecting pipeline stubs in pipe_tasks and obs_subaru, in the hopes of providing a more gentle transition. But we didn't (and still don't) have a way to mark pipelines as deprecated, and as a result a lot of users (including important ones, like PanDA developers and NAOJ production operators) found and used those stubs instead of better-suited pipelines in drp_pipe.

      And since those stubs weren't (at least for the most part) being tested, anywhere, they were easy to break, and were recently broken on DM-35551, causing one of two issues that torpedoed the w_2022_29 build. I'm not quite sure what happened, but I think it's past time to just remove these and force all code depending on them to be updated, even if that's disruptive; at present there's a good chance any such code is unwittingly using suboptimal configurations already.

      Attachments

        Issue Links

          Activity

            lskelvin Lee Kelvin added a comment - - edited

            Incorrect use of an obs_* pipeline also came up recently in a discussion with quentin.leboulch on Slack. In this case, an old-style pipeline YAML used in the obs_lsst package was importing a file in an ingredients directory which no longer exists (see RFC-927 for more details).

            PS - I notice that a reference to the old-style ingredients directory is also still made in the ctrl_mpexec docs - I will also fix that on this ticket.

            lskelvin Lee Kelvin added a comment - - edited Incorrect use of an obs_* pipeline also came up recently in a discussion with quentin.leboulch on Slack . In this case, an old-style pipeline YAML used in the obs_lsst package was importing a file in an ingredients directory which no longer exists (see RFC-927 for more details). PS - I notice that a reference to the old-style ingredients directory is also still made in the ctrl_mpexec docs - I will also fix that on this ticket.
            lskelvin Lee Kelvin added a comment - - edited obs_cfht PR: https://github.com/lsst/obs_cfht/pull/110 obs_decam PR: https://github.com/lsst/obs_decam/pull/237 obs_lsst PR: https://github.com/lsst/obs_lsst/pull/459 obs_subaru PR: https://github.com/lsst/obs_subaru/pull/455 pipe_tasks PR: https://github.com/lsst/pipe_tasks/pull/809 ctrl_mpexec PR: https://github.com/lsst/ctrl_mpexec/pull/249 Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/39110/pipeline
            jbosch Jim Bosch added a comment -

            Looks good!  I'm impressed you caught the docs change in ctrl_mpexec.

            jbosch Jim Bosch added a comment - Looks good!  I'm impressed you caught the docs change in ctrl_mpexec.
            lskelvin Lee Kelvin added a comment -

            Thanks Jim! Jenkins was green, branches merged and deleted, cheers!

            lskelvin Lee Kelvin added a comment - Thanks Jim! Jenkins was green, branches merged and deleted, cheers!

            People

              lskelvin Lee Kelvin
              jbosch Jim Bosch
              Jim Bosch
              Jim Bosch, Lee Kelvin, Quentin Le Boulc'h
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.