Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12450

Implement RFC-407: improve interface for clobbering vs. reusing outputs

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP F17-5, DRP F17-6
    • Team:
      Data Release Production

      Description

      See RFC for description.

      IMO, this has bitten too many DRP team members recently to just put it on the backlog; I think it's really costing us time and money. Even if the RFC needs to stew for a while before merging, I'd like to get at least a partial implementation on a branch while it's fresh in my mind.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12531 Implement RFC-409: only check configurations/schemas/versions in output repos

          • Done
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-407 Improve interface for clobbering vs. reusing intermediate data products

          • Implemented
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12766 coaddDriver with --cores > 1 produces MPI_Abort(MPI_COMM_WORLD, 1)

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            An implementation of the proposal currently up on RFC-407 has been pushed on the branch for this ticket. It has not been fully tested. This ticket will stay "In Progress" until the RFC closes, and it's possible I'll add an attempt to address problem (3) on the RFC as well.

            Show
            jbosch Jim Bosch added a comment - An implementation of the proposal currently up on RFC-407 has been pushed on the branch for this ticket. It has not been fully tested. This ticket will stay "In Progress" until the RFC closes, and it's possible I'll add an attempt to address problem (3) on the RFC as well.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Changes are ready for review. I'd like to split it up as follows:

            Paul Price, while I don't want to burden you with a formal review while traveling, since you're the main user of the clobber/overwrite options, I want to make sure you have a chance to weigh in here as well. I'm most concerned that the new behavior (described on RFC-407) is acceptable to you; you're welcome but not expected to verify that the code here accomplishes that. If you do want to take a look but won't be able to do so immediately, just let me know so I can hold off on the merge until you have a chance.

            Finally, Nate Lust should note that I've added a doDetection config option to multibandDriver that is now required to enable re-running detection. I believe this will only be needed when adding fake sources at the coadd stage, and it should be included in any instructions for doing so.

            I will be writing a community post announcing these changes and describing their usage once we're ready to merge. I think it's premature to do that now.

            Show
            jbosch Jim Bosch added a comment - Changes are ready for review. I'd like to split it up as follows: Nate Pease : daf_persistence Hsin-Fang Chiang : pipe_base, pipe_drivers, pipe_tasks (recommend approaching them in that order) Paul Price , while I don't want to burden you with a formal review while traveling, since you're the main user of the clobber/overwrite options, I want to make sure you have a chance to weigh in here as well. I'm most concerned that the new behavior (described on RFC-407 ) is acceptable to you; you're welcome but not expected to verify that the code here accomplishes that. If you do want to take a look but won't be able to do so immediately, just let me know so I can hold off on the merge until you have a chance. Finally, Nate Lust should note that I've added a doDetection config option to multibandDriver that is now required to enable re-running detection. I believe this will only be needed when adding fake sources at the coadd stage, and it should be included in any instructions for doing so. I will be writing a community post announcing these changes and describing their usage once we're ready to merge. I think it's premature to do that now.
            Hide
            Permalink
            npease Nate Pease added a comment -

            daf_persistence looks fine to me.

            Show
            npease Nate Pease added a comment - daf_persistence looks fine to me.
            Hide
            Permalink
            price Paul Price added a comment -

            Sounds good to me. I'm glad you've found a way to reduce these long-annoying problems.

            Show
            price Paul Price added a comment - Sounds good to me. I'm glad you've found a way to reduce these long-annoying problems.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Looks good to me. Two comments on the PRs.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Looks good to me. Two comments on the PRs.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Merged to master.

            Show
            jbosch Jim Bosch added a comment - Merged to master.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Hsin-Fang Chiang, Nate Pease
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, Nate Pease, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: