Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29121

ApPipeTask's --reuse-outputs-from does not work with external preprocessing

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
    • Story Points:
      1
    • Epic Link:
    • Sprint:
      AP S21-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      As described in this Community post, ApPipeTask assumes that any calexps or difference images will always be in an output repository or rerun, which causes the --reuse-outputs-from flag to ignore products created by an external pipeline (in this case, hscPipe 8.4).

      The bug should be fixable by removing the write=True flag from any datasetExists checks.

        Attachments

          Activity

          Hide
          Permalink
          gkovacs Gabor Kovacs [X] (Inactive) added a comment -

          I briefly looked at datasetExists and I cannot fully see why it requires the existence in all output locations should there be multiple ones. But in our case the problem was that the dataset was not present in the (single) output repo, and was not looked at the input repos due to the write=True flag. The fix looks good to me.

          Show
          gkovacs Gabor Kovacs [X] (Inactive) added a comment - I briefly looked at datasetExists and I cannot fully see why it requires the existence in all output locations should there be multiple ones. But in our case the problem was that the dataset was not present in the (single) output repo, and was not looked at the input repos due to the write=True flag. The fix looks good to me.

            People

            Assignee:
            krzys Krzysztof Findeisen
            Reporter:
            krzys Krzysztof Findeisen
            Reviewers:
            Gabor Kovacs [X] (Inactive)
            Watchers:
            Eric Bellm, Gabor Kovacs [X] (Inactive), Ian Sullivan, Krzysztof Findeisen, Meredith Rawls
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.