Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29121

ApPipeTask's --reuse-outputs-from does not work with external preprocessing

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • ap_pipe
    • 1
    • AP S21-4 (March)
    • Alert Production
    • No

    Description

      As described in this Community post, ApPipeTask assumes that any calexps or difference images will always be in an output repository or rerun, which causes the --reuse-outputs-from flag to ignore products created by an external pipeline (in this case, hscPipe 8.4).

      The bug should be fixable by removing the write=True flag from any datasetExists checks.

      Attachments

        Activity

          I briefly looked at datasetExists and I cannot fully see why it requires the existence in all output locations should there be multiple ones. But in our case the problem was that the dataset was not present in the (single) output repo, and was not looked at the input repos due to the write=True flag. The fix looks good to me.

          gkovacs Gabor Kovacs [X] (Inactive) added a comment - I briefly looked at datasetExists and I cannot fully see why it requires the existence in all output locations should there be multiple ones. But in our case the problem was that the dataset was not present in the (single) output repo, and was not looked at the input repos due to the write=True flag. The fix looks good to me.

          People

            krzys Krzysztof Findeisen
            krzys Krzysztof Findeisen
            Gabor Kovacs [X] (Inactive)
            Eric Bellm, Gabor Kovacs [X] (Inactive), Ian Sullivan, Krzysztof Findeisen, Meredith Rawls
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.