Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30563

ap_verify failing to find dataset type fakes_deepDiff_warpedExp

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      Kian-Tat Lim reported on Slack that ap_verify Gen3 failed on the weekly with the following error:

      RuntimeError: Error finding datasets of type fakes_deepDiff_warpedExp in collections [ap_verify-output]; it is impossible for any such datasets to be found in any of those collections, most likely because the dataset type is not registered.
       This error may become a successful query that returns no results in the future, because queries with no results are not usually considered an error.
      

       

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            $5 says that adding `doWriteWarpedExp: True` to ap_pipe/ApPipe.yaml under the imageDifference config will fix this.

            Show
            mrawls Meredith Rawls added a comment - $5 says that adding `doWriteWarpedExp: True` to ap_pipe/ApPipe.yaml under the imageDifference config will fix this.
            Hide
            mrawls Meredith Rawls added a comment -

            I believe the problem is solved in DM-29338, which I just put in review with Ian Sullivan.

            Show
            mrawls Meredith Rawls added a comment - I believe the problem is solved in DM-29338 , which I just put in review with Ian Sullivan .
            Hide
            krzys Krzysztof Findeisen added a comment -

            What's the status of this ticket? In particular, do we know what caused it to fail starting June 3?

            Show
            krzys Krzysztof Findeisen added a comment - What's the status of this ticket? In particular, do we know what caused it to fail starting June 3?
            Hide
            mrawls Meredith Rawls added a comment -

            I merged DM-29338 on Friday, so if this problem doesn't happen again, it is fixed. I believe the issue was imageDifference not persisting the warped exposure which diaPipe expects as an input (both for regular sources and fake sources). 

            Show
            mrawls Meredith Rawls added a comment - I merged DM-29338 on Friday, so if this problem doesn't happen again, it is fixed. I believe the issue was imageDifference not persisting the warped exposure which diaPipe expects as an input (both for regular sources and fake sources). 
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            I think I found the (current) problem. The workaround on DM-29338 caused some of the input/output connections, but not others, to stop using fakesType, so when ApVerifyWithFakes.yaml updates fakesType, some of the outputs no longer match other tasks' inputs. This manifests as

            RuntimeError: Error finding datasets of type deepDiff_diaSrc_schema in collections [ap_verify-output]
            

            (where the registered dataset type is fakes_deepDiff_diaSrc_schema).

            I saw on Slack that this is supposed to be a workaround for DM-30210, but I can't find any contracts that use connection names rather than template parameters (and said contracts don't catch the problem because they don't test the actual connection names...). Can you point me to the lines that forced you to use explicit connections in ApPipe.yaml?

            Show
            krzys Krzysztof Findeisen added a comment - - edited I think I found the (current) problem. The workaround on DM-29338 caused some of the input/output connections, but not others, to stop using fakesType , so when ApVerifyWithFakes.yaml updates fakesType , some of the outputs no longer match other tasks' inputs. This manifests as RuntimeError: Error finding datasets of type deepDiff_diaSrc_schema in collections [ap_verify-output] (where the registered dataset type is fakes_deepDiff_diaSrc_schema ). I saw on Slack that this is supposed to be a workaround for DM-30210 , but I can't find any contracts that use connection names rather than template parameters (and said contracts don't catch the problem because they don't test the actual connection names...). Can you point me to the lines that forced you to use explicit connections in ApPipe.yaml ?
            Hide
            mrawls Meredith Rawls added a comment -

            What forced us to use explicit connections was an empty quantum graph (various data types not being found) when we didn't have them. Just setting `coaddName` is insufficient if there is something constructed from `coaddName` that appears as a contract.

            So there is a new fakes-related failure now that I've merged DM-29338? Sorry I missed that.

            Show
            mrawls Meredith Rawls added a comment - What forced us to use explicit connections was an empty quantum graph (various data types not being found) when we didn't have them. Just setting `coaddName` is insufficient if there is something constructed from `coaddName` that appears as a contract. So there is a new fakes-related failure now that I've merged DM-29338 ? Sorry I missed that.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I discussed with Meredith Rawls offline and we agreed that I'd expand the DM-29338 hard-coding to include all the fakesType/coaddName connections, including ones like imageDifference:connections.outputSchema that had been left alone until now. I'll tag with a comment that this is temporary until DM-30210 is fixed.

            Show
            krzys Krzysztof Findeisen added a comment - I discussed with Meredith Rawls offline and we agreed that I'd expand the DM-29338 hard-coding to include all the fakesType / coaddName connections, including ones like imageDifference:connections.outputSchema that had been left alone until now. I'll tag with a comment that this is temporary until DM-30210 is fixed.
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for tweaking the language: it makes more sense now.

            We should probably have a ticket about removing the `if self.config.coaddName == 'dcr':` in `ImageDifferenceTask:runQuantum`, and/or replacing it with the Connection `coaddName` (or doing that particular configuration in the Config or Connections, if its more appropriate there).

            Show
            Parejkoj John Parejko added a comment - Thanks for tweaking the language: it makes more sense now. We should probably have a ticket about removing the `if self.config.coaddName == 'dcr':` in `ImageDifferenceTask:runQuantum`, and/or replacing it with the Connection `coaddName` (or doing that particular configuration in the Config or Connections, if its more appropriate there).

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              John Parejko
              Watchers:
              Eric Bellm, Ian Sullivan, John Parejko, Kian-Tat Lim, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.