Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30843

HealSparsePropertyMapTask should check for input consistency

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      Running HealSparsePropertyMapTask on the w_2021_24 DC2 reprocessing crashes on making y-band maps with a ValueError. This is because of an upstream failure such that the visit summary for 306188 was not created, which is a required input. The correct result is for the HealSparsePropertyMapTask to check for consistency of inputs and raise NoWorkFound.

        Attachments

          Activity

          No builds found.
          erykoff Eli Rykoff created issue -
          Hide
          jbosch Jim Bosch added a comment -

          I'm a bit worried I steered you in the wrong direction by talking about possibilities - if this raises NoWorkFound, it will be impossible to ever run it on partial focal planes, even when that's what the user wants to do. I would at least add a doRequireFullVisits config option or similar to control whether this check happens.

          Show
          jbosch Jim Bosch added a comment - I'm a bit worried I steered you in the wrong direction by talking about possibilities - if this raises NoWorkFound , it will be impossible to ever run it on partial focal planes, even when that's what the user wants to do. I would at least add a doRequireFullVisits config option or similar to control whether this check happens.
          Hide
          erykoff Eli Rykoff added a comment -

          We may need to discuss this in person.  The issue is not that it needs full visits. It's that it needs the visit summary table which has the data which goes into the map! It doesn't care if there's a missing detector if that detector didn't go into the coadd. This is also going to be a problem with the select good seeing visits task.
          The thing is, reading the visit summary table is so much faster/efficient than dealing with all the calexps...

          Show
          erykoff Eli Rykoff added a comment - We may need to discuss this in person.  The issue is not that it needs full visits. It's that it needs the visit summary table which has the data which goes into the map! It doesn't care if there's a missing detector if that detector didn't go into the coadd. This is also going to be a problem with the select good seeing visits task. The thing is, reading the visit summary table is so much faster/efficient than dealing with all the calexps...
          Hide
          erykoff Eli Rykoff added a comment -

          I had a chat with Jim Bosch about this and we came to two conclusions.

          First, the correct exception for the task to raise is RepeatableQuantumError. That will signal to the workflow system that if it tries to run this task again with the same inputs it will get the same result (a missing visit summary).

          Second, the fundamental issue is that upstream if single frame processing (the singleFrame pipeline subset) fails in some way in isr/characterizeImage/calibrate then you cannot just restart with the coaddition subset. There are a bunch more tasks in the singleFrame subset including consolidateSourceTable, consolidateVisitSummary, makeCcdVisitTable, and makeVisitTable that need to be run because while they may not be direct inputs to the coaddition (thus allowing coaddition to continue), they are inputs to map making/QA/reporting/etc. I'm not sure how to enforce this, but I believe this is what can happen to lead to unhappy downstream tasks.

          Show
          erykoff Eli Rykoff added a comment - I had a chat with Jim Bosch about this and we came to two conclusions. First, the correct exception for the task to raise is RepeatableQuantumError . That will signal to the workflow system that if it tries to run this task again with the same inputs it will get the same result (a missing visit summary). Second, the fundamental issue is that upstream if single frame processing (the singleFrame pipeline subset) fails in some way in isr/characterizeImage/calibrate then you cannot just restart with the coaddition subset. There are a bunch more tasks in the singleFrame subset including consolidateSourceTable , consolidateVisitSummary , makeCcdVisitTable , and makeVisitTable that need to be run because while they may not be direct inputs to the coaddition (thus allowing coaddition to continue), they are inputs to map making/QA/reporting/etc. I'm not sure how to enforce this, but I believe this is what can happen to lead to unhappy downstream tasks.
          erykoff Eli Rykoff made changes -
          Field Original Value New Value
          Status To Do [ 10001 ] In Progress [ 3 ]
          erykoff Eli Rykoff made changes -
          Reviewers Nate Lust [ nlust ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          nlust Nate Lust added a comment -

          looks fine

          Show
          nlust Nate Lust added a comment - looks fine
          nlust Nate Lust made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          erykoff Eli Rykoff made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          erykoff Eli Rykoff made changes -
          Epic Link DM-30473 [ 509199 ]

            People

            Assignee:
            erykoff Eli Rykoff
            Reporter:
            erykoff Eli Rykoff
            Reviewers:
            Nate Lust
            Watchers:
            Eli Rykoff, Jim Bosch, Nate Lust
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.