Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14823

Clean up handling of extra data ID information in SuperTask.run



    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      The default implementation of SuperTask.runQuantum currently passes additional output data ID information to run, as this is necessary in at least some contexts in which run needs to be able to group input datasets.  However, the way this is passed is confusing (the kwargs generated do not have names that suggest that they are IDs), and the need for these IDs may be sufficiently rare that most SuperTasks should not be required to accept them.

      One possibility for how to address this would be:

      • runQuantum always passes just a single data ID (the quantum data ID, not the data of either inputs or outputs) to run, as an always-optional dataId keyword argument (i.e. SuperTasks must permit this argument to be None).  That will at least meet the needs of SuperTasks that want to use the data ID for diagnostic or custom-provenance purposes (see also DM-14821).
      • SuperTasks that need to do data ID grouping in run should override runQuantum themselves.
      • To make the above easier / less verbose, we should look for ways to make some of the logic in the default implementation of runQuantum available to subclasses that override that method (e.g. via utility methods that do some of the work).

      I'm open to other ideas as well, and I should note that I have not thought much about how this proposal would change which of our concrete SuperTasks would need to override runQuantum.


          Issue Links


            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-14816 [ DM-14816 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            salnikov Andy Salnikov made changes -
            Epic Link DM-14661 [ 106365 ]
            salnikov Andy Salnikov made changes -
            Sprint BG3_F18_07 [ 759 ]
            salnikov Andy Salnikov made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            salnikov Andy Salnikov made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            salnikov Andy Salnikov made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]


              salnikov Andy Salnikov
              jbosch Jim Bosch
              Jim Bosch
              Andy Salnikov, Jim Bosch
              0 Vote for this issue
              2 Start watching this issue




                  No builds found.