Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5002

Make ci_hsc resumable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      Science Pipelines DM-W16-6
    • Team:
      Data Release Production

      Description

      if ci_hsc fails for any reason, (or is cancelled) it must start from the beginning of processing again. This is because of the use of functools.partial to generate dynamic function. These differ enough in their byte code that scons thinks each build has a new function definition passed to the env.command function. Using lambda would suffer from the same problem. This ticket should change how the function signature is calculated such that scons can be resumed.

      This work does not prevent this from being used as a ci tool, as the .scons directory can be deleted which will force the whole SConstruct file to run again.

        Attachments

          Activity

          Hide
          nlust Nate Lust added a comment -

          An example of scons skipping work which is already done on visit 903334 ccd 100:

          # SCONSFLAGS="-j 1" `which python` `which scons` --debug=explain sfm-903334-100
          scons: Reading SConscript files ...
          Mkdir(".scons")
          scons: done reading SConscript files.
          scons: Building targets ...
          scons: `sfm-903334-100' is up to date.
          scons: done building targets.
          

          Show
          nlust Nate Lust added a comment - An example of scons skipping work which is already done on visit 903334 ccd 100: # SCONSFLAGS="-j 1" `which python` `which scons` --debug=explain sfm-903334-100 scons: Reading SConscript files ... Mkdir( ".scons" ) scons: done reading SConscript files. scons: Building targets ... scons: `sfm-903334-100' is up to date . scons: done building targets.
          Hide
          swinbank John Swinbank added a comment -

          Logic looks fine. A few comments on the PR requesting some minor reorganization. Otherwise, good to merge.

          Show
          swinbank John Swinbank added a comment - Logic looks fine. A few comments on the PR requesting some minor reorganization. Otherwise, good to merge.

            People

            • Assignee:
              nlust Nate Lust
              Reporter:
              nlust Nate Lust
              Reviewers:
              John Swinbank
              Watchers:
              John Swinbank, Nate Lust
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel