Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28646

Add ability to create trimmed registry during quantum graph build

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As described in DMTN-177 pipe_base needs to be updated to enable the ability to create a new subset registry (sqlite) containing everything needed for the pipeline execution. This includes all output datasets and all necessary inputs (including calibrations).

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            Andy,
            For testing I inserting the following into `CmdLineFwk.makeGraph`. It is not a full long term solution, but I wanted to highlight what I did to get things working:
            ```
            @@ -540,6 +541,19 @@ class CmdLineFwk:
            skipExisting=args.skip_existing)
            qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query)

            + # new stuff below
            + newArgs = copy.deepcopy(args)
            + butler = _ButlerFactory.makeReadButler(args)
            + butlerLight = LightWeightButlerBuilder(butler, args.input)
            +
            + def builderShim(config):
            + newArgs.butler_config = config
            + newButler = _ButlerFactory.makeWriteButler(newArgs)
            + return newButler
            +
            + butlerLight.build(qgraph, '/home/nate2/lightweightButlerV1/', run, clobber=True,
            + butlerModifier=builderShim)
            +

            1. count quanta in graph and give a warning if it's empty and return None
              nQuanta = len(qgraph)
              if nQuanta == 0:
              ```

            I don't know if we would need this at some point, but I designed the object to take a butler to be initialized, and then the build function can be used to build multiple butlers from different `QuantumGraphs`

            Show
            nlust Nate Lust added a comment - Andy, For testing I inserting the following into `CmdLineFwk.makeGraph`. It is not a full long term solution, but I wanted to highlight what I did to get things working: ``` @@ -540,6 +541,19 @@ class CmdLineFwk: skipExisting=args.skip_existing) qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query) + # new stuff below + newArgs = copy.deepcopy(args) + butler = _ButlerFactory.makeReadButler(args) + butlerLight = LightWeightButlerBuilder(butler, args.input) + + def builderShim(config): + newArgs.butler_config = config + newButler = _ButlerFactory.makeWriteButler(newArgs) + return newButler + + butlerLight.build(qgraph, '/home/nate2/lightweightButlerV1/', run, clobber=True, + butlerModifier=builderShim) + count quanta in graph and give a warning if it's empty and return None nQuanta = len(qgraph) if nQuanta == 0: ``` I don't know if we would need this at some point, but I designed the object to take a butler to be initialized, and then the build function can be used to build multiple butlers from different `QuantumGraphs`
            Hide
            salnikov Andy Salnikov added a comment -

            Looks good, few comments on PR.

            Show
            salnikov Andy Salnikov added a comment - Looks good, few comments on PR.

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Brian Yanny, Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.