# Add ability to create trimmed registry during quantum graph build

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Team:
Data Release Production
• Urgent?:
No

#### Description

As described in DMTN-177 pipe_base needs to be updated to enable the ability to create a new subset registry (sqlite) containing everything needed for the pipeline execution. This includes all output datasets and all necessary inputs (including calibrations).

#### Activity

Hide
Nate Lust added a comment -

Andy,
For testing I inserting the following into CmdLineFwk.makeGraph. It is not a full long term solution, but I wanted to highlight what I did to get things working:

@@ -540,6 +541,19 @@ class CmdLineFwk:
skipExisting=args.skip_existing)
qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query)

+ # new stuff below
+ newArgs = copy.deepcopy(args)
+ butlerLight = LightWeightButlerBuilder(butler, args.input)
+
+ def builderShim(config):
+ newArgs.butler_config = config
+ newButler = _ButlerFactory.makeWriteButler(newArgs)
+ return newButler
+
+ butlerLight.build(qgraph, '/home/nate2/lightweightButlerV1/', run, clobber=True,
+ butlerModifier=builderShim)
+

1. count quanta in graph and give a warning if it's empty and return None
nQuanta = len(qgraph)
if nQuanta == 0:


I don't know if we would need this at some point, but I designed the object to take a butler to be initialized, and then the build function can be used to build multiple butlers from different QuantumGraphs

Show
Nate Lust added a comment - Andy, For testing I inserting the following into CmdLineFwk.makeGraph. It is not a full long term solution, but I wanted to highlight what I did to get things working:  @@ -540,6 +541,19 @@ class CmdLineFwk: skipExisting=args.skip_existing) qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query) + # new stuff below + newArgs = copy.deepcopy(args) + butler = _ButlerFactory.makeReadButler(args) + butlerLight = LightWeightButlerBuilder(butler, args.input) + + def builderShim(config): + newArgs.butler_config = config + newButler = _ButlerFactory.makeWriteButler(newArgs) + return newButler + + butlerLight.build(qgraph, '/home/nate2/lightweightButlerV1/', run, clobber=True, + butlerModifier=builderShim) + count quanta in graph and give a warning if it's empty and return None nQuanta = len(qgraph) if nQuanta == 0:  I don't know if we would need this at some point, but I designed the object to take a butler to be initialized, and then the build function can be used to build multiple butlers from different QuantumGraphs
Hide
Andy Salnikov added a comment -

Looks good, few comments on PR.

Show
Andy Salnikov added a comment - Looks good, few comments on PR.

#### People

Assignee:
Nate Lust
Reporter:
Tim Jenness
Reviewers:
Andy Salnikov
Watchers:
Andy Salnikov, Brian Yanny, Jim Bosch, Nate Lust, Tim Jenness