# Add CHAINED collection creation to BPS

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Team:
Data Facility
• Urgent?:
No

#### Description

I think ctrl_bps could easily behave more like pipetask in a way that would be convenient for users, by giving it the ability to created CHAINED output collections.

I think the simplest thing to do would be to take both a CHAINED collection name and a RUN collection name, and forward the CHAINED collection name to the init-only pipetask run, passing it as --output (in addition to the current --output-run option passed).

If you want to be a bit more ambitious, if ctrl_bps generated its own timestamp at submit time, not init-only-job time, it could create the RUN collection name from the CHAINED collection name, too. You can use lsst.obs.base.Instrument.makeCollectionTimestamp to make one in the format we use elsewhere (and that's a static method; you don't need an Instrument instance).

#### Activity

Hide
Michelle Gower added a comment -

Also verified that pipetask used the value passed in for --output-run so that the submit directory does indeed match the butler run collection as requested by the yaml.

 $ls -1d submit/u/mgower/pipelines_check/* submit/u/mgower/pipelines_check/20210225T230154Z submit/u/mgower/pipelines_check/20210225T231948Z Show Michelle Gower added a comment - Also verified that pipetask used the value passed in for --output-run so that the submit directory does indeed match the butler run collection as requested by the yaml.$ ls -1d submit/u/mgower/pipelines_check/* submit/u/mgower/pipelines_check/20210225T230154Z submit/u/mgower/pipelines_check/20210225T231948Z
Hide
Meredith Rawls added a comment -

Yes, that chaining looks beautiful, thank you! I figured out how to set it manually after the fact, but this default behavior in the example is far better.

Show
Meredith Rawls added a comment - Yes, that chaining looks beautiful, thank you! I figured out how to set it manually after the fact, but this default behavior in the example is far better.
Hide
Michelle Gower added a comment -

While testing changes prior to pushing, I noticed that 2nd, 3rd, ... runs of pipelines_check were failing in the last Quantum even though the 1st one completed successfully. If one has "--output" in the init pipetask command, one currently one must also have it in the --skip-init-writes pipetask command.

Show
Michelle Gower added a comment - While testing changes prior to pushing, I noticed that 2nd, 3rd, ... runs of pipelines_check were failing in the last Quantum even though the 1st one completed successfully. If one has "--output" in the init pipetask command, one currently one must also have it in the --skip-init-writes pipetask command.
Hide
Michelle Gower added a comment -

Since this is just small changes to pipetask command line calls in example yaml and documentation I thought I would have you review the change because you created the ticket. If you are too swamped, just change the reviewer to Mikolaj Kowalik.

Show
Michelle Gower added a comment - Since this is just small changes to pipetask command line calls in example yaml and documentation I thought I would have you review the change because you created the ticket. If you are too swamped, just change the reviewer to Mikolaj Kowalik .
Hide
Jim Bosch added a comment -

One small doc suggestion, and an unrelated thing that might be worth fixing, on the PR.

Show
Jim Bosch added a comment - One small doc suggestion, and an unrelated thing that might be worth fixing, on the PR.

#### People

Assignee:
Michelle Gower
Reporter:
Jim Bosch
Reviewers:
Jim Bosch
Watchers:
Jim Bosch, Meredith Rawls, Michelle Gower