Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28960

Add CHAINED collection creation to BPS

    XMLWordPrintable

    Details

      Description

      I think ctrl_bps could easily behave more like pipetask in a way that would be convenient for users, by giving it the ability to created CHAINED output collections.

      I think the simplest thing to do would be to take both a CHAINED collection name and a RUN collection name, and forward the CHAINED collection name to the init-only pipetask run, passing it as --output (in addition to the current --output-run option passed).

      If you want to be a bit more ambitious, if ctrl_bps generated its own timestamp at submit time, not init-only-job time, it could create the RUN collection name from the CHAINED collection name, too. You can use lsst.obs.base.Instrument.makeCollectionTimestamp to make one in the format we use elsewhere (and that's a static method; you don't need an Instrument instance).

        Attachments

          Activity

          Hide
          mgower Michelle Gower added a comment -

          Also verified that pipetask used the value passed in forĀ --output-run so that the submit directory does indeed match the butler run collection as requested by the yaml.

          $ ls -1d submit/u/mgower/pipelines_check/*
          submit/u/mgower/pipelines_check/20210225T230154Z
          submit/u/mgower/pipelines_check/20210225T231948Z
          

          Show
          mgower Michelle Gower added a comment - Also verified that pipetask used the value passed in forĀ --output-run so that the submit directory does indeed match the butler run collection as requested by the yaml. $ ls -1d submit/u/mgower/pipelines_check/* submit/u/mgower/pipelines_check/20210225T230154Z submit/u/mgower/pipelines_check/20210225T231948Z
          Hide
          mrawls Meredith Rawls added a comment -

          Yes, that chaining looks beautiful, thank you! I figured out how to set it manually after the fact, but this default behavior in the example is far better.

          Show
          mrawls Meredith Rawls added a comment - Yes, that chaining looks beautiful, thank you! I figured out how to set it manually after the fact, but this default behavior in the example is far better.
          Hide
          mgower Michelle Gower added a comment -

          While testing changes prior to pushing, I noticed that 2nd, 3rd, ... runs of pipelines_check were failing in the last Quantum even though the 1st one completed successfully. If one has "--output" in the init pipetask command, one currently one must also have it in the --skip-init-writes pipetask command.

          Show
          mgower Michelle Gower added a comment - While testing changes prior to pushing, I noticed that 2nd, 3rd, ... runs of pipelines_check were failing in the last Quantum even though the 1st one completed successfully. If one has "--output" in the init pipetask command, one currently one must also have it in the --skip-init-writes pipetask command.
          Hide
          mgower Michelle Gower added a comment -

          Since this is just small changes to pipetask command line calls in example yaml and documentation I thought I would have you review the change because you created the ticket. If you are too swamped, just change the reviewer to Mikolaj Kowalik.

          Show
          mgower Michelle Gower added a comment - Since this is just small changes to pipetask command line calls in example yaml and documentation I thought I would have you review the change because you created the ticket. If you are too swamped, just change the reviewer to Mikolaj Kowalik .
          Hide
          jbosch Jim Bosch added a comment -

          One small doc suggestion, and an unrelated thing that might be worth fixing, on the PR.

          Show
          jbosch Jim Bosch added a comment - One small doc suggestion, and an unrelated thing that might be worth fixing, on the PR.

            People

            Assignee:
            mgower Michelle Gower
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            Jim Bosch
            Watchers:
            Jim Bosch, Meredith Rawls, Michelle Gower
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: