Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24330

add ability to run an obs_base command via the butler command

    Details

    • Story Points:
      8
    • Sprint:
      DB_S20_02
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      Make the butler command in daf_butler able to call the register command in obs_base (at python/lsst/obs/base/instrument.py).

      To do this, when obs_base is setup, it should add a colon separated list of CLI python code modules or functions. For example set it to lsst.obs.base.cli.

      Then in butler command itself do something like:

      for module in os.environ["DAF_BUTLER_PLUGINS"].split(":"):
       blah = doImport(cls) 

      and maybe have the imported class import the subcommands to click somehow
      ...it may help to be explicit and have the env var contain the actual command function (so lsst.obs.base.cli.register or somesuch). 

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I was thinking that it would not be a path but would be a colon separated list of CLI python code modules or functions. So for example set it to lsst.obs.base.cli (since we would have the code in the python path).

            Then in butler command itself you do something like:

            for module in os.environ["DAF_BUTLER_PLUGINS"].split(":"):
                blah = doImport(cls)
            

            and maybe have the imported class import the subcommands to click somehow (I don't know what the equivalent of from x import * would be or how importing code registers with click.

            Maybe it helps to be explicit and have the env var contain the actual command function (so lsst.obs.base.cli.register or somesuch).

            Show
            tjenness Tim Jenness added a comment - I was thinking that it would not be a path but would be a colon separated list of CLI python code modules or functions. So for example set it to lsst.obs.base.cli (since we would have the code in the python path). Then in butler command itself you do something like: for module in os.environ[ "DAF_BUTLER_PLUGINS" ].split( ":" ): blah = doImport( cls ) and maybe have the imported class import the subcommands to click somehow (I don't know what the equivalent of from x import * would be or how importing code registers with click. Maybe it helps to be explicit and have the env var contain the actual command function (so lsst.obs.base.cli.register or somesuch).
            Hide
            npease Nate Pease added a comment -

            Tim Jenness thanks for the clarification, I updated the description.

            Show
            npease Nate Pease added a comment - Tim Jenness thanks for the clarification, I updated the description.
            Show
            npease Nate Pease added a comment - https://github.com/lsst/ci_hsc_gen3/pull/16 https://github.com/lsst/daf_butler/pull/258/ https://github.com/lsst/obs_base/pull/226  
            Hide
            npease Nate Pease added a comment -

            Tim, can you nominate anyone else you think should look at this?

            Show
            npease Nate Pease added a comment - Tim, can you nominate anyone else you think should look at this?
            Hide
            npease Nate Pease added a comment -

            Tim Jenness is this good for a CLO post related to these changes?

            Topic: 

            Butler and obs_base command line scripts have changed to a single command with subcommands

            Tags:

            hsc, butler, obspackages

            Body:

            The following butler scripts have been replaced with a top-level butler command, with git-style subcommands:

            `dumpButlerConfig` is now `butler dump-config`
            `makeButlerRepo` is now `butler create`
            `validateButlerConfiguration` is now `butler validate-config`

            Supported options have remained the same, but some of the names have changed with the intention of standardizing on common option names.

            the ci_hsc_gen3 script `registrerInstruments` has been replaced with two commands for registering the instrument and writing the calibrations:
            `butler register`
            `butler write-curated-calibrations`

            A list of all available `butler` subcommands is available by the standard `--help` option; `butler --help`.

            The butler command is defined in `daf_butler`, but subcommand "plugins" are loaded dynamically so make sure you have `setup` the packages you intend to use before calling `butler --help` or attempting to use sub-commands.

            Similarly, help for each of the subcommands is available by calling `butler <subcommand> --help`.

            Show
            npease Nate Pease added a comment - Tim Jenness is this good for a CLO post related to these changes? Topic:  Butler and obs_base command line scripts have changed to a single command with subcommands Tags: hsc, butler, obspackages Body: The following butler scripts have been replaced with a top-level butler command, with git-style subcommands: `dumpButlerConfig` is now `butler dump-config` `makeButlerRepo` is now `butler create` `validateButlerConfiguration` is now `butler validate-config` Supported options have remained the same, but some of the names have changed with the intention of standardizing on common option names. the ci_hsc_gen3 script `registrerInstruments` has been replaced with two commands for registering the instrument and writing the calibrations: `butler register` `butler write-curated-calibrations` A list of all available `butler` subcommands is available by the standard `--help` option; `butler --help`. The butler command is defined in `daf_butler`, but subcommand "plugins" are loaded dynamically so make sure you have `setup` the packages you intend to use before calling `butler --help` or attempting to use sub-commands. Similarly, help for each of the subcommands is available by calling `butler <subcommand> --help`.
            Hide
            tjenness Tim Jenness added a comment -

            That looks okay. Not sure hsc is needed as a tag. Looking at it I do wonder if we want butler config-dump and butler config-validate as the subcommands so that they would be more directly config related and sort together in the help. Jim Bosch do you have an opinion on that?

            Show
            tjenness Tim Jenness added a comment - That looks okay. Not sure hsc is needed as a tag. Looking at it I do wonder if we want butler config-dump and butler config-validate as the subcommands so that they would be more directly config related and sort together in the help. Jim Bosch do you have an opinion on that?
            Hide
            jbosch Jim Bosch added a comment -

            we want butler config-dump and butler config-validate as the subcommands so that they would be more directly config related and sort together in the help

            , and I think that's a pattern we'll want to use for other commands, too. On that note, I'd recommend butler register-instrument, as we'll also have (at least) butler register-skymap.

            Show
            jbosch Jim Bosch added a comment - we want butler config-dump and butler config-validate as the subcommands so that they would be more directly config related and sort together in the help , and I think that's a pattern we'll want to use for other commands, too. On that note, I'd recommend butler register-instrument , as we'll also have (at least) butler register-skymap .
            Hide
            tjenness Tim Jenness added a comment -

            Looks good. Some minor comments on GitHub.

            Show
            tjenness Tim Jenness added a comment - Looks good. Some minor comments on GitHub.

              People

              • Assignee:
                npease Nate Pease
                Reporter:
                npease Nate Pease
                Reviewers:
                Tim Jenness
                Watchers:
                Jim Bosch, Nate Pease, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: