Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19618

Allow gen3.py to be called with an external butler configuration

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None

      Description

      Currently the butler configuration used for gen 3 testing in ci_hsc is hard coded into the lsst.ci.hsc.gen3 module. This makes it hard to use for oracle testing where we want to override the configuration. Update gen3.py to be a class allowing an override butler and add command line to the gen3.py script to use that functionality.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          I've reworked the gen3 wrapper stuff in ci_hsc so in theory you can specify a different configuration. I also set the config dir so I think Michelle Gower this means that you won't need to specify the top level root yourself (or override the one in datastore).

          Show
          tjenness Tim Jenness added a comment - I've reworked the gen3 wrapper stuff in ci_hsc so in theory you can specify a different configuration. I also set the config dir so I think Michelle Gower this means that you won't need to specify the top level root yourself (or override the one in datastore).
          Hide
          mgower Michelle Gower added a comment -

          I can override butler configuration (e.g., oracle values) using the new command line option.   Has my vote for ready to merge.

          My notes from running in case we ever need to look back:

          • Cannot override the butler root in any config file(s).  gen3.py has hardcoded ${CI_HSC_DIR}/DATA
          • Problems occur when trying to copy DATA from a previous run of ci_hsc (even same version).
          • My override config file for gen3.py needed to use the includeConfigs to point to main butler.yaml file
          Show
          mgower Michelle Gower added a comment - I can override butler configuration (e.g., oracle values) using the new command line option.   Has my vote for ready to merge. My notes from running in case we ever need to look back: Cannot override the butler root in any config file(s).  gen3.py has hardcoded ${CI_HSC_DIR}/DATA Problems occur when trying to copy DATA from a previous run of ci_hsc (even same version). My override config file for gen3.py needed to use the includeConfigs to point to main butler.yaml file
          Hide
          tjenness Tim Jenness added a comment -

          Thanks. Merged.

          Show
          tjenness Tim Jenness added a comment - Thanks. Merged.

            People

            • Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch, Michelle Gower
              Watchers:
              Jim Bosch, Michelle Gower, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel