Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24584

create an ingestRaws butler command

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DB_S20_02
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      There is a script in ci_hsc_gen3 that does the desired steps but doesn't allow for the customization the generic script would need: https://github.com/lsst/ci_hsc_gen3/blob/master/bin.src/ingestRaws.py

      the ingest task is here: https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/ingest.py#L160

      For the command-line script, we'll want to create (and apply overrides to) a RawIngestConfig instance, use that to construct a RawIngestTask instance, and then call RawIngestTask.run.  This will be a common pattern for a lot of commands, and we have a history of -c <override> and -C <config-file> options (in CmdLineTask and pipetask) to apply the config overrides in a consistent way; Jim Bosch can find the code in pipe_base or ctrl_mpexec that we could build on.  

        Attachments

          Issue Links

          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23577 Write ingest images script for gen3

          • Invalid
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24920 improve butler CLI mocking

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24923 eliminate redundant class name & name requirement in butler command test cases

          • Done

            Activity

            Ascending order - Click to sort in descending order
            9 older comments
            Hide
            Permalink
            npease Nate Pease added a comment - - edited

            note, Jim Bosch's Gen3 middleware CLI tools paper contains an entry for ingest: (is this the same as ingest-raws?)

             
            ingest

            butler ingest REPO` `FILES` `[-j NCORES] [--create-repo] [--register INSTR] [--sync] [--output-run COLLECTION] <PEXCFG>

            Ingest raw data into a repository.

            Show
            npease Nate Pease added a comment - - edited note,  Jim Bosch 's Gen3 middleware CLI tools paper contains an entry for ingest : (is this the same as ingest-raws ?)   ingest butler ingest REPO` `FILES` `[-j NCORES] [--create-repo] [--register INSTR] [--sync] [--output-run COLLECTION] <PEXCFG> Ingest raw data into a repository.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Yes, that "ingest" command is documented as "Ingest raw data into a repository" – if we are ever ingesting non-raws then the command might get confusing so "ingest-raws" sounds better for now.

            Show
            tjenness Tim Jenness added a comment - Yes, that "ingest" command is documented as "Ingest raw data into a repository" – if we are ever ingesting non-raws then the command might get confusing so "ingest-raws" sounds better for now.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Nate Pease I'm very confused. You asked me to review these PRs but now I see they've all been merged. What happened?

            Show
            tjenness Tim Jenness added a comment - Nate Pease I'm very confused. You asked me to review these PRs but now I see they've all been merged. What happened?
            Hide
            Permalink
            npease Nate Pease added a comment -

            Tim Jenness I just messaged you in the middleware Slack channel.

            Show
            npease Nate Pease added a comment - Tim Jenness I just messaged you in the middleware Slack channel.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I've had a look at the pull requests and I have a couple of overall comments:

            1. I don't like how test subclasses are now required to specify duplicate information. Subclasses should specify the Instrument class or instrument class name and the name of the instrument itself can come directly from that.
            2. We might now be able to put all the CLI wrappers in a single file since by definition they are small.
            3. I prefer YAML to using eval for checking output.
            4. Maybe DAF_BUTLER_MOCK should be called CLI_BUTLER_MOCK_ENV or something since it's an override environment for doing click mock testing.
            Show
            tjenness Tim Jenness added a comment - I've had a look at the pull requests and I have a couple of overall comments: I don't like how test subclasses are now required to specify duplicate information. Subclasses should specify the Instrument class or instrument class name and the name of the instrument itself can come directly from that. We might now be able to put all the CLI wrappers in a single file since by definition they are small. I prefer YAML to using eval for checking output. Maybe DAF_BUTLER_MOCK should be called CLI_BUTLER_MOCK_ENV or something since it's an override environment for doing click mock testing.

              People

              Assignee:
              npease Nate Pease
              Reporter:
              npease Nate Pease
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Jim Bosch, Nate Pease, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: