create an ingestRaws butler command

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Epic Link:
• Sprint:
DB_S20_02
• Team:
Data Access and Database
• Urgent?:
No

Description

There is a script in ci_hsc_gen3 that does the desired steps but doesn't allow for the customization the generic script would need: https://github.com/lsst/ci_hsc_gen3/blob/master/bin.src/ingestRaws.py

the ingest task is here: https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/ingest.py#L160

For the command-line script, we'll want to create (and apply overrides to) a RawIngestConfig instance, use that to construct a RawIngestTask instance, and then call RawIngestTask.run.  This will be a common pattern for a lot of commands, and we have a history of -c <override> and -C <config-file> options (in CmdLineTask and pipetask) to apply the config overrides in a consistent way; Jim Bosch can find the code in pipe_base or ctrl_mpexec that we could build on.

Activity

Hide
Nate Pease added a comment - - edited

note, Jim Bosch's Gen3 middleware CLI tools paper contains an entry for ingest: (is this the same as ingest-raws?)

ingest

 butler ingest REPO FILES [-j NCORES] [--create-repo] [--register INSTR] [--sync] [--output-run COLLECTION]

Ingest raw data into a repository.

Show
Nate Pease added a comment - - edited note,  Jim Bosch 's Gen3 middleware CLI tools paper contains an entry for ingest : (is this the same as ingest-raws ?)   ingest butler ingest REPO FILES [-j NCORES] [--create-repo] [--register INSTR] [--sync] [--output-run COLLECTION] <PEXCFG> Ingest raw data into a repository.
Hide
Tim Jenness added a comment -

Yes, that "ingest" command is documented as "Ingest raw data into a repository" – if we are ever ingesting non-raws then the command might get confusing so "ingest-raws" sounds better for now.

Show
Tim Jenness added a comment - Yes, that "ingest" command is documented as "Ingest raw data into a repository" – if we are ever ingesting non-raws then the command might get confusing so "ingest-raws" sounds better for now.
Hide
Tim Jenness added a comment -

Nate Pease I'm very confused. You asked me to review these PRs but now I see they've all been merged. What happened?

Show
Tim Jenness added a comment - Nate Pease I'm very confused. You asked me to review these PRs but now I see they've all been merged. What happened?
Hide
Nate Pease added a comment -

Tim Jenness I just messaged you in the middleware Slack channel.

Show
Nate Pease added a comment - Tim Jenness I just messaged you in the middleware Slack channel.
Hide
Tim Jenness added a comment -

I've had a look at the pull requests and I have a couple of overall comments:

1. I don't like how test subclasses are now required to specify duplicate information. Subclasses should specify the Instrument class or instrument class name and the name of the instrument itself can come directly from that.
2. We might now be able to put all the CLI wrappers in a single file since by definition they are small.
3. I prefer YAML to using eval for checking output.
4. Maybe DAF_BUTLER_MOCK should be called CLI_BUTLER_MOCK_ENV or something since it's an override environment for doing click mock testing.
Show
Tim Jenness added a comment - I've had a look at the pull requests and I have a couple of overall comments: I don't like how test subclasses are now required to specify duplicate information. Subclasses should specify the Instrument class or instrument class name and the name of the instrument itself can come directly from that. We might now be able to put all the CLI wrappers in a single file since by definition they are small. I prefer YAML to using eval for checking output. Maybe DAF_BUTLER_MOCK should be called CLI_BUTLER_MOCK_ENV or something since it's an override environment for doing click mock testing.

People

Assignee:
Nate Pease
Reporter:
Nate Pease
Reviewers:
Tim Jenness
Watchers:
Andy Salnikov, Jim Bosch, Nate Pease, Tim Jenness
Votes:
0 Vote for this issue
Watchers:
4 Start watching this issue

Dates

Created:
Updated:
Resolved: