# Improve raw data handling in gen2convert

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Team:
Data Release Production

## Description

DM-15189 improves support for raw data by adding new Formatters that provide the adaptation logic provided by std_ hooks in Gen2.  But those Formatters aren't used in gen2convert yet, because gen2convert doesn't see raw data as special.  As a result, loading raw with a Gen3 Butler pointed at a converted repo just gets you a DecoratedImage, not an Exposure.

The simplest fix is probably to have gen2convert just run raw ingest instead of transferring raws via the usual mechanism.  The tricky part of that is to provide a way for gen2convert to identify the RawIngestTask specialization for a particular camera.  We need to solve that problem anyway, but I'd hoped to get some CameraGeom and package organization problems in better shape before tackling it.  An alternative may be to just force the user to provide the task name in the gen2convert configuration for now.

Tentatively assigning this to me since I have a pretty good idea in my head of how to approach this, but Christopher Waters is also a possibility.

## Activity

Hide
Jim Bosch added a comment -

This is basically done, except:

• it's branched off DM-15034, so I'd like to merge that before putting this in review, unless it drags out;
• there are no tests for the actual feature of the ticket (though other changes are covered by existing tests).

I'd like to defer the tests to DM-16803, because I don't want to hack something new into ci_hsc when we'd get a lot more value (even short-term) from a bit of refactoring to make tests like these easy to write.  But I also don't want to block this ticket on that one, so I'm inclined to just live on the edge for a bit, given that one-off testing works and there'll be a lot more one-off testing of this functionality in the near term on DM-16467.

Show
Jim Bosch added a comment - This is basically done, except: it's branched off DM-15034 , so I'd like to merge that before putting this in review, unless it drags out; there are no tests for the actual feature of the ticket (though other changes are covered by existing tests). I'd like to defer the tests to DM-16803 , because I don't want to hack something new into ci_hsc when we'd get a lot more value (even short-term) from a bit of refactoring to make tests like these easy to write.  But I also don't want to block this ticket on that one, so I'm inclined to just live on the edge for a bit, given that one-off testing works and there'll be a lot more one-off testing of this functionality in the near term on DM-16467 .
Hide
Jim Bosch added a comment - - edited

Christopher Waters, sorry for two reviews in rapid succession, but I think you're interested in this one: gen2convert now retrieves the correct formatter from the Instrument class, which has been refactored to provide the same to Gen3 ingest.  I've also updated ci_hsc to write a "camera" dataset that you should be able to take as an input in the ISR PipelineTask to work around Gen3 raws not having a camera attached directly.

In case Jira doesn't spot all of the PRs, changes are in daf_butler, obs_base, obs_subaru, and ci_hsc.

Show
Jim Bosch added a comment - - edited Christopher Waters , sorry for two reviews in rapid succession, but I think you're interested in this one: gen2convert now retrieves the correct formatter from the Instrument class, which has been refactored to provide the same to Gen3 ingest.  I've also updated ci_hsc to write a "camera" dataset that you should be able to take as an input in the ISR PipelineTask to work around Gen3 raws not having a camera attached directly. In case Jira doesn't spot all of the PRs, changes are in daf_butler , obs_base , obs_subaru , and ci_hsc .
Hide
Christopher Waters added a comment -

I've added comments on the pull requests.  My only major concern is a carryover from working on DM-16467, where calibration data needs to be in a collection, but that collection needs to JOIN correctly with all other collections in the repository.  I had to hack this in my testing database, and although not technically part of this ticket, this ticket does move all calibration data into a calib repository.

Show
Christopher Waters added a comment - I've added comments on the pull requests.  My only major concern is a carryover from working on DM-16467 , where calibration data needs to be in a collection, but that collection needs to JOIN correctly with all other collections in the repository.  I had to hack this in my testing database, and although not technically part of this ticket, this ticket does move all calibration data into a calib repository.
Hide
Jim Bosch added a comment -

Merged to master.

Show
Jim Bosch added a comment - Merged to master.

## People

• Assignee:
Jim Bosch
Reporter:
Jim Bosch
Reviewers:
Christopher Waters
Watchers:
Christopher Waters, Jim Bosch