# Support Gen 3 ingestion of ap_verify datasets

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Sprint:
AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July)
• Team:
• Urgent?:
No

#### Description

lsst.ap.verify.DatasetIngestTask is responsible for ingesting an ap_verify dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in ingest_dataset.py that lets users choose between the two.

Note that the current dataset format includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with DM-21862, which wants to use HiTS to run single-frame processing.

#### Activity

Hide
Krzysztof Findeisen added a comment - - edited

Following some discussion started by Meredith Rawls on #dm-naming-things, the Gen 3 repository for non-raws will be called preloaded/. Better to name the data products by what they are, not what they aren't.

Show
Krzysztof Findeisen added a comment - - edited Following some discussion started by Meredith Rawls on #dm-naming-things , the Gen 3 repository for non-raws will be called preloaded/ . Better to name the data products by what they are, not what they aren't.
Hide
Krzysztof Findeisen added a comment - - edited

This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows:

• The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on DM-24260, along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest.
• To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is DM-24844.
• This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
Show
Krzysztof Findeisen added a comment - - edited This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows: The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on DM-24260 , along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest. To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is DM-24844 . This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
Hide
Krzysztof Findeisen added a comment -

Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786.

Show
Krzysztof Findeisen added a comment - Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786 .
Hide
Krzysztof Findeisen added a comment -

I've asked Jim Bosch to review this as an expert on RawIngestTask. Changes:

• ap_verify has the main work
• obs_base has a bug fix related to transfer configuration
• ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
Show
Krzysztof Findeisen added a comment - I've asked Jim Bosch to review this as an expert on RawIngestTask. Changes: ap_verify has the main work obs_base has a bug fix related to transfer configuration ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
Hide
Jim Bosch added a comment -

Looks good.  As suggested, I did not really look at the test data repos, but I left a few comments for ap_verify and did not have any for obs_base.

Show
Jim Bosch added a comment - Looks good.  As suggested, I did not really look at the test data repos, but I left a few comments for ap_verify and did not have any for obs_base.

#### People

Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
Jim Bosch
Watchers:
Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness