Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Story Points:6
-
Epic Link:
-
Sprint:AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July)
-
Team:Alert Production
-
Urgent?:No
Description
lsst.ap.verify.DatasetIngestTask is responsible for ingesting an ap_verify dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in ingest_dataset.py that lets users choose between the two.
Note that the current dataset format includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.
Some overlap with DM-21862, which wants to use HiTS to run single-frame processing.
Attachments
Issue Links
- blocks
-
- Done
-
DM-25806 Support parallel ap_verify ingestion in Gen 3
- Done
- has to be done after
-
- Done
- is blocked by
-
DM-25786 Cannot import sphgeom objects in Gen 3
- Done
-
- Done
-
- Done
- relates to
-
- To Do
-
- Done
-
- Done
-
- Done
-
- Won't Fix
-
- Done
-
- Done
-
- Done
-
- Done
Activity
This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows:
- The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on
DM-24260, along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest. - To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is
DM-24844. - This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
Krzysztof Findeisen
added a comment - - edited This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows:
The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on DM-24260 , along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest.
To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is DM-24844 .
This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786.
Krzysztof Findeisen
added a comment - Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786 . I've asked Jim Bosch to review this as an expert on `RawIngestTask`. Changes:
- ap_verify has the main work
- obs_base has a bug fix related to transfer configuration
- ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
Krzysztof Findeisen
added a comment - I've asked Jim Bosch to review this as an expert on `RawIngestTask`. Changes:
ap_verify has the main work
obs_base has a bug fix related to transfer configuration
ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
Looks good. As suggested, I did not really look at the test data repos, but I left a few comments for ap_verify and did not have any for obs_base.
Following some discussion started by Meredith Rawls on #dm-naming-things, the Gen 3 repository for non-raws will be called preloaded/. Better to name the data products by what they are, not what they aren't.