Details
-
Type:
Improvement
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Team:Alert Production
Description
DM-29857 implements scripts for regenerating repositories from scratch, as well as individual scripts for each step. However, the scripts for ap_verify_ci_hits2015 and ap_verify_hits2015 are redundant, in that the CI script reprocesses a subset of the HiTS data from scratch. This adds about 10 hours of unnecessary runtime to any update of both repositories.
Add a way for an update of ap_verify_ci_hits2015 to reuse the collections created for updating ap_verify_hits2015; this may be as simple as adding an optional dataset filter to the CI ingest scripts' CLI.
Assuming we stop maintaining ap_verify_hits2015 per
RFC-882andDM-36385, there will no longer be any redundancy and we will just want to use the CI dataset's standalone script as-is.