Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:8
-
Epic Link:
-
Team:SQuaRE
Description
The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the validation_data_hsc
1. [x] Provide HTM reference catalogs for the validation_data_[cfht,decam,hsc]. DM-13701
2. [x] Updating the processing configs to use these catalogs.
3. [x] Update the HSC processing to include the transmission files.
4. [x] Update the curated processed files based on these new reductions.
5. [x] Update the examples in validate_drp/examples and the equivalent lsst_ci/scripts to use these catalogs.
6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
Attachments
Issue Links
- blocks
-
- Done
- duplicates
-
DM-13376 Replace testdata a.net refcats with Indexed refcats
- Won't Fix
- is blocked by
-
DM-13701 Provide HTM/PS1 refcats for validation_data_(hsc, decam, cfht)
- Done
- is duplicated by
-
- Invalid
- is triggering
-
DM-17544 new refcats in validation_data repos are invalid off lsst-dev
- Done
- relates to
-
- Done
-
- Won't Fix
Activity
Thanks. I'll make sure all the Tickets and Epics get wrapped up when this work is done.
Michael Wood-Vasey
added a comment - Thanks. I'll make sure all the Tickets and Epics get wrapped up when this work is done. Per agreement with Michael Wood-Vasey, Chris Morrison [X] will provide appropriate refcats on DM-13701, covering the first bullet point in this ticket description. Michael will then handle the rest of the transition.
Michael Wood-Vasey
added a comment - - edited validation_data_cfht and obs_cfht have been updated on DM-14868 to run with the new PS1 HTM cats (based on the updated DM-13071 with the ref_cats in the root of the validation_* repo). These look fine, but I'm a little worried that the a.net files are still there. Shouldn't we take those out at the same time?
I was ambivalent. My principle was keeping them for a cycle to make any future comparisons or checks easy to do and then removing them.
But, in practice, it probably doesn't matter.
Michael Wood-Vasey
added a comment - I was ambivalent. My principle was keeping them for a cycle to make any future comparisons or checks easy to do and then removing them.
But, in practice, it probably doesn't matter. validation_data_hsc updated curated files have been pushed. (200 GB).
Check them out on lsst-dev01 in /project/wmwv/validation_data_hsc
Michael Wood-Vasey
added a comment - validation_data_hsc updated curated files have been pushed. (200 GB).
Check them out on lsst-dev01 in /project/wmwv/validation_data_hsc Simon Krughoff Oh, you don't get off that easily. There will be another review request coming once I have the KPM and related performance comparisons 
Michael Wood-Vasey
added a comment - Simon Krughoff Oh, you don't get off that easily. There will be another review request coming once I have the KPM and related performance comparisons What is the status of this work? I'd like to get jointcal and the other testdata to use Indexed refcats, which would build on this.
It's all done except for the updating validate_drp and lsst_ci scripts.
Michael Wood-Vasey
added a comment - It's all done except for the updating validate_drp and lsst_ci scripts. Thanks for the prompting. I can get this done by Friday.
Michael Wood-Vasey
added a comment - Thanks for the prompting. I can get this done by Friday. Any more news? Do you need help with any of it?
Hi Michael Wood-Vasey, any updates on this ticket? I'm currently waiting on this finishing before I can set the pessimistic matcher as the stack default: DM-14857. Thanks.
I'm dreadfully sorry for the delay.
Michael Wood-Vasey
added a comment - I'm dreadfully sorry for the delay. It happens. Do have an idea of a when you'll be able to continue working on the rest of the ticket?
Any more news on this?
Sorry, I should've updated this ticket. I've discussed this with Michael Wood-Vasey, and he reckons it'll be done by the end of this week.
Things are in shape and looking good. I'm running a final Jenkins job now.
Michael Wood-Vasey
added a comment - Things are in shape and looking good. I'm running a final Jenkins job now. Jenkins passed. Merged.
Again, my apologies for the delay. Thank you to everyone for your patience.
Michael Wood-Vasey
added a comment - Jenkins passed. Merged.
Again, my apologies for the delay. Thank you to everyone for your patience. Thank you, Michael Wood-Vasey!
There was already an epic that was to contain stories for each test dataset:
DM-13376.