Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14868

Adapt validation_data_* processing to use HTM catalogs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the validation_data_hsc

      1. [x] Provide HTM reference catalogs for the validation_data_[cfht,decam,hsc]. DM-13701
      2. [x] Updating the processing configs to use these catalogs.
      3. [x] Update the HSC processing to include the transmission files.
      4. [x] Update the curated processed files based on these new reductions.
      5. [x] Update the examples in validate_drp/examples and the equivalent lsst_ci/scripts to use these catalogs.
      6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14857 Switch the default matcher to PessimisticB.

          • Done
          duplicates

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-13376 Replace testdata a.net refcats with Indexed refcats

          • Won't Fix
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. DM-13701 Provide HTM/PS1 refcats for validation_data_(hsc, decam, cfht)

          • Done
          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11648 Switch validate_drp DECam test to use HTM reference catalog

          • Invalid
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. DM-17544 new refcats in validation_data repos are invalid off lsst-dev

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11395 Update testdata_jointcal refcats to new Indexed format

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13298 change to using LSST style index files

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            wmwood-vasey Michael Wood-Vasey created issue -
            wmwood-vasey Michael Wood-Vasey made changes -
            Field Original Value New Value
            Epic Link DM-5510 [ 23343 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Risk Score 0
            Parejkoj John Parejko made changes -
            Link This issue duplicates DM-13376 [ DM-13376 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-11395 [ DM-11395 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            There was already an epic that was to contain stories for each test dataset: DM-13376.

            Show
            Parejkoj John Parejko added a comment - There was already an epic that was to contain stories for each test dataset: DM-13376 .
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            Thanks. I'll make sure all the Tickets and Epics get wrapped up when this work is done.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Thanks. I'll make sure all the Tickets and Epics get wrapped up when this work is done.
            Hide
            Permalink
            swinbank John Swinbank added a comment -

            Per agreement with Michael Wood-Vasey, Chris Morrison [X] will provide appropriate refcats on DM-13701, covering the first bullet point in this ticket description. Michael will then handle the rest of the transition.

            Show
            swinbank John Swinbank added a comment - Per agreement with Michael Wood-Vasey , Chris Morrison [X] will provide appropriate refcats on DM-13701 , covering the first bullet point in this ticket description. Michael will then handle the rest of the transition.
            swinbank John Swinbank made changes -
            Link This issue is blocked by DM-13701 [ DM-13701 ]
            swinbank John Swinbank made changes -
            Link This issue is duplicated by DM-11648 [ DM-11648 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}.
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Updating the processing configs to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment - - edited

            validation_data_cfht and obs_cfht have been updated on DM-14868 to run with the new PS1 HTM cats (based on the updated DM-13071 with the ref_cats in the root of the validation_* repo).

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited validation_data_cfht and obs_cfht have been updated on DM-14868 to run with the new PS1 HTM cats (based on the updated DM-13071 with the ref_cats in the root of the validation_* repo).
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Updating the processing configs to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            These look fine, but I'm a little worried that the a.net files are still there. Shouldn't we take those out at the same time?

            Show
            krughoff Simon Krughoff added a comment - These look fine, but I'm a little worried that the a.net files are still there. Shouldn't we take those out at the same time?
            krughoff Simon Krughoff made changes -
            Status To Do [ 10001 ] Reviewed [ 10101 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            I was ambivalent. My principle was keeping them for a cycle to make any future comparisons or checks easy to do and then removing them.

            But, in practice, it probably doesn't matter.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - I was ambivalent. My principle was keeping them for a cycle to make any future comparisons or checks easy to do and then removing them. But, in practice, it probably doesn't matter.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            validation_data_hsc updated curated files have been pushed. (200 GB).

            Check them out on lsst-dev01 in /project/wmwv/validation_data_hsc

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - validation_data_hsc updated curated files have been pushed. (200 GB). Check them out on lsst-dev01 in /project/wmwv/validation_data_hsc
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            Simon Krughoff Oh, you don't get off that easily. There will be another review request coming once I have the KPM and related performance comparisons

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Simon Krughoff Oh, you don't get off that easily. There will be another review request coming once I have the KPM and related performance comparisons
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Watchers John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff [ John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff ] Chris Morrison, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff [ Chris Morrison, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Attachment report_performance_comparison.rst [ 34353 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            What is the status of this work? I'd like to get jointcal and the other testdata to use Indexed refcats, which would build on this.

            Show
            Parejkoj John Parejko added a comment - What is the status of this work? I'd like to get jointcal and the other testdata to use Indexed refcats, which would build on this.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            It's all done except for the updating validate_drp and lsst_ci scripts.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - It's all done except for the updating validate_drp and lsst_ci scripts.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            Thanks for the prompting. I can get this done by Friday.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Thanks for the prompting. I can get this done by Friday.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Any more news? Do you need help with any of it?

            Show
            Parejkoj John Parejko added a comment - Any more news? Do you need help with any of it?
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Hi Michael Wood-Vasey, any updates on this ticket? I'm currently waiting on this finishing before I can set the pessimistic matcher as the stack default: DM-14857. Thanks.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Hi Michael Wood-Vasey , any updates on this ticket? I'm currently waiting on this finishing before I can set the pessimistic matcher as the stack default: DM-14857 . Thanks.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            I'm dreadfully sorry for the delay.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - I'm dreadfully sorry for the delay.
            Hide
            Permalink
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            It happens. Do have an idea of a when you'll be able to continue working on the rest of the ticket?

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - It happens. Do have an idea of a when you'll be able to continue working on the rest of the ticket?
            wmwood-vasey Michael Wood-Vasey made changes -
            Link This issue blocks DM-14857 [ DM-14857 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Any more news on this?

            Show
            Parejkoj John Parejko added a comment - Any more news on this?
            Hide
            Permalink
            swinbank John Swinbank added a comment -

            Sorry, I should've updated this ticket. I've discussed this with Michael Wood-Vasey, and he reckons it'll be done by the end of this week.

            Show
            swinbank John Swinbank added a comment - Sorry, I should've updated this ticket. I've discussed this with Michael Wood-Vasey , and he reckons it'll be done by the end of this week.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            Things are in shape and looking good. I'm running a final Jenkins job now.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Things are in shape and looking good. I'm running a final Jenkins job now.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [x] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            Hide
            Permalink
            wmwood-vasey Michael Wood-Vasey added a comment -

            Jenkins passed. Merged.

            Again, my apologies for the delay. Thank you to everyone for your patience.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Jenkins passed. Merged. Again, my apologies for the delay. Thank you to everyone for your patience.
            wmwood-vasey Michael Wood-Vasey made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Story Points 8
            Hide
            Permalink
            swinbank John Swinbank added a comment -

            Thank you, Michael Wood-Vasey!

            Show
            swinbank John Swinbank added a comment - Thank you, Michael Wood-Vasey !
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-17544 [ DM-17544 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-13298 [ DM-13298 ]

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              wmwood-vasey Michael Wood-Vasey
              Watchers:
              Chris Morrison [X] (Inactive), Colin Slater, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.