Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14868

Adapt validation_data_* processing to use HTM catalogs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the validation_data_hsc

      1. [x] Provide HTM reference catalogs for the validation_data_[cfht,decam,hsc]. DM-13701
      2. [x] Updating the processing configs to use these catalogs.
      3. [x] Update the HSC processing to include the transmission files.
      4. [x] Update the curated processed files based on these new reductions.
      5. [x] Update the examples in validate_drp/examples and the equivalent lsst_ci/scripts to use these catalogs.
      6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14857 Switch the default matcher to PessimisticB.

          • Done
          duplicates

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-13376 Replace testdata a.net refcats with Indexed refcats

          • Won't Fix
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. DM-13701 Provide HTM/PS1 refcats for validation_data_(hsc, decam, cfht)

          • Done
          is duplicated by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11648 Switch validate_drp DECam test to use HTM reference catalog

          • Invalid
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. DM-17544 new refcats in validation_data repos are invalid off lsst-dev

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11395 Update testdata_jointcal refcats to new Indexed format

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13298 change to using LSST style index files

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            wmwood-vasey Michael Wood-Vasey created issue -
            wmwood-vasey Michael Wood-Vasey made changes -
            Field Original Value New Value
            Epic Link DM-5510 [ 23343 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Risk Score 0
            Parejkoj John Parejko made changes -
            Link This issue duplicates DM-13376 [ DM-13376 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-11395 [ DM-11395 ]
            swinbank John Swinbank made changes -
            Link This issue is blocked by DM-13701 [ DM-13701 ]
            swinbank John Swinbank made changes -
            Link This issue is duplicated by DM-11648 [ DM-11648 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}.
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_{cfht,decam,hsc}}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Use the processing to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Updating the processing configs to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [ ] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [ ] Updating the processing configs to use these catalogs.
            3. [ ] Consider updating the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            krughoff Simon Krughoff made changes -
            Status To Do [ 10001 ] Reviewed [ 10101 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [ ] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Watchers John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff [ John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff ] Chris Morrison, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff [ Chris Morrison, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Attachment report_performance_comparison.rst [ 34353 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [ ] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Link This issue blocks DM-14857 [ DM-14857 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [ ] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            		The "new" standard for the DM Stack processing is to use HTM style catalogs intead of astrometry.net formatted catalogs. The processing of the {{validation_data_hsc}}

            1. [x] Provide HTM reference catalogs for the {{validation_data_[cfht,decam,hsc]}}. DM-13701
            2. [x] Updating the processing configs to use these catalogs.
            3. [x] Update the HSC processing to include the transmission files.
            4. [x] Update the curated processed files based on these new reductions.
            5. [x] Update the examples in {{validate_drp/examples}} and the equivalent {{lsst_ci/scripts}} to use these catalogs.
            6. [x] Compare SRD performance with old-style and new-style catalogs to look for and characterize any significant differences.
            wmwood-vasey Michael Wood-Vasey made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Story Points 8
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-17544 [ DM-17544 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-13298 [ DM-13298 ]

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              wmwood-vasey Michael Wood-Vasey
              Watchers:
              Chris Morrison [X] (Inactive), Colin Slater, John Parejko, John Swinbank, Michael Wood-Vasey, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.