Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3142

Port HSC optimisations for reading astrometry.net catalog

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None

      Description

      Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues:

      While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.

        Attachments

          Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. DM-3154 meas_astrom still using eups in tests

          • Done

            Activity

            Ascending order - Click to sort in descending order
            price Paul Price created issue -
            price Paul Price made changes -
            Field Original Value New Value
            Description Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues:

            * [https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178|HSC-1178: Faster construction of Astrometry.net catalog]
            * [https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178|HSC-1179: Assertion failure in astrometry.net]

            While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.             		Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues:

            * [HSC-1178: Faster construction of Astrometry.net catalog|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]
            * [HSC-1179: Assertion failure in astrometry.net|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]

            While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.
            price Paul Price made changes -
            Description Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues:

            * [HSC-1178: Faster construction of Astrometry.net catalog|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]
            * [HSC-1179: Assertion failure in astrometry.net|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]

            While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.             		Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues:

            * [HSC-1087: Make astrometry faster|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1087]
            * [HSC-1143: Floating point exception in astrometry|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1143]
            * [HSC-1178: Faster construction of Astrometry.net catalog|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]
            * [HSC-1179: Assertion failure in astrometry.net|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178]

            While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.
            swinbank John Swinbank made changes -
            Epic Link DM-1912 [ 15945 ]
            swinbank John Swinbank made changes -
            Sprint Science Pipelines DM-S15-5 [ 162 ]
            Issue Type Story [ 10001 ] Improvement [ 4 ]
            swinbank John Swinbank made changes -
            Labels HSC
            tjenness Tim Jenness made changes -
            Link This issue blocks DM-3154 [ DM-3154 ]
            price Paul Price made changes -
            Reviewers Russell Owen [ rowen ]
            Status To Do [ 10001 ] In Review [ 10004 ]
            rowen Russell Owen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            price Paul Price made changes -
            Status Reviewed [ 10101 ] In Review [ 10004 ]
            rowen Russell Owen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            price Paul Price made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            price Paul Price made changes -
            Story Points 1 3
            swinbank John Swinbank made changes -
            Labels HSC

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Russell Owen
              Watchers:
              Dominique Boutigny, Kian-Tat Lim, Lauren MacArthur, Paul Price, Russell Owen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.