Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18036

Convert stack demo refcat to HTM indexed

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Test Data
    • Labels:
      None

      Description

      lsst_dm_stack_demo uses an astrometry.net reference catalog. In order to remove a.net support from the stack, we need to convert this refcat from a.net format to HTM Indexed format. Simon Krughoff claimed he had some code to do such conversions in the past.

      Converting the existing refcat is probably better than creating a new refcat because any changes to the refcat contents would be reflected in changes in the demo output, and we don't want to have to go and vet those changes again.

        Attachments

          Issue Links

            Activity

            Hide
            krughoff Simon Krughoff added a comment -

            This is done, as far as I'm concerned. Note that this also turned up a bug in the FITS reader.

            Show
            krughoff Simon Krughoff added a comment - This is done, as far as I'm concerned. Note that this also turned up a bug in the FITS reader.
            Hide
            krughoff Simon Krughoff added a comment - - edited

            Please have a look at the two PRs associated with this ticket. Note that there was a bug in the FITS reader that I fixed in meas_algorithms using astropy Table s.

            Show
            krughoff Simon Krughoff added a comment - - edited Please have a look at the two PRs associated with this ticket. Note that there was a bug in the FITS reader that I fixed in meas_algorithms using astropy Table s.
            Hide
            krughoff Simon Krughoff added a comment -

            Two more things:
            1) build for this ticket is here
            2) This is considered a blocker for the v19 release, so if it's not something you can review relatively quickly, we should re-assign it.

            Show
            krughoff Simon Krughoff added a comment - Two more things: 1) build for this ticket is here 2) This is considered a blocker for the v19 release, so if it's not something you can review relatively quickly, we should re-assign it.
            Hide
            krughoff Simon Krughoff added a comment -

            I was asked for some timing.

            >>> def from_fits():
            ...     hdl = fits.open('stars.fits')
            ...     arr = hdl[1].data
            ...
            >>> def from_table():
            ...     table = Table.read('stars.fits')
            ...     arr = table.as_array()
            ...
            >>> timeit(from_table, number=500)
            7.41295113787055
            >>> timeit(from_fits, number=500)
            4.18857777165249
            >>> timeit(from_table, number=500)
            7.639025655109435
            >>> timeit(from_fits, number=500)
            3.9214696479029953
            

            So, going straight to array from FITS is significantly faster, but I don't think it's a deal breaker. The Table version does give us the big benefit of putting the values in native-endian order out of the box. Otherwise we'd have to play games with the arrays read in via fits.open.

            Show
            krughoff Simon Krughoff added a comment - I was asked for some timing. >>> def from_fits(): ... hdl = fits.open('stars.fits') ... arr = hdl[1].data ... >>> def from_table(): ... table = Table.read('stars.fits') ... arr = table.as_array() ... >>> timeit(from_table, number=500) 7.41295113787055 >>> timeit(from_fits, number=500) 4.18857777165249 >>> timeit(from_table, number=500) 7.639025655109435 >>> timeit(from_fits, number=500) 3.9214696479029953 So, going straight to array from FITS is significantly faster, but I don't think it's a deal breaker. The Table version does give us the big benefit of putting the values in native-endian order out of the box. Otherwise we'd have to play games with the arrays read in via fits.open .
            Hide
            erykoff Eli Rykoff added a comment -

            Thanks. This is good to know for further reference.

            Show
            erykoff Eli Rykoff added a comment - Thanks. This is good to know for further reference.

              People

              • Assignee:
                krughoff Simon Krughoff
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Eli Rykoff
                Watchers:
                Eli Rykoff, Gabriele Comoretto, John Parejko, John Swinbank, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel