Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19491

Enumerate options for creating new HTM Indexed refcats

    Details

    • Templates:
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP S19-5, AP S19-6
    • Team:
      Alert Production

      Description

      After some slack discussion, Simon Krughoff, Colin Slater, and Eli Rykoff have all provided me with some suggestions about how to make refcats.

      There are the DR2 .csv files on lsst-dev here: /project/shared/data/gaia_dr2/gaia_source/csv

      and a Parquet version here: /project/shared/data/gaia_dr2/gaia_source.parquet

      meas_algorithms has IngestIndexReferenceTask, which I've been told is very slow. Eli Rykoff suggested using fgcmOutputProduct._outputStandardStars from fgcmcal, but it does not handle proper motion or parallax or many of the other fields. I should run some simple tests to see just how slow the "standard" code is and make some guesses as to what might be necessary to speed it up. And similarly, compare it with Eli Rykoff's code and decide whether we should just use something like that. As part of this, I should look at reading Parquet vs. CSV.

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Watchers:
                Colin Slater, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Paul Price, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel