Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19491

Enumerate options for creating new HTM Indexed refcats

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP S19-5, AP S19-6
    • Team:
      Alert Production

      Description

      After some slack discussion, Simon Krughoff, Colin Slater, and Eli Rykoff have all provided me with some suggestions about how to make refcats.

      There are the DR2 .csv files on lsst-dev here: /project/shared/data/gaia_dr2/gaia_source/csv

      and a Parquet version here: /project/shared/data/gaia_dr2/gaia_source.parquet

      meas_algorithms has IngestIndexReferenceTask, which I've been told is very slow. Eli Rykoff suggested using fgcmOutputProduct._outputStandardStars from fgcmcal, but it does not handle proper motion or parallax or many of the other fields. I should run some simple tests to see just how slow the "standard" code is and make some guesses as to what might be necessary to speed it up. And similarly, compare it with Eli Rykoff's code and decide whether we should just use something like that. As part of this, I should look at reading Parquet vs. CSV.

        Attachments

          Activity

          Parejkoj John Parejko created issue -
          Parejkoj John Parejko made changes -
          Field Original Value New Value
          Epic Link DM-19473 [ 263512 ]
          Parejkoj John Parejko made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          swinbank John Swinbank made changes -
          Sprint AP S19-5 [ 833 ] AP S19-5, AP S19-6 [ 833, 834 ]
          Parejkoj John Parejko made changes -
          Attachment gaia_dr2_config.py [ 38022 ]
          Attachment profileIngestRefcat.py [ 38023 ]
          Parejkoj John Parejko made changes -
          Resolution Done [ 10000 ]
          Status In Progress [ 3 ] Done [ 10002 ]

            People

            • Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Colin Slater, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Paul Price, Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel