Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33279

Add star matching task for input to fgcmcal, unique psf star selection, etc.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      For many, many years we have needed a way to consistently select stars for consistent reserving of psf stars for overlapping observations (and possible cross-band as well). At some point it was thought that the output from fgcmcal would be suitable for this, as it does the global star matching. However, the order of operations has made that difficult to implement in a useful way for psf star selection.

      This ticket is to add a task to match all the (relatively) isolated stars with observations above a configurable s/n ratio and output in a useful format that can be ingested by both the new psf builder (which will be run after calibrate and matching), and by the fgcmcal. At its core this will be a port of the matching done within fgcmcal, but done in a more parallelizable, memory-aware, and butler-compatible way.

      After some discussion with Jim Bosch I have decided that the task will:

      • Take sourceTable_visit catalogs as input. Some quick benchmarking shows that if you have at least ~5 detectors overlapping a region it is faster to read in the full sourceTable_visit (with 100 detectors of data!) than it is to read in 5 individual sourceTable files.
      • At first this task will run per-tract to be compatible with existing repos. The ability to run per-healpix (with different resolutions!) should be easy to add when the middleware supports that.
      • The task will run on all bands for a consistent set of matched stars.
      • Will select stars above a configurable s/n limit in each observation.
      • The output will consist of 3 files per quantum.
      • There will be a list of unique ids, ra/dec positions, and pointers to the index table (location and number of observations).
      • The index table will have lists of indices to the observation table.
      • The observation table will have ra_obs, dec_obs, visit, detector, and several other configurable rows from the source tables, including the sourceId (default will be those rows necessary for input to fgcmcal and jointcal). The fraction of star observations with sufficient s/n at high latitude is ~5% of the total of a catalog, and this is a small subset of rows, so the storage overhead should be minimal but the convenience for the known consumers of these catalogs will be great.

      TBD is performance in low latitude high stellar density regions.

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            I'm going to update the PR after the new conda env propagates.

            Show
            erykoff Eli Rykoff added a comment - I'm going to update the PR after the new conda env propagates.
            Hide
            fred3m Fred Moolekamp added a comment -

            This looks good. Way to go on writing a lot of useful unit tests!

            Show
            fred3m Fred Moolekamp added a comment - This looks good. Way to go on writing a lot of useful unit tests!

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Fred Moolekamp
              Watchers:
              Eli Rykoff, Fred Moolekamp, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.