Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-257

Support multiple reference catalogs


    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
    • Location:
      This issue


      It has been known for some time that a limitation of processCcd.py is that the same reference catalog must be used both for astrometric and photometric calibration. The issue was never addressed because the way we were using a.net index files (with an environment variable) precluded having two catalogs. With the availability of the Gaia catalogs this has become an even bigger problem because Gaia is suited for astrometric calibration for any essentially any survey, but may not be suitable for photometric calibration. Dominique Boutigny has helped motivated this work with his CFHT work.

      With the HTM sharded index files, this is no longer a problem as multiple reference catalogs could, in principle, inhabit the same data repository. To make this a reality, several issues need to be addressed. Below are the issues and my proposed solutions.


      1. PhotoCalTask needs to take a reference loader task
      2. CalibrateTask needs to be able to configure which catalogs are used for astrometry and photometry
      3. Repositories need a way to hold multiple catalogs and refer to them via configuration.
      4. We need multiple indexed catalogs

      Proposed solutions – all work done on DM-8232 (in obs_base, meas_algorithms, meas_astrom, and pipe_tasks):

      1. Create a CalibBaseConfig to hold the pieces that are common to calibration tasks: e.g. the loadAndMatch method, constructor, and configuration for the matcher. This allows both AstrometryTask and PhotoCalTask to be constructed in the same way and to have access to the necessary methods.
      2. This just means duplicating the refObjLoader config items and naming them so that it's obvious which one should be use from astrometric calibration and which for photometric calibration.
      3. Create an algorithm registry (IndexerRegistry) so the indexing algorithm and associated configuration can be stored in a persisted config file and reconstituted by the reference object loader. Also, add the necessary templates in obs_base to allow persistence and retrieval of the reference catalogs and their configs.
      4. Build Gaia and SDSS HTM indexed files.

      How to set up a repository with the proposed solution:
      Colin Slater made HTM indexed reference files for SDSS and GAIA in



      An example data repository for CFHT might look like the following.

      ├── _mapper
      ├── raw
      │   ├── 05AL01
      │   ├── 07AL04
      │   └── 08AL04
      ├── ref_cats
      │   ├── gaia -> /datasets/refcats/gaia_DR1_v1/photo_astro_ref/
      │   └── sdss -> /scratch/ctslater/sdss_dr9_refcat/photo_astro_ref/
      └── registry.sqlite3

      The name of the directory that contains the indexed shards is the reference dataset name ('gaia' and 'sdss' in this case). The default name is "cal_ref_cat". In other words, the template in obs_file is template: ref_cats/%(name)s/%(pixel_id)s.fits. Note that obs packages will need to update their filter maps in the config overrides to use the new names.

      An example repository and the ticket git repositories are available in /lsst8/krughoff/8232_example. To run the example, do something like the following:

      $> source source /ssd/lsstsw/stack/loadLSST.bash
      $> setup lsst_distrib -t w_2016_47
      $> cd /lsst8/krughoff/8232_example
      $> setup -k -r meas_algorithms
      $> setup -k -r meas_astrom
      $> setup -k -r obs_base
      $> setup -k -r pipe_tasks
      $> setup -k -r obs_cfht # for the filter maps
      $> processCcd.py data/ --output ${USER}_example --id visit=793169 ccd=1 --config calibrate.photoRefObjLoader.ref_dataset_name='sdss' calibrate.astromRefObjLoader.ref_dataset_name='gaia' charImage.installSimplePsf.fwhm=1.


          Issue Links



              • Assignee:
                krughoff Simon Krughoff
                krughoff Simon Krughoff
                Colin Slater, Dominique Boutigny, Hsin-Fang Chiang, Jim Bosch, John Parejko, Kian-Tat Lim, Nate Pease, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                9 Start watching this issue


                • Created:
                  Planned End:

                  Summary Panel