Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28107

Convert/ingest PS1 and Gaia refcats into gen3 repo for DECam use

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      AP S21-1 (December), AP S21-2 (January)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Turns out you need reference catalogs for single frame processing!

      DECam uses PS1 for photometry and Gaia for astrometry, so I need both. I think the process presently involves using `butler convert`, somehow. I'm pretty sure it's not as simple as adding a couple symlinks to my repo.

        Attachments

        1. export_refcats_to_gen3.py
          1 kB
        2. export_templates.py
          0.4 kB
        3. import_refcats.py
          0.5 kB
        4. import_templates.py
          0.6 kB

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            I spent the better part of a day troubleshooting this with John Parejko, Krzysztof Findeisen, and a few others who chimed in.

            We realized the ap_verify_hits2015 dataset has the correct refcat shards, and we "just" need to export them using the gen3 butler and then import them into the repo I plan to use. To this end, we have starter scripts in `/project/mrawls/hits2014-3` called `export_refcats_to_gen3.py` and `import_refcats.py`. The export appears to work, but the import has an error that was discussed on Slack and not yet resolved.

            Once the refcats are in place, I have a draft `APTemplate.yaml` pipeline (in the same directory) which should allow me to do single frame processing and then build coadds for use as templates. I can then proceed with DM-27370.

            Show
            mrawls Meredith Rawls added a comment - I spent the better part of a day troubleshooting this with John Parejko , Krzysztof Findeisen , and a few others who chimed in. We realized the ap_verify_hits2015 dataset has the correct refcat shards, and we "just" need to export them using the gen3 butler and then import them into the repo I plan to use. To this end, we have starter scripts in `/project/mrawls/hits2014-3` called `export_refcats_to_gen3.py` and `import_refcats.py`. The export appears to work, but the import has an error that was discussed on Slack  and not yet resolved. Once the refcats are in place, I have a draft `APTemplate.yaml` pipeline (in the same directory) which should allow me to do single frame processing and then build coadds for use as templates. I can then proceed with DM-27370 .
            Hide
            mrawls Meredith Rawls added a comment -

            I've managed to export refcats from one repo and import them to another.

            The other necessary step (for a pipeline which includes building coadds, like the one I'm running to build templates) is registering a skymap. With all the raws in place, I ran `butler register-skycap . -c name="decam_hits2014_v1"`.

            Both the hits2014-3 and hits2015-3 repos now have panstarrs and gaia refcats in them, as well as registered skymaps, for use with decam processing.

            Show
            mrawls Meredith Rawls added a comment - I've managed to export refcats from one repo and import them to another. The other necessary step (for a pipeline which includes building coadds, like the one I'm running to build templates) is registering a skymap. With all the raws in place, I ran `butler register-skycap . -c name="decam_hits2014_v1"`. Both the hits2014-3 and hits2015-3 repos now have panstarrs and gaia refcats in them, as well as registered skymaps, for use with decam processing.
            Hide
            mrawls Meredith Rawls added a comment -

            The final step I had to do to get my pipeline to run (besides set various configs correctly) was `butler define-visits REPO DECam`.

            To use the refcats in calibration (lsst.pipe.tasks.calibrate.CalibrateTask), these configs are necessary:

            astromRefObjLoader.ref_dataset_name: 'gaia'
            #astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean'  # not yet implemented in gen3, DM-27843
            python: |        
              config.astromRefObjLoader.filterMap = {}        
              config.astromRefObjLoader.filterMap['g'] = 'phot_g_mean'        
              config.astromRefObjLoader.filterMap['r'] = 'phot_g_mean'        
              config.astromRefObjLoader.filterMap['i'] = 'phot_g_mean'        
              config.astromRefObjLoader.filterMap['z'] = 'phot_g_mean'        
              config.astromRefObjLoader.filterMap['y'] = 'phot_g_mean'      
            photoRefObjLoader.ref_dataset_name: 'panstarrs'      
            photoCal.photoCatName: 'panstarrs'      
            connections.astromRefCat: 'gaia'      
            connections.photoRefCat: 'panstarrs' 

            Show
            mrawls Meredith Rawls added a comment - The final step I had to do to get my pipeline to run (besides set various configs correctly) was `butler define-visits REPO DECam`. To use the refcats in calibration (lsst.pipe.tasks.calibrate.CalibrateTask), these configs are necessary: astromRefObjLoader.ref_dataset_name: 'gaia' #astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean' # not yet implemented in gen3, DM- 27843 python: | config.astromRefObjLoader.filterMap = {} config.astromRefObjLoader.filterMap[ 'g' ] = 'phot_g_mean' config.astromRefObjLoader.filterMap[ 'r' ] = 'phot_g_mean' config.astromRefObjLoader.filterMap[ 'i' ] = 'phot_g_mean' config.astromRefObjLoader.filterMap[ 'z' ] = 'phot_g_mean' config.astromRefObjLoader.filterMap[ 'y' ] = 'phot_g_mean' photoRefObjLoader.ref_dataset_name: 'panstarrs' photoCal.photoCatName: 'panstarrs' connections.astromRefCat: 'gaia' connections.photoRefCat: 'panstarrs'
            Hide
            mrawls Meredith Rawls added a comment -

            John Parejko, would you review this? The only code is a functional `APTemplate.yaml` now that I have managed to get refcats for the HiTS fields into my gen3 repo.

            Show
            mrawls Meredith Rawls added a comment - John Parejko , would you review this? The only code is a functional `APTemplate.yaml` now that I have managed to get refcats for the HiTS fields into my gen3 repo.
            Hide
            mrawls Meredith Rawls added a comment -

            One followup regarding skymaps. I mentioned that I ran `butler register-skymap REPO -c name='name'` in both of my gen3 repos. Turns out this resulted in two identical skymaps in all but name, which really confused things when I went to export the deepCoadd templates I made in the 2014 repo to the 2015 repo. With help from Jim, this is how I resolved the situation.

            (1) do export with elements=() in the call to }}{{saveDatasets

            (2) manually edit data_id mentions of origin repo skymap name to be destination repo skymap name in the yaml created during export (do not edit any file paths!)

            (3) skip dimensions skymap, tract, and patch when calling import_.

            I will attach all the export/import scripts I used to this ticket for future reference.

            Show
            mrawls Meredith Rawls added a comment - One followup regarding skymaps. I mentioned that I ran `butler register-skymap REPO -c name='name'` in both of my gen3 repos. Turns out this resulted in two identical skymaps in all but name, which really confused things when I went to export the deepCoadd templates I made in the 2014 repo to the 2015 repo. With help from Jim, this is how I resolved the situation. (1) do  export  with  elements=()  in the call to  }}{{saveDatasets (2) manually edit  data_id  mentions of origin repo skymap name to be destination repo skymap name in the yaml created during export (do not edit any file paths!) (3) skip dimensions skymap, tract, and patch when calling  import_. I will attach all the export/import scripts I used to this ticket for future reference.
            Hide
            Parejkoj John Parejko added a comment -

            A few docs clarifications, otherwise this looks good.

            Show
            Parejkoj John Parejko added a comment - A few docs clarifications, otherwise this looks good.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              John Parejko
              Watchers:
              John Parejko, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.