Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30624

Document gen2 to gen3 refcat ingestion

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

      Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:

      butler convert $REPO --gen2root $GEN2REPO -C convert_refcats.py

      the script fails with:

      RuntimeError: No exposures given.

      (full output log attached to this ticket).

      This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms` use existing butler tools to provide updated instructions in the existing pipelines.lsst.io reference catalog documentation.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29543 Gen3 refcat converter

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29857 Create pure Gen 3 dataset management scripts for ap_verify datasets

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lskelvin Lee Kelvin added a comment - - edited

            Comments below copied from a thread by Jim Bosch on Slack, providing instructions for which bits of the currently existing code need to be extracted and placed in a stand-alone script:

            1. First, we look in the ref_cats directory for things that look like reference catalogs, read their configs, and extract the SkyPixDimensions they use (i.e. HTM levels): https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L147-L168
            2. Then we iterate over the files in those directories and make FileDataset instances from them; those are things you can pass to Butler.ingest: https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L180-L188
            3. And then here's an optional block to set up collection structure like that in /repo at NCSA: https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L209-L226
            Show
            lskelvin Lee Kelvin added a comment - - edited Comments below copied from a thread by Jim Bosch on Slack , providing instructions for which bits of the currently existing code need to be extracted and placed in a stand-alone script: First, we look in the ref_cats directory for things that look like reference catalogs, read their configs, and extract the SkyPixDimensions they use (i.e. HTM levels): https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L147-L168 Then we iterate over the files in those directories and make FileDataset instances from them; those are things you can pass to Butler.ingest : https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L180-L188 And then here's an optional block to set up collection structure like that in /repo at NCSA: https://github.com/lsst/obs_base/blob/6b735c201ac1d701d70df1083f0d4e7b56da318e/python/lsst/obs/base/gen2to3/rootRepoConverter.py#L209-L226
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            I believe I have described the process to do this on this Community post. If that does work for the user, I'll document it here.

            https://community.lsst.org/t/issues-with-ingestcalib-py-using-decam-images/5602/41?u=parejkoj

            The updated docs in DM-29543 will have a note about this.

            Show
            Parejkoj John Parejko added a comment - - edited I believe I have described the process to do this on this Community post. If that does work for the user, I'll document it here. https://community.lsst.org/t/issues-with-ingestcalib-py-using-decam-images/5602/41?u=parejkoj The updated docs in DM-29543 will have a note about this.
            Hide
            Permalink
            lskelvin Lee Kelvin added a comment -

            Thanks John Parejko, I've followed your instructions on community.lsst.io and made a few modifications. I think I've accurately documented all of this on pipelines.lsst.io. Would you have time to review this ticket?

            Show
            lskelvin Lee Kelvin added a comment - Thanks John Parejko , I've followed your instructions on community.lsst.io and made a few modifications. I think I've accurately documented all of this on pipelines.lsst.io. Would you have time to review this ticket?
            Hide
            Permalink
            lskelvin Lee Kelvin added a comment -

            Thanks for the comments on GH John. I've restructured the text as you suggest, pulling it out into its own Section 5. Logically, it might make more sense to put this new section before the 'refcat-ingest' section, however, I didn't want to break the flow from Sections 1-4. I hope this reads well to you too, let me know.

            Show
            lskelvin Lee Kelvin added a comment - Thanks for the comments on GH John. I've restructured the text as you suggest, pulling it out into its own Section 5. Logically, it might make more sense to put this new section before the 'refcat-ingest' section, however, I didn't want to break the flow from Sections 1-4. I hope this reads well to you too, let me know.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thanks for writing this up. I've used this script to generate the .ecsv files for gaia, ps1, and sdss on lsst-devl, as noted here: https://community.lsst.org/t/gaia-dr2-reference-catalog-in-lsst-format/3901/18?u=parejkoj

            Show
            Parejkoj John Parejko added a comment - Thanks for writing this up. I've used this script to generate the .ecsv files for gaia, ps1, and sdss on lsst-devl, as noted here: https://community.lsst.org/t/gaia-dr2-reference-catalog-in-lsst-format/3901/18?u=parejkoj
            Hide
            Permalink
            lskelvin Lee Kelvin added a comment -

            Thanks again. Ticket merged and branch deleted, cheers!

            Show
            lskelvin Lee Kelvin added a comment - Thanks again. Ticket merged and branch deleted, cheers!

              People

              Assignee:
              lskelvin Lee Kelvin
              Reporter:
              lskelvin Lee Kelvin
              Reviewers:
              John Parejko
              Watchers:
              Ian Sullivan, John Parejko, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.