Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31704

Design and document reference catalog dataset policy

    XMLWordPrintable

    Details

    • Urgent?:
      No

      Description

      The Common Dataset Policy document has a refcat section that is currently gen2 specific: all gen2 refcats live in one directory tree, which is where we put the readme files that describe them. We need to update this document to reflect how we plan to manage gen3 refcats and their associated documentation (currently in various readme files in the gen2 /datasets/ref_cats/ directory tree).

      For the /repo/main gen3 repo on lsst-devl, we can continue to keep refcats that directory, and ingest them with --transfer=direct so that they stay together in the same path. For S3 or other bucket-based storage, we'll have to come up with something other policy.

      That whole document is very gen2-specific, but I thought it served a very useful purpose for managing large ingests; coming up with the equivalent gen3 policies and updating it for gen3 overall would be good, I think.

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Eli Rykoff, Ian Sullivan, Jim Bosch, John Parejko, Kian-Tat Lim, Nate Lust, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.