Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7985

Migration of scientific datasets

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None

    Description

      Migration of scientific datasets to butler repository structure per RFC-95.

      Attachments

        Issue Links

          Activity

            jalt Jason Alt [X] (Inactive) added a comment - - edited

            Using this format:

            /datasets/<camera>/repo/ (This is where Butler root locates. It has _mapper, registry.sqlite3, links to actual data, etc)
            /datasets/<camera>/repo/rerun/ (later the processed results go here.)
            /datasets/<camera>/preprocessed/ (For example /datasets/sdss/preprocessed/dr9/, /datasets/decam/preprocessed/cp/, etc)
            /datasets/<camera>/raw/<survey-name>/<actual raw data> (where actual files live, uningested. They don't change with Butler template changes.)

            Final format may change pending discussions.

            /datasets/hsc/raw/commissioning/ (source: /lsst3/HSC/, owner: price )
            /datasets/hsc/raw/newhorizons/ (source: /lsst8/newhorizons, owner: ctslater)
            /datasets/sdss/preprocessed/dr9/ (source: /lsst7/sdss/dr9/, owner: mjuric )
            /datasets/sdss/preprocessed/dr7/ (source: /datasets/stripe82/dr7 owner: daues)
            /datasets/all-sky (source: /lsst/all-sky-ASIVA, owner: Mike Fitzgerald)
            /datasets/all-sky-ASIVA (source: /lsst/all-sky-ASIVA, owner: Jacques Sebag)

            /datasets/refcats/gaia_DR1_v1 (source: /gpfs/fs0/home/ctslater/gaia_refcat/*, owner: ctslater)
            /datasets/refcats/astrometry_net_data/ (source: /lsst7/astrometry_net_data/*, owner: price)

            /dataset/decam/data/ (source: /lsst8/decam, owner: mwv)
            /datasets/lsstSim (source: external)
            /datasets/decam/preprocessed/hits (source: TBD)

            jalt Jason Alt [X] (Inactive) added a comment - - edited Using this format: /datasets/<camera>/repo/ (This is where Butler root locates. It has _mapper, registry.sqlite3, links to actual data, etc) /datasets/<camera>/repo/rerun/ (later the processed results go here.) /datasets/<camera>/preprocessed/ (For example /datasets/sdss/preprocessed/dr9/, /datasets/decam/preprocessed/cp/, etc) /datasets/<camera>/raw/<survey-name>/<actual raw data> (where actual files live, uningested. They don't change with Butler template changes.) Final format may change pending discussions. /datasets/hsc/raw/commissioning/ (source: /lsst3/HSC/, owner: price ) /datasets/hsc/raw/newhorizons/ (source: /lsst8/newhorizons, owner: ctslater) /datasets/sdss/preprocessed/dr9/ (source: /lsst7/sdss/dr9/, owner: mjuric ) /datasets/sdss/preprocessed/dr7/ (source: /datasets/stripe82/dr7 owner: daues) /datasets/all-sky (source: /lsst/all-sky-ASIVA, owner: Mike Fitzgerald) /datasets/all-sky-ASIVA (source: /lsst/all-sky-ASIVA, owner: Jacques Sebag) /datasets/refcats/gaia_DR1_v1 (source: /gpfs/fs0/home/ctslater/gaia_refcat/*, owner: ctslater) /datasets/refcats/astrometry_net_data/ (source: /lsst7/astrometry_net_data/*, owner: price) /dataset/decam/data/ (source: /lsst8/decam, owner: mwv) /datasets/lsstSim (source: external) /datasets/decam/preprocessed/hits (source: TBD)

            Managed science datasets have been migrated.

            plutchak Joel Plutchak (Inactive) added a comment - Managed science datasets have been migrated.

            People

              korrapa2 Harathi Korrapati [X] (Inactive)
              plutchak Joel Plutchak (Inactive)
              Jason Alt [X] (Inactive), Joel Plutchak (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.