Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-262

Add ref_cats_htm to /datasets

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:

      Description

      I would like to have a location on datasets to collect symlinks to HTM indexed catalogs that can be used by the new multi-reference catalog system: DM-8232. My suggestion is that it will be called /datasets/ref_cats_htm. There are currently only two catalogs that would be linked there. One is /datasets/refcats/gaia_DR1_v1/photo_astro_ref/, the other is /scratch/ctslater/sdss_dr9_refcat/photo_astro_ref/ which I've asked to be moved to /datasets/refcats/sdss_dr9/photo_astro_ref/ in DM-8672.

      I am requesting a new name ref_cats_htm in addition to refcats since we now have two types of catalogs: a.net indexed and HTM indexed. Having a single place for all reference catalogs is probably good. We can then create a collection of just the HTM indexed catalogs in the proposed location.

      I do not expect us to be changing the available catalogs often or that we will ever have more than a handful of these. To be concrete, my proposal is that, once DM-8672 is closed, the structure that looks like the following:

      datasets
      		 
      ├── ref_cats_htm
      		 
      │   ├── gaia -> /datasets/refcats/gaia_DR1_v1/photo_astro_ref/
      		 
      │   └── sdss -> /datasets/refcats/sdss_dr9/photo_astro_ref/

        Attachments

          Issue Links

          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-8708 Implement RFC-262

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-8232 Make ref config name configurable

          • Done

          Technical task - Created by JIRA Agile - do not edit or delete. Issue type for a technical task. DM-8118 /datasets/refcats/gaia_DR1_v1 (source: /gpfs/fs0/home/ctslater/gaia_refcat/*)

          • Done

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-95 Verification datasets: filesystem organization and argument parser features

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-249 Common Dataset Organization and Policy

          • Implemented

            Activity

            Ascending order - Click to sort in descending order
            16 older comments
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            How about this:

            • All reference catalogs live within /datasets/refcats/.
            • Within /datasets/refcats/ there are /datasets/refcats/astrometry_net_data/ and /datasets/refcats/htm/. Real data live in subfolders of either folder with descriptive names. For example, /datasets/refcats/htm/gaia_DR1_v1 and /datasets/refcats/htm/sdss_dr9 (RFC-264).
            • The collections of HTM indexed catalogs that Simon Krughoff needs can live in another folder. This folder will contain many symlinks. It can be /datasets/refcats/htm_best or /datasets/refcats/htm/best/ or any suggestion of a better name? Simon Krughoff what do you think?
            Show
            hchiang2 Hsin-Fang Chiang added a comment - How about this: All reference catalogs live within /datasets/refcats/ . Within /datasets/refcats/ there are /datasets/refcats/astrometry_net_data/ and /datasets/refcats/htm/ . Real data live in subfolders of either folder with descriptive names. For example, /datasets/refcats/htm/gaia_DR1_v1 and /datasets/refcats/htm/sdss_dr9 ( RFC-264 ). The collections of HTM indexed catalogs that Simon Krughoff needs can live in another folder. This folder will contain many symlinks. It can be /datasets/refcats/htm_best or /datasets/refcats/htm/best/ or any suggestion of a better name? Simon Krughoff what do you think?
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            Hsin-Fang Chiang This sounds just fine to me, for the most part. I'm slightly worried that the name htm_best implies some sort of vetting on our part, so maybe we call it htm_baseline or htm_current.

            Jason Alt [X] It sounds like this is pretty much in line with an earlier suggestion from you. Are you happy enough with this solution to call this RFC adopted?

            Show
            krughoff Simon Krughoff added a comment - Hsin-Fang Chiang This sounds just fine to me, for the most part. I'm slightly worried that the name htm_best implies some sort of vetting on our part, so maybe we call it htm_baseline or htm_current . Jason Alt [X] It sounds like this is pretty much in line with an earlier suggestion from you. Are you happy enough with this solution to call this RFC adopted?
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            htm_baseline or htm_current sounds good

            Show
            hchiang2 Hsin-Fang Chiang added a comment - htm_baseline or htm_current sounds good
            Hide
            Permalink
            jalt Jason Alt [X] (Inactive) added a comment -

            This sounds great.

            Show
            jalt Jason Alt [X] (Inactive) added a comment - This sounds great.
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            Marking this as adopted. I've created DM-8708 to do the implementation.

            Show
            krughoff Simon Krughoff added a comment - Marking this as adopted. I've created DM-8708 to do the implementation.

              People

              Assignee:
              krughoff Simon Krughoff
              Reporter:
              krughoff Simon Krughoff
              Watchers:
              Colin Slater, Hsin-Fang Chiang, Jason Alt [X] (Inactive), Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.