Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-362

Add a “RC” rerun link in /datasets/hsc/repo/rerun

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      (1) Currently our Common Dataset Organization and Policy requires ticket numbers for public reruns. I would like to modify this policy to also allow project-approved tags. While ticket numbers are nice and remove the need of extra README files, tags can be more meaningful to humans. This involves changes in the policy page. New tags still need RFC. The first use case for such tags is for HSC RC reprocessing; see (2).

      (2) We have recently started regular reprocessing of the HSC RC dataset (see more details on this confluence page). We would like a standard place in the rerun folder /datasets/hsc/repo/rerun/RC/ so it’s easy for anybody to find a particular (bi-)weekly rerun. Practically this RC is linked to the /project/hsc_rc/ space as previously provisioned.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Is "the" HSC RC dataset actually constant? Might it be modified in future, or other RC datasets added in parallel alongside it? Should we include some version information in the tag?

            Show
            swinbank John Swinbank added a comment - Is "the" HSC RC dataset actually constant? Might it be modified in future, or other RC datasets added in parallel alongside it? Should we include some version information in the tag?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for pointing that out, John Swinbank!

            I understand the current RC dataset will need to be expanded at some point (late this year?). But how often will it change?

            If "the RC set" changes frequently but follows a ~fixed rule, this tag can be used for all RC-like reprocessing (3 tracts, <10TB, results going into /project space, etc).

            If we only need a ~one-time change or changes are rare, how about naming this "RC1", so later we have "RC2", and so on? However "RC1" may be no longer needed once we have established enough reruns in "RC2", in the current plan.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for pointing that out, John Swinbank ! I understand the current RC dataset will need to be expanded at some point (late this year?). But how often will it change? If "the RC set" changes frequently but follows a ~fixed rule, this tag can be used for all RC-like reprocessing (3 tracts, <10TB, results going into /project space, etc). If we only need a ~one-time change or changes are rare, how about naming this "RC1", so later we have "RC2", and so on? However "RC1" may be no longer needed once we have established enough reruns in "RC2", in the current plan.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Jim Bosch or Robert Lupton, do you have opinions whether we want (a) one single /datasets/hsc/repo/rerun/RC/ link for any "RC-like" HSC reprocessing, or (b) versioning the link names with the specific datasets used in the reprocessing?

            I'm mildly inclined to (a) for now, so we have the flexibility and convenience in the short term (~months). Meanwhile we learn more how this dataset and weekly reprocessing evolve and we can re-access later?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jim Bosch or Robert Lupton , do you have opinions whether we want (a) one single /datasets/hsc/repo/rerun/RC/ link for any "RC-like" HSC reprocessing, or (b) versioning the link names with the specific datasets used in the reprocessing? I'm mildly inclined to (a) for now, so we have the flexibility and convenience in the short term (~months). Meanwhile we learn more how this dataset and weekly reprocessing evolve and we can re-access later?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I see no objections so will go ahead, mark this Adopted, and create the RC link (option (a) above for now)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I see no objections so will go ahead, mark this Adopted, and create the RC link (option (a) above for now)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            This is implemented now. If we want a new link for new datasets (e.g. DM-11345) we will create a new RFC later.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - This is implemented now. If we want a new link for new datasets (e.g. DM-11345 ) we will create a new RFC later.

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Greg Daues, Hsin-Fang Chiang, John Swinbank, Paul Domagala [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End: