Details
-
Type:
RFC
-
Status: Adopted
-
Resolution: Unresolved
-
Component/s: DM
-
Labels:None
Description
While the Gen3 Butler provides some intrinsic structure to its data repositories, considerably more is left to convention (often encoded in higher-level packages, like obs_base). This RFC is a proposal for how to organize data repositories in detail, focusing on collection naming conventions, filesystem locations, and developer workflows. The immediate focus is the environment at NCSA, but it is hoped that much of this will hold for the IDF and USDF as well.
The content of the proposal is on the tickets/DM-27147 branch of DMTN-167, which you can find:
To improve threading (there are a lot of disparate topics here), I'd prefer to have most discussion on the PR. If you want to make a big-picture comment, just find a semi-relevant line. Please also watch the PR if you'd like to keep up with updates. I do plan to update the technote in response to uncontroversial comments throughout the RFC.
There is one topic that I would like to keep here on Jira: the "meta" discussion of where these policies should live after they are adopted. I assume the answer is mostly the DM Developer Guide, but that may not be appropriate for everything.
There are also code branches for DM-27147, which include the changes to the codebase necessary to implement these policies. I'm hoping to get these reviewed and merged before the RFC completes, as I'm confident they'll bring us much closer to whatever the agreed-upon policy ends up being, and I strongly suspect we'll want to start trying to actually stand up a shared repository based on this proposal before we've given it enough time to gather comments.
Attachments
Issue Links
- is triggering
-
- To Do
-
- Done
-
- Done
-
- Done
-
- Done
- relates to
-
- Done
- mentioned in
-
Page Loading...
-
-
-
-
-
-
-
-
-
Activity
It is my understanding based upon conversation in #dm-middleware that we'll need another gen3 repo area for the CCS-written O raw images. (I'm not sure if this will remain restricted to ComCam, so naming it something like CC_O may not be ideal.)
Also, have we gotten confirmation that there will be no collisions between the DC2-mini inputs and the DR6 inputs since this is putting them all in the same repo? Or is the DC2-mini more like the temporary RC2 repos and just goes away with only the DR6 inputs going into the new repo?
I believe the future DC2-mini runs can be performed using a subset of the DR6-transfer raws (and calibs, etc) as inputs, so they can just be different output collections in the same repo. They're all Run2.2i, and we should even be able to include other (simulation) Run versions in that same repo if needed.
I think this is ready for adoption, aside from attaching implementation tickets. I'm sure there will be some work for the middleware team (and me in particular) as part of that, but I think it probably makes sense for the main implementation to be an NCSA one. Michelle Butler [X], can I ask you to ask someone to do that?
I've adopted this with what I believe are most of the "end condition" triggering tickets in place. I believe Michelle Gower is working on some other triggering tickets that we can consider blockers on these ones. I would love for someone other than me to add new tickets modeled on DM-28636 for the CCSO and NCSA-teststand-data repositories detailing the work to be done for those (including what instruments they will have and where the data for them currently lives).
This technote has improved markedly since you first opened this RFC - at first, it was close to impossible for me to slog through and understand. A combination of "me actually using gen3" and you working very hard (to add examples, clarify various concepts, etc.) has transformed it into an invaluable and friendly resource. Thank you!