Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-273

Develop and then create the organizational structure for DM Confluence space

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Sprint:
      DevOps Sprint 1
    • Team:
      SQuaRE

      Description

      Before we start populating the DM Confluence space with active pages, we should define an overall organizational structure/taxonomy.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            I was thinking I would start building out pieces of this, at least in terms of system design, in conjunction with SAT meetings on design issues/holes. I'd start each week by creating an outline of the Confluence space in a given topic area and filling in what I knew about it, then allow others to comment and fill in additional points, and then have the SAT meeting to go over conflicts and missing parts.

            I'm quite happy to leave development of other parts of the DM space to others.

            Show
            ktl Kian-Tat Lim added a comment - I was thinking I would start building out pieces of this, at least in terms of system design, in conjunction with SAT meetings on design issues/holes. I'd start each week by creating an outline of the Confluence space in a given topic area and filling in what I knew about it, then allow others to comment and fill in additional points, and then have the SAT meeting to go over conflicts and missing parts. I'm quite happy to leave development of other parts of the DM space to others.
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            Although Confluence content can at some level evolve organically, there are a few things to bear in mind. Perhaps the first thing is to decide what content needs it own space, vs. what can live within the space of some larger context. For instance, it is somewhat painful to develop some pages, then decide they need their own space afterward (speaking from bitter experience). Related to this, one should decide on a lifecycle model for the content (maintained or strictly archival, versioned, etc.).

            I think it would help if we comment here on the topics and/or categories of information we hope to capture, proposed lifecycle model, and ideas for structure within the Greater DM. That would give us a fighting chance of creating a DM Confluence presence that won't need major revision for awhile.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - Although Confluence content can at some level evolve organically, there are a few things to bear in mind. Perhaps the first thing is to decide what content needs it own space, vs. what can live within the space of some larger context. For instance, it is somewhat painful to develop some pages, then decide they need their own space afterward (speaking from bitter experience). Related to this, one should decide on a lifecycle model for the content (maintained or strictly archival, versioned, etc.). I think it would help if we comment here on the topics and/or categories of information we hope to capture, proposed lifecycle model, and ideas for structure within the Greater DM. That would give us a fighting chance of creating a DM Confluence presence that won't need major revision for awhile.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            I've been pondering the organization of information related to the rolling Development cycle. I consider it to be primarily archival. The Release under current development will be under active change. On completion/release, its development pages will go static and be maintained for trolling. Here's a strawman; ignore the lack of layup. '[xxxxxx->]' indicates an off page link.

            On Front Page for DM:

            Releases

            • Summer 2014 Under development [Development->]
            • Winter 2014 [8.0.0->] , [8.0.1->] [Development->]
              etc

            [Development->] page for a specific Release
            Release: Summer 2014
            Goal: <some text outlining the LDM-240 rolling wave deliverables + timely extras>
            Teams:

            • DevOps [Docs->] [Meetings->]
              + Focus: <stt on general focus of team's sprints>
              + Team: Mario Juric*, Robyn Allsman, Dick Shaw
            • Meas Overhaul [Docs->] [Meetings->]
              + Focus: <stt on general focus of team's sprints>
              + Team: Jim Bosch*, Yusra Al Sayyad, Perry Gee
              ....

            [Meetings->] page for a specific team for a specific Release

            • [date->] type: <one of Design, Review, Sprint End, Standup, etc....>
            • [date->] type: <one of Design, Review, Sprint End, Standup, etc....>
              etc

            [Docs->] page for a specific team for a specific Release - the front page should be index of a collection of docs; or what/how-ever the team wants to record its documents.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - I've been pondering the organization of information related to the rolling Development cycle. I consider it to be primarily archival. The Release under current development will be under active change. On completion/release, its development pages will go static and be maintained for trolling. Here's a strawman; ignore the lack of layup. ' [xxxxxx->] ' indicates an off page link. On Front Page for DM: Releases Summer 2014 Under development [Development->] Winter 2014 [8.0.0->] , [8.0.1->] [Development->] etc [Development->] page for a specific Release Release: Summer 2014 Goal: <some text outlining the LDM-240 rolling wave deliverables + timely extras> Teams: DevOps [Docs->] [Meetings->] + Focus: <stt on general focus of team's sprints> + Team: Mario Juric*, Robyn Allsman, Dick Shaw Meas Overhaul [Docs->] [Meetings->] + Focus: <stt on general focus of team's sprints> + Team: Jim Bosch*, Yusra Al Sayyad, Perry Gee .... [Meetings->] page for a specific team for a specific Release [date->] type: <one of Design, Review, Sprint End, Standup, etc....> [date->] type: <one of Design, Review, Sprint End, Standup, etc....> etc [Docs->] page for a specific team for a specific Release - the front page should be index of a collection of docs; or what/how-ever the team wants to record its documents.
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            Robyn, I like your outline a lot, and I think it will work. I have a couple of thoughts:

            1. For the title of the section (top DM Space), relabel "Releases" to "Release Development" or "Development Cycles". My thought was to avoid some confusion from external users, who might think "Oh, this is where I should to to obtain a new release, or find release notes."

            2. To what extent should "Team Docs" be maintained? I think the expectation should be clear, so that authoritative reference documentation is not confused with "this is what we thought about this topic when we discussed it last" pages (which currently describes most of the Twiki content). Design reference documents would be in the first category, I should think; as distinguished from a design discussion document.

            3. For reference docs, we should think about how to make them highly visible and easily accessible, rather than have them buried within a team Space. I would argue that such docs must have an owner, who is tasked with maintaining them for as long as necessary.

            4. What about externally published technical papers? Where do the source and publication docs live? I'm not sure if they will appear in the successor to DocuShare, but we should at least link to them.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - Robyn, I like your outline a lot, and I think it will work. I have a couple of thoughts: 1. For the title of the section (top DM Space), relabel "Releases" to "Release Development" or "Development Cycles". My thought was to avoid some confusion from external users, who might think "Oh, this is where I should to to obtain a new release, or find release notes." 2. To what extent should "Team Docs" be maintained? I think the expectation should be clear, so that authoritative reference documentation is not confused with "this is what we thought about this topic when we discussed it last" pages (which currently describes most of the Twiki content). Design reference documents would be in the first category, I should think; as distinguished from a design discussion document. 3. For reference docs, we should think about how to make them highly visible and easily accessible, rather than have them buried within a team Space. I would argue that such docs must have an owner, who is tasked with maintaining them for as long as necessary. 4. What about externally published technical papers? Where do the source and publication docs live? I'm not sure if they will appear in the successor to DocuShare, but we should at least link to them.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Dick, I take your point about the design docs having more permanence than I ascribe to the rest of the development cycle docs. I think the permanent design docs will always be vetted by the SAT. These would fit within the topical groupings that K-T is proposing for the SAT Confluence content.

            Regarding external publications, we need a "Publications" index for

            • DM Tech Talks (for youtube, slideshows, etc)
            • Reports and Reviews
            • Sundry Publications of Interest
              I'll defer the location issue to later or to others whichever comes first.
            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Dick, I take your point about the design docs having more permanence than I ascribe to the rest of the development cycle docs. I think the permanent design docs will always be vetted by the SAT. These would fit within the topical groupings that K-T is proposing for the SAT Confluence content. Regarding external publications, we need a "Publications" index for DM Tech Talks (for youtube, slideshows, etc) Reports and Reviews Sundry Publications of Interest I'll defer the location issue to later or to others whichever comes first.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Adding the Release info to the Data Management page.
            Adding an outline for the Summer 2014 Development page
            Adding placeholders for the sundry DM Management teams

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Adding the Release info to the Data Management page. Adding an outline for the Summer 2014 Development page Adding placeholders for the sundry DM Management teams
            Hide
            ktl Kian-Tat Lim added a comment -

            Here's my proposal, a little late.

            • DM Space (intended for all developers)
              • Mission Statement
              • Links to SW User Guide spaces for each release
              • Link to Dev Guide space for internal DM developers
              • External document index
                • Data Challenge reports
                • Papers
                • Presentations
                • Reviews
              • Standards and Policies
                • Glossary
                • Coding Standards
              • System Design
                • Detailed expansion of LDMs
                • Implementation notes (by functional area)
              • Development environment/Tools
                • git
                • buildbot
                • gcc, gdb
                • Python, pdb
                • Profiling and valgrind
                • doxygen/sphinx
                • Other dev tools
            • Dev Guide Space (intended for internal DM developers)
              • New developer information
              • Communications
                • JIRA, Confluence, Stash
                • HipChat
                • Google Hangouts
                • AT Conference
              • DM Organization
                • DMLT
                  • Meeting notes/action items
                • TCT
                  • Meeting notes/action items
                • SAT
                  • Meeting notes/action items
                • AWG
                  • Meeting notes/action items
                • DBWG
                  • Meeting notes/action items
                • IWG
                  • Meeting notes/action items
              • Per-release development information
                • Planning documents
                • Team organization
                  • Meeting notes/action items
                • Long code reviews
            • Per-release documentation, one space per version
              • Installing
              • Tutorial/demo
              • Using
              • Links to doxygen/sphinx output
            Show
            ktl Kian-Tat Lim added a comment - Here's my proposal, a little late. DM Space (intended for all developers) Mission Statement Links to SW User Guide spaces for each release Link to Dev Guide space for internal DM developers External document index Data Challenge reports Papers Presentations Reviews Standards and Policies Glossary Coding Standards System Design Detailed expansion of LDMs Implementation notes (by functional area) Development environment/Tools git buildbot gcc, gdb Python, pdb Profiling and valgrind doxygen/sphinx Other dev tools Dev Guide Space (intended for internal DM developers) New developer information Communications JIRA, Confluence, Stash HipChat Google Hangouts AT Conference DM Organization DMLT Meeting notes/action items TCT Meeting notes/action items SAT Meeting notes/action items AWG Meeting notes/action items DBWG Meeting notes/action items IWG Meeting notes/action items Per-release development information Planning documents Team organization Meeting notes/action items Long code reviews Per-release documentation, one space per version Installing Tutorial/demo Using Links to doxygen/sphinx output
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            K-T: Your proposal is a great start. I have some detailed questions and comments. Also, I think most entries in the outline should be tagged with annotations, such as owner, whether the content is living or archival, controlled in any way, etc. This is very difficult to do in JIRA comments. I suggest we migrate your outline to another medium (e.g., a spreadsheet) and continue the refinement.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - K-T: Your proposal is a great start. I have some detailed questions and comments. Also, I think most entries in the outline should be tagged with annotations, such as owner, whether the content is living or archival, controlled in any way, etc. This is very difficult to do in JIRA comments. I suggest we migrate your outline to another medium (e.g., a spreadsheet) and continue the refinement.
            Hide
            ktl Kian-Tat Lim added a comment -

            The above has been placed in a Google Doc with commenting and editing privileges enabled.

            Show
            ktl Kian-Tat Lim added a comment - The above has been placed in a Google Doc with commenting and editing privileges enabled.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            K-T: the structure looks good.

            What is the value-added of using a separate space for the outside users and developers and the internal DM developers? Will the internal developers section be restricted in any way?

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - K-T: the structure looks good. What is the value-added of using a separate space for the outside users and developers and the internal DM developers? Will the internal developers section be restricted in any way?
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            I like the outside users/internal developers split. There are times when I'd like to be able to have a place for a new document that I'm working on that it's ready for outside eyes yet, and it'd be nice to have a staging area. That's especially true for documents that might never be published externally for one reason or another.

            Show
            spietrowicz Steve Pietrowicz added a comment - I like the outside users/internal developers split. There are times when I'd like to be able to have a place for a new document that I'm working on that it's ready for outside eyes yet, and it'd be nice to have a staging area. That's especially true for documents that might never be published externally for one reason or another.
            Hide
            ktl Kian-Tat Lim added a comment -

            I was mostly just following the existing pattern with SW User Guide and Dev Guide. To some extent, I think it will be more attractive to external contributors if we don't burden them with all of our internal organizational details.

            Show
            ktl Kian-Tat Lim added a comment - I was mostly just following the existing pattern with SW User Guide and Dev Guide. To some extent, I think it will be more attractive to external contributors if we don't burden them with all of our internal organizational details.
            Hide
            ktl Kian-Tat Lim added a comment -

            Steve Pietrowicz: Individual pages can be protected, I believe. I don't think we want the (vast majority of the) internal developer pages to be unviewable by the public; that would be contrary to the openness/transparency of our development. There are certain types of pages (like product evaluations) that should probably be restricted, as Jacek Becla has pointed out in the past.

            Show
            ktl Kian-Tat Lim added a comment - Steve Pietrowicz : Individual pages can be protected, I believe. I don't think we want the (vast majority of the) internal developer pages to be unviewable by the public; that would be contrary to the openness/transparency of our development. There are certain types of pages (like product evaluations) that should probably be restricted, as Jacek Becla has pointed out in the past.
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            We should perhaps be mindful of two separable issues: How we create and organize content for a particular audience, and the way certain (limited) content would be protected. Material in the latter category might involve pages under development, preparation for a new product release, and content that is not relevant for, say, external users such as machine/cluster status or availability. For unprotected material, Confluence supports incorporating it into multiple contexts (though the implementation is a little clunky).

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - We should perhaps be mindful of two separable issues: How we create and organize content for a particular audience, and the way certain (limited) content would be protected. Material in the latter category might involve pages under development, preparation for a new product release, and content that is not relevant for, say, external users such as machine/cluster status or availability. For unprotected material, Confluence supports incorporating it into multiple contexts (though the implementation is a little clunky).
            Hide
            ktl Kian-Tat Lim added a comment -

            I think the amount of protected content should be extremely small. Pages under development should normally still be public, perhaps with "DRAFT" stamped on them. Preparation for a new release as well. Machine/cluster status held in Confluence, by its nature, will be less dynamic, and I don't see much reason for it to be non-public.

            Show
            ktl Kian-Tat Lim added a comment - I think the amount of protected content should be extremely small. Pages under development should normally still be public, perhaps with "DRAFT" stamped on them. Preparation for a new release as well. Machine/cluster status held in Confluence, by its nature, will be less dynamic, and I don't see much reason for it to be non-public.
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            @KT: I beg to differ. When pages are under construction the content can change, shift or be deleted at any moment. There is rarely a reason to offer content that has not been prepared. Making a draft (i.e., mostly coherent, complete content, ready for review) is another matter. We don't release software while developers are still typing the code, or technical papers until they have been written; we ought not do that for documentation either.

            Preparation for new releases means we are describing features that are not available until a release has been cut. So the pages really ought to be restricted to the DM team until then (for review, etc.). The current situation is a little different, in that we don't yet have comprehensive, complete documents. So we're still in effect documenting the current release (and almost anything is an improvement). But this will change with future releases.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - @KT: I beg to differ. When pages are under construction the content can change, shift or be deleted at any moment. There is rarely a reason to offer content that has not been prepared. Making a draft (i.e., mostly coherent, complete content, ready for review) is another matter. We don't release software while developers are still typing the code, or technical papers until they have been written; we ought not do that for documentation either. Preparation for new releases means we are describing features that are not available until a release has been cut. So the pages really ought to be restricted to the DM team until then (for review, etc.). The current situation is a little different, in that we don't yet have comprehensive, complete documents. So we're still in effect documenting the current release (and almost anything is an improvement). But this will change with future releases.
            Hide
            ktl Kian-Tat Lim added a comment -

            We don't release software while it's under development, but we do make it available in git/cgit/Stash for people to see, even for private branches and personal repositories. Near-total transparency in development, including development of documentation, is the only way to have a true Open Source project. The reason to offer content that has not been prepared is to get early feedback from knowledgeable contributors.

            Otherwise, we might as well just use DocuShare.

            Show
            ktl Kian-Tat Lim added a comment - We don't release software while it's under development, but we do make it available in git/cgit/Stash for people to see, even for private branches and personal repositories. Near-total transparency in development, including development of documentation, is the only way to have a true Open Source project. The reason to offer content that has not been prepared is to get early feedback from knowledgeable contributors. Otherwise, we might as well just use DocuShare.
            Hide
            rhl Robert Lupton added a comment -

            I totally agree with KT. We need a mechanism to say, "Look here — this is the current version" but that mechanism should not be to hide the nascent next release.

            We should have a "need-to-hide" policy; by default everything's open.

            Show
            rhl Robert Lupton added a comment - I totally agree with KT. We need a mechanism to say, "Look here — this is the current version" but that mechanism should not be to hide the nascent next release. We should have a "need-to-hide" policy; by default everything's open.
            Hide
            mjuric Mario Juric added a comment -

            I agree with K-T; we gain nothing by keeping drafts private. That's been the experience of open source projects. Here's a random example: http://clang.llvm.org/docs/ReleaseNotes.html

            I realize there's a psychological barrier to people not wanting to put their draft or unfinished thoughts online, but we just have to grow out of it. Yes, those drafts will occasionally have embarrassing mistakes or be dead ends, but that won't reflect poorly on anyone (nor on the organization as a whole). Quite the opposite! The few eyes that see such cases and tell us about it will make our work easier.

            I'd therefore prefer not to have any separation into public/private or internal/external spaces; unless something is vendor-proprietary or needs to be hidden for privacy reasons, we should have it in the open.

            We've functioned that way on the old Trac, and I don't think anyone's even noticed .

            Show
            mjuric Mario Juric added a comment - I agree with K-T; we gain nothing by keeping drafts private. That's been the experience of open source projects. Here's a random example: http://clang.llvm.org/docs/ReleaseNotes.html I realize there's a psychological barrier to people not wanting to put their draft or unfinished thoughts online, but we just have to grow out of it. Yes, those drafts will occasionally have embarrassing mistakes or be dead ends, but that won't reflect poorly on anyone (nor on the organization as a whole). Quite the opposite! The few eyes that see such cases and tell us about it will make our work easier. I'd therefore prefer not to have any separation into public/private or internal/external spaces; unless something is vendor-proprietary or needs to be hidden for privacy reasons, we should have it in the open. We've functioned that way on the old Trac, and I don't think anyone's even noticed .
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            It's great to get really feedback when you're ready to get it. It's annoying if you're in the middle of something and start getting feedback when you don't feel it's ready.

            See: http://dilbert.com/strips/comic/1995-05-29/

            Show
            spietrowicz Steve Pietrowicz added a comment - It's great to get really feedback when you're ready to get it. It's annoying if you're in the middle of something and start getting feedback when you don't feel it's ready. See: http://dilbert.com/strips/comic/1995-05-29/
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            I think a better analogy is that we do not make code available to anyone until the developer takes some action through git. Until then the developer is charged with writing the code, constructing and executing unit tests, and preparing for system integration testing. Similarly, for documentation I would like to finish the paragraph, table, figure, or whatever and present what is intended before receiving comments from users who may be confused with inadvertently incorrect info. It's not a matter of "growing out" of a practice, it's about presenting professional documentation.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - I think a better analogy is that we do not make code available to anyone until the developer takes some action through git. Until then the developer is charged with writing the code, constructing and executing unit tests, and preparing for system integration testing. Similarly, for documentation I would like to finish the paragraph, table, figure, or whatever and present what is intended before receiving comments from users who may be confused with inadvertently incorrect info. It's not a matter of "growing out" of a practice, it's about presenting professional documentation.
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            Agreed

            Show
            spietrowicz Steve Pietrowicz added a comment - Agreed
            Hide
            ktl Kian-Tat Lim added a comment -

            I think you should move away from the code analogies Instead of making the case for less transparency for documentation development, you're making the case for having more transparent code development. Developers should be taking some action through git every day, if not every hour. Push early and often should be the watchword. If people are developing in isolation on their own machines for more than a day or two, we're doing something wrong.

            The problem that has been stated is that there may be unwanted early feedback. There are multiple ways of suppressing this, each of which has side effects. The simplest, to my mind, is to mark work in progress as such (which we do with code), and request that feedback be held until the work is more complete. This could perhaps be further ameliorated by dividing the work into small sections that can each be completed rapidly, even if an overall reorganization is required later.

            Think Wikipedia, not Britannica.

            Show
            ktl Kian-Tat Lim added a comment - I think you should move away from the code analogies Instead of making the case for less transparency for documentation development, you're making the case for having more transparent code development. Developers should be taking some action through git every day, if not every hour. Push early and often should be the watchword. If people are developing in isolation on their own machines for more than a day or two, we're doing something wrong. The problem that has been stated is that there may be unwanted early feedback. There are multiple ways of suppressing this, each of which has side effects. The simplest, to my mind, is to mark work in progress as such (which we do with code), and request that feedback be held until the work is more complete. This could perhaps be further ameliorated by dividing the work into small sections that can each be completed rapidly, even if an overall reorganization is required later. Think Wikipedia, not Britannica.
            Hide
            mjuric Mario Juric added a comment -

            @KTL +1e100

            I understand the worry about unwanted feedback, but I think past experience has shown that we rarely get that. And occasionally, we get feedback that's unwanted by the developer, but really keeps them on track to delivering what we need rather than what they think should be done (btw., thanks to everyone who's complained about my mods to EUPS while they were still in progress ).

            Look guys, bottom line is that we need to be open for this to work. And we will be. We're trying to build the best possible piece of software for LSST that the collaborations and the community will trust, use, and ultimately contribute to. We can't do it without fostering a community, and being transparent with our development process, work in progress, plans and thoughts. That's the way the software world is going. That's also the way modern facility projects are going (I recently had a great chat about this with the JWST project scientist).

            Expectations and the ways people work with software or on big software projects have changed dramatically since the day of IRAF and large software manuals. People know what beta is, and what a draft is, and are aware that software is always a work in progress if you expose those traits to them. Especially the crowd that's going to use this code in 2020-ies and 2030-ies. If we start closing of parts for whatever reasons we'll drive people away, esp. the kinds of people we want to hire or get contributions from. That would be a waste of our talents, and Uncle Sam's dollars.

            Show
            mjuric Mario Juric added a comment - @KTL +1e100 I understand the worry about unwanted feedback, but I think past experience has shown that we rarely get that. And occasionally, we get feedback that's unwanted by the developer, but really keeps them on track to delivering what we need rather than what they think should be done (btw., thanks to everyone who's complained about my mods to EUPS while they were still in progress ). Look guys, bottom line is that we need to be open for this to work. And we will be. We're trying to build the best possible piece of software for LSST that the collaborations and the community will trust, use, and ultimately contribute to. We can't do it without fostering a community, and being transparent with our development process, work in progress, plans and thoughts. That's the way the software world is going. That's also the way modern facility projects are going (I recently had a great chat about this with the JWST project scientist). Expectations and the ways people work with software or on big software projects have changed dramatically since the day of IRAF and large software manuals. People know what beta is, and what a draft is, and are aware that software is always a work in progress if you expose those traits to them. Especially the crowd that's going to use this code in 2020-ies and 2030-ies. If we start closing of parts for whatever reasons we'll drive people away, esp. the kinds of people we want to hire or get contributions from. That would be a waste of our talents, and Uncle Sam's dollars.
            Hide
            spietrowicz Steve Pietrowicz added a comment - - edited

            This isn't a matter of being open or not. Everyone's for that.

            I, and probably others, would like to have a chance to get our thoughts together before presenting new docs to everyone. In other words, we'd like to be able to say "Here's the first draft I feel I can share with everyone. This isn't finalized, and it's a draft. Please take a look." I don't think there's anything wrong with that.

            Show
            spietrowicz Steve Pietrowicz added a comment - - edited This isn't a matter of being open or not. Everyone's for that. I, and probably others, would like to have a chance to get our thoughts together before presenting new docs to everyone. In other words, we'd like to be able to say "Here's the first draft I feel I can share with everyone. This isn't finalized, and it's a draft. Please take a look." I don't think there's anything wrong with that.
            Hide
            mjuric Mario Juric added a comment -

            Then begin the document with:

            "Here's the first draft I feel I can share with everyone. This isn't finalized, and it's a draft. Please take a look."

            or, while you're not there yet:

            This is a random grab-bag of uncooked ideas that you shouldn't be looking at unless asked for. Please ignore.

            There's even markup that can make these bold+underlined+red .

            (and, as a last resort, there are also Personal Spaces where pages can be developed in privacy, if one strongly wishes to do so).

            Show
            mjuric Mario Juric added a comment - Then begin the document with: "Here's the first draft I feel I can share with everyone. This isn't finalized, and it's a draft. Please take a look." or, while you're not there yet: This is a random grab-bag of uncooked ideas that you shouldn't be looking at unless asked for. Please ignore. There's even markup that can make these bold+underlined+red . (and, as a last resort, there are also Personal Spaces where pages can be developed in privacy, if one strongly wishes to do so).
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            All caps and blinking too?

            Show
            spietrowicz Steve Pietrowicz added a comment - All caps and blinking too?
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Dick and I got together yesterday to discuss the layout. It remains mostly as stated by K-T but there were a few changes. Notably the contents of the DevGuide space was exchanged with the DM space. The DevGuide contents are more static and reflect the processes, rules and requirements by which the developers perform their work. The DM space reflects the instant-in-time development effort; its contents are frequently in flux.

            We considered the need for labels to annotate containers and pages to give a clue about who's responsible for curating, how often should they be reviewed for update, who may edit them. Proposals are:

            • curator: c_<name>
            • controlling group: g_<name> or g_none where name = {TCT, SAT, DMLT, APWG, DBWG, MWG, TWG}

              ; done thru permissions or a label.

            • longevity: {living (in flux), versioned (stable once Released), archival (stable)

            Confluence Spaces

            Developer User Guide Space (intended for developers of all stripes)

            • Mission Statement
            • Links to SW User Guide spaces for each release (living; controlled, c_Shaw)
            • Link to DM Guide space for internal/advanced DM developers (living, controlled?)
            • External document index (archival, curator is lead author)
              • Data Challenge reports
              • Papers
              • Presentations
              • Reviews
            • Standards and Policies (mostly: archival, controlled: g_tct
              • Glossary
              • Coding Standards
              • Procedures for: design and code review, etc
            • System Design (living, controlled, g_sat)
              • Detailed expansion of LDMs (including Requirements)
              • Implementation notes (by functional area)
            • Development environment/Tools (Some tutorial-level material e.g., illustrate the process of checking out code, creating a branch, etc. in the DM way.) Expect Developers to later add FAQs.)
              • git
              • buildbot
              • gcc, gdb
              • Python, pdb
              • Profiling and valgrind
              • doxygen/sphinx
              • Other dev tools

            DM Space (intended for internal DM developers) (living)

            • New developer information
              • Accounts
              • Machines
                • NCSA development cluster
                • lsst-web
                • lsst-db
              • Development/environment Tools (advanced usage)
              • Process (Agile, etc.)
            • Communications (living)
              • JIRA, Confluence, Stash
              • HipChat
              • Google Hangouts
              • AT Conference
              • DM Calendars
              • Mail Lists
            • DM Organization
              • DMLT (DM Leadership Team)
                • Meeting notes/action items
              • TCT (DM Technical Control Team)
                • Meeting notes/action items
              • SAT (DM System Architecture Team)
                • Meeting notes/action items
              • AWG (Applications Working Group)
                • Meeting notes/action items
              • DBWG (Database Working Group)
                • Meeting notes/action items
              • IWG (Interface Working Group)
                • Meeting notes/action items
              • TWG (Testing Working Group)
                • Meeting notes/action items
            • Per-release development information
              • Planning documents (This includes strawmans, design & implementation discussions and a pointer to final SAT approved design doc maintained in the permanent location for DM SAT design documents. These docs would be within each team’s block
              • Team organization (archival, not controlled)
                • Meeting notes/action items
              • Long code reviews (Code reviews may be considered either meetings or plain docs but belong with the Team spawning them. Except for the SAT approved designs.

            Software User Guide Spaces: Per-release documentation, one space per version; archival (with comments) for all but current

            • Installing
            • Tutorial/demo
            • Using
            • Links to doxygen/sphinx output
            • High-level descriptions of packages
              • Descriptions of meta concepts (measurement, calibration, data organization)
            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Dick and I got together yesterday to discuss the layout. It remains mostly as stated by K-T but there were a few changes. Notably the contents of the DevGuide space was exchanged with the DM space. The DevGuide contents are more static and reflect the processes, rules and requirements by which the developers perform their work. The DM space reflects the instant-in-time development effort; its contents are frequently in flux. We considered the need for labels to annotate containers and pages to give a clue about who's responsible for curating, how often should they be reviewed for update, who may edit them. Proposals are: curator: c_<name> controlling group: g_<name> or g_none where name = {TCT, SAT, DMLT, APWG, DBWG, MWG, TWG} ; done thru permissions or a label. longevity: {living (in flux), versioned (stable once Released), archival (stable) Confluence Spaces Developer User Guide Space (intended for developers of all stripes) Mission Statement Links to SW User Guide spaces for each release (living; controlled, c_Shaw) Link to DM Guide space for internal/advanced DM developers (living, controlled?) External document index (archival, curator is lead author) Data Challenge reports Papers Presentations Reviews Standards and Policies (mostly: archival, controlled: g_tct Glossary Coding Standards Procedures for: design and code review, etc System Design (living, controlled, g_sat) Detailed expansion of LDMs (including Requirements) Implementation notes (by functional area) Development environment/Tools (Some tutorial-level material e.g., illustrate the process of checking out code, creating a branch, etc. in the DM way.) Expect Developers to later add FAQs.) git buildbot gcc, gdb Python, pdb Profiling and valgrind doxygen/sphinx Other dev tools DM Space (intended for internal DM developers) (living) New developer information Accounts Machines NCSA development cluster lsst-web lsst-db Development/environment Tools (advanced usage) Process (Agile, etc.) Communications (living) JIRA, Confluence, Stash HipChat Google Hangouts AT Conference DM Calendars Mail Lists DM Organization DMLT (DM Leadership Team) Meeting notes/action items TCT (DM Technical Control Team) Meeting notes/action items SAT (DM System Architecture Team) Meeting notes/action items AWG (Applications Working Group) Meeting notes/action items DBWG (Database Working Group) Meeting notes/action items IWG (Interface Working Group) Meeting notes/action items TWG (Testing Working Group) Meeting notes/action items Per-release development information Planning documents (This includes strawmans, design & implementation discussions and a pointer to final SAT approved design doc maintained in the permanent location for DM SAT design documents. These docs would be within each team’s block Team organization (archival, not controlled) Meeting notes/action items Long code reviews (Code reviews may be considered either meetings or plain docs but belong with the Team spawning them. Except for the SAT approved designs. Software User Guide Spaces: Per-release documentation, one space per version; archival (with comments) for all but current Installing Tutorial/demo Using Links to doxygen/sphinx output High-level descriptions of packages Descriptions of meta concepts (measurement, calibration, data organization)
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            I juggled a few items around in your proposal . Read the last comment. I don't think we can cast anything in concrete until we all become comfortable with the added capabilities that Confluence gives us in comparison to TracWiki.

            A deviation from the proposal has already occurred to me based on the Confluence meetings template which automatically aggregates/indexes ALL meetings in a space. Depending on how many individual meetings are recorded, this could simplify our meeting lookup or make it unwieldy. I am leaning on trying it for a while and then splitting out into sprint buckets if the overall number of meetings gets too large (probably using a per sprint Meetings template).

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - I juggled a few items around in your proposal . Read the last comment. I don't think we can cast anything in concrete until we all become comfortable with the added capabilities that Confluence gives us in comparison to TracWiki. A deviation from the proposal has already occurred to me based on the Confluence meetings template which automatically aggregates/indexes ALL meetings in a space. Depending on how many individual meetings are recorded, this could simplify our meeting lookup or make it unwieldy. I am leaning on trying it for a while and then splitting out into sprint buckets if the overall number of meetings gets too large (probably using a per sprint Meetings template).
            Hide
            ktl Kian-Tat Lim added a comment -

            Looks pretty good to me. Let's try it and iterate as necessary. The one thing I still bristle a little bit at is "controlled" (and "living", "versioned", "archival"). Any really controlled documents need to be in DocuShare (or its successor). I'd say that it might be sufficient to set the expectation that a page with a designated curator/group should not be edited without the curator's permission. Other pages are "living" by default. Would that work, and reduce the number of labels?

            Show
            ktl Kian-Tat Lim added a comment - Looks pretty good to me. Let's try it and iterate as necessary. The one thing I still bristle a little bit at is "controlled" (and "living", "versioned", "archival"). Any really controlled documents need to be in DocuShare (or its successor). I'd say that it might be sufficient to set the expectation that a page with a designated curator/group should not be edited without the curator's permission. Other pages are "living" by default. Would that work, and reduce the number of labels?
            Hide
            shaw Richard Shaw [X] (Inactive) added a comment -

            Kian-Tat Lim "Controlled" in this sense means no change allowed except through an official DM process. Ex: coding standards, which are in the domain of the TCT (? or is it SAT?). In such a case (there aren't too many) the owner of the Confluence page (or the Group, which we haven't established yet) would have exclusive ability to edit said page. There is I think an important distinction between "controlled" and "configured" where the latter is a Project-level control, and requirements & design documents at that level live in the black hole known as DocuShare.

            Show
            shaw Richard Shaw [X] (Inactive) added a comment - Kian-Tat Lim "Controlled" in this sense means no change allowed except through an official DM process. Ex: coding standards, which are in the domain of the TCT (? or is it SAT?). In such a case (there aren't too many) the owner of the Confluence page (or the Group, which we haven't established yet) would have exclusive ability to edit said page. There is I think an important distinction between "controlled" and "configured" where the latter is a Project-level control, and requirements & design documents at that level live in the black hole known as DocuShare.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            K-T,
            Thanks for your comments and the earlier outline. Iteration it is. We can hash out the labels as we go along.

            Regarding the labels: Let's work on a word change here. Instead of 'controlled' what feeling do you get about 'approved'? It's not so rigid. The idea is to let folks know that the imprimatur of some body has looked over and agreed with the contents/intent of the page. The label bestows slightly more weigh, by virtue of the review status, than a random page written by a developer. Think about your trust level for random web pages vs web pages on a respected site.

            I agree that the Project controlled documents need to be archived in Docushare (or follow-on). But we have many documents which don't rise to that exacting standard yet inform the Developers about management's expectations on them. For example,the TCT decided some years back to allow upreving of selected 3rd party products to occur on a semi-annual period without reapproval by the TCT (if they first verified the uprev worked,etc). This is not an LDM controlled rule but an internal DM Policy. (Under the new division of labor, it might now be a SAT Policy instead of a TCT Policy.)

            Regarding 'living', 'versioned', 'archived' - I think we only need one designating 'versioned' to indicate that a particular page is anchored to a specific Release. We wouldn't need to designate 'versioned' on the various versioned Spaces we have; that would be redundant. Dick should respond with his sentiments on this, too.

            I view 'curator' as the person who is responsible for periodically looking over their document fiefdom for obsolete pages which either should be updated or removed. Confluence has nice searching facilities for searching on page labels. However, I wouldn't want to inhibit anyone who was willing to update a page's content on their own. I believe it's a 'Buck stops here' concept for Dick. So when he notices a disreputable page, he knows to whom to talk in order to get it fixed.

            I'm moving this Issue into Done.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - K-T, Thanks for your comments and the earlier outline. Iteration it is. We can hash out the labels as we go along. Regarding the labels: Let's work on a word change here. Instead of 'controlled' what feeling do you get about 'approved'? It's not so rigid. The idea is to let folks know that the imprimatur of some body has looked over and agreed with the contents/intent of the page. The label bestows slightly more weigh, by virtue of the review status, than a random page written by a developer. Think about your trust level for random web pages vs web pages on a respected site. I agree that the Project controlled documents need to be archived in Docushare (or follow-on). But we have many documents which don't rise to that exacting standard yet inform the Developers about management's expectations on them. For example,the TCT decided some years back to allow upreving of selected 3rd party products to occur on a semi-annual period without reapproval by the TCT (if they first verified the uprev worked,etc). This is not an LDM controlled rule but an internal DM Policy. (Under the new division of labor, it might now be a SAT Policy instead of a TCT Policy.) Regarding 'living', 'versioned', 'archived' - I think we only need one designating 'versioned' to indicate that a particular page is anchored to a specific Release. We wouldn't need to designate 'versioned' on the various versioned Spaces we have; that would be redundant. Dick should respond with his sentiments on this, too. I view 'curator' as the person who is responsible for periodically looking over their document fiefdom for obsolete pages which either should be updated or removed. Confluence has nice searching facilities for searching on page labels. However, I wouldn't want to inhibit anyone who was willing to update a page's content on their own. I believe it's a 'Buck stops here' concept for Dick. So when he notices a disreputable page, he knows to whom to talk in order to get it fixed. I'm moving this Issue into Done.

              People

              Assignee:
              robyn Robyn Allsman [X] (Inactive)
              Reporter:
              ktl Kian-Tat Lim
              Reviewers:
              Kian-Tat Lim
              Watchers:
               
              Votes:
              2 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.