Location:This JIRA issue
To better facilitate DM communications and community support, and following suggestions and input from DM members, we propose adoption of the Discourse forum platform. A demonstration forum is at http://community.lsst.org and is available right now for anyone to create an account and use. This RFC is to determine if the demonstration forum at community.lsst.org should be designated as an official DM communications channel.
Online communications underpin the efficiency of DM due to our geographical distribution and size. To date we use a myriad of tools to address various niches: JIRA, Confluence, Blue Jeans/Google Hangouts, HipChat, email lists, and to a minor extent, GitHub.
We have identified real pain-points with the current toolset that will likely get worse as the project evolves.
- Our primary communications channel, HipChat, is built around a gatekeeper model. This is a real impediment to communicating with people on the fringe of our collaboration. As a recent example, recall the discussions of how to put DES astronomers in contact with DM on HipChat. There was a real cost (money per user) that complicated this matter. This will only get worse. As the project grows, more stack users will want to be in contact with DM. We want to build a community, but that community won't fit on HipChat.
- Technical support won't scale on HipChat. When a new DM user or developer joins the fold, our admitted issues with documentation necessitates that they get real-time tech support on HipChat to get up an running. This works on a small scale, but as hiring ramps up, this type of repetitive tech support will become a drain on DM developers.
- There is an unhealthy fear of missing out on HipChat. Important, useful and consequential conversations happen on HipChat everyday, but if you're not in a HipChat room at that moment you're either not part of a decision making process, or worse, you're not aware of what's going on. We do have a policy where important decisions are summarized on `dm-devel`, but this is not always followed. The real-time stream design of HipChat makes it hard for multiple conversations to happen at the same time, or for conversations on the same topic that happen sporadically over several days to be linked together. HipChat can be searched, but search is often difficult with chat logs.
- Email lists have terrible user experience. Email lists are okay for real-time communications, but their archives are often useless. See https://lists.lsst.org/mailman/private/dm-devel/. There is no way to search the entire archive without browsing each month. (Mailman encourages people to use Google), which is not an ideal UX either) This becomes an impediment to new users who want to be responsible and search for an answer before creating new and potentially repetitive traffic on a mailing list.
- The information architecture of our confluence site is ill considered. A new hire or user needs to do a lot of spelunking to find information. For example, the Confluence Questions site is only visible to those logged into Confluence. This makes no sense. Information that is hidden behind a login wall or in a wiki page are often neglected.
Our proposed solution: community.lsst.org
community.lsst.org is built on the Discourse.org forum platform. Discourse is open source and built by the same team who brought us StackOverflow. We firmly believe this is the best tool that will foster the best content and experience.
We believe that community.lsst.org will absorb traffic some from HipChat, Confluence and the email lists, though this RFC is not to deprecate those services at this time. The technical features of the Discourse platform are:
- Discourse will be hosted by discourse.org, automatically giving us their CDN and security expertise. community.lsst.org will be far faster and more reliable than both HipChat and Confluence.
- Anyone can create an account, at any time, without any need for DM action. When someone downloads our open source stack and needs help, they can instantly become part of our community and get in touch, or just read the archives.
- Information is easy to find, whether it is being created now or was created months or years ago. Conversations are organized into topics that have useful titles. These topics can be tagged and are also organized into categories. It is trivial for a new user to simply browse the site by category or tag to get up to speed. Discourse is proactive in linking information by showing suggested topics and showing titles of linked topics in sidebars. Discourse also has great search.
- Discourse provides smart and user-customizable notifications. Anyone can watch/track categories or topics to be notified when there is new content. User '@' mentions can also draw people into conversations, like on GitHub or Twitter. It is also possible to subscribe via RSS to new topics. Finally, Discourse can email an activity digest to anyone who has not logged on recently. There should be no fear of missing out on community.lsst.org.
- Discourse can replace email lists, as outlined in this help document: http://community.lsst.org/t/how-to-subscribe-to-emails-of-all-new-posts-in-categories-or-tags/37?u=jsick
- Discourse is an effective Q&A platform. We've created a Q&A category and enabled an 'accepted answer plugin.' When a question is answered, either the original poster or a moderator can mark the answer as accepted. This makes it easy for new users to find answers without any uncertainly.
- Discourse allows posts to be marked as 'wikis,' making it easy for members to curate how-to guides. Regular posts can also be edited by very trusted users and moderators. Change logs and diffs are easily available to anyone.
- Discourse can be navigated almost entirely with keyboard shortcuts. It is made for power users.
- The Discourse community will largely moderate itself. New users are given limited powers that will prevent spam. Users can also flag posts to alert moderators of problematic content or users. Five flags on a post automatically hide a post until it can be managed by a moderator.
- Discourse allows for private messaging either on a 1-1 or group basis.
- Discourse allows for permissions to be set on a per category basis. Groups can be granted 'private' categories as needed.
- It is easy to write beautiful content on a Discourse site. Markdown text and LaTeX math are allowed. Images can be dragged and dropped into a post.
- Discourse allows us to set explicitly community guidelines, terms of service, privacy policies and content licensing policies. We suggest content on community.lsst.org be permissively licensed CC-NC-SA.
We anticipate that over time a corpus of useful information will grow on community.lsst.org, particularly in the Q&A category. Just as StackOverflow is immensely useful to those who only search but don't ask questions, we expect the same of our community. community.lsst.org will nicely complement our more formal documentation efforts.
We also anticipate that this community will scale gracefully. Besides serving the needs of DM personnel, this community will serve the astronomy community as a whole in adopting and using our software and data. Although it is outside the scope of this RFC, we foresee that other groups within LSST, such as the science collaborations, may choose to participate in community.lsst.org. Thanks to Discourse's categories and user groups, this scope evolution can be gracefully accommodated.
I don't see anything here where Discourse is different than, for example, people using Dropbox or Google Docs to share information in a non-public setting - or, in fact Hipchat private rooms. If I am missing something, please let me know.
To return to the sub-thread about retiring some other forms of communication:
I support retiring mailing lists as Discourse is brought in - definitely dm-devel and dm-users. I'm not as certain about dm-announce and we should take some care with dm-staff. dm-announce was meant to be very low traffic but visible, perhaps, to a very large audience. This still seems like a good use of a mailing list.
dm-staff has a quite different function from the other lists. The others are essentially voluntary associations, or ones associated with particular responsibilities. dm-staff was, I think, meant to be a channel that would be sure to reach all the team members, and only them. To move this to Discourse requires a implementation that assures that this idea remains intact.
I'd move the others first, and then let Jonathan Sick and others think through later how best to replicate dm-staff's function.
Just a reminder to everyone watching that this RFC will close very soon.
Thanks everyone for your comments. It seems that we're in consensus that community.lsst.org is a good thing with the caveat that we should deprecate the DM mailing lists that are used for back-and-forth discussion, namely dm-devel and dm-user. dm-announce and dm-staff will be remain, but I'll work on forwarding public announcements to dm-announce and forwarding traffic from DM Team or DM Notifications to dm-staff.
The issue surrounding Discourse's threading seems to be resolved thanks to clarification about how the forum was intended to work.
I also acknowledge that some guidelines may need to be written about what content should be on HipChat vs Discourse, though I feel that this can be shaped more organically than through RFC.
If you have any small issues about the forum (functionality, the setup of categories, etc), you can voice those to me as I will be allocating some effort to improving and maintaining the forum.
If you have any major objections; now is a good time to voice them.
This RFC has been accepted; community.lsst.org is an official DM communication platform with the following notes:
- Hosting will transitioned to the Civilized Discourse Construction Kit, Inc.. At that time we will gain their SSL support, CDN network, and security monitoring, among other services,
- We will deprecate the dm-devel and dm-users email lists in favour of the forum,
- We will arrange for Announcement posts to be forwarded to dm-announce,
- We will arrange for DM Team and DM Notifications posts to be forwarded to dm-staff.
As with all developer support work that SQuaRE does, we will always be receptive to feedback about community.lsst.org. If a consensus forms, after real usage, that community.lsst.org is unsuitable for internal team discussions, then we will be happy to entertain and investigate other platforms.
If and when Discourse becomes a general internal developer and external user support tool, I could see discussions being posted to it that would relate to and potentially expose some amounts of "internal" data such as:
If Discourse stays around into Operations, we'd also have questions and discussions about "protected user" data such as:
I believe that protecting such discussions with the private category and private message Discourse features meets the requirements of the Information Security Policy for managing authorization to view "internal" or even "protected user" information, and we should inform all Discourse users of these controls.
If adopted, Discourse should of course be listed in the Project Management Office Information Security Plan or the Data Management Information Security Plan.