Talk pages consultation 2019/Staff working group notes

November 8, 2018

Attending: Danny Horn, Jason Linehan, Marshall Miller, Quiddity, Trevor Bolliger, Whatamidoing (WMF), Ryan Kaldari, Adam Baso, Joe Matazzoni, Rita Ho, Gergo Tisza, Stephane Bisson, Amir Aharoni, Corey Floyd

Goals:

Introduce the project, goals and approach
Ask and answer questions
Generate more questions and next steps

Introduction: Alchemy

Danny: Alchemy is an ancient protoscientific tradition, which people commonly think of as people trying to turn lead into gold. The thing that people don't understand about alchemy is that they weren't trying to get rich; it wasn't a money-making operation. The alchemists wanted to turn base lead into pure gold because they thought it would purify their souls, taking flawed people and making them spiritually pure.

The "Talk pages consultation" project is an alchemical process. It's about making a better communication tool, and it's also about making us into better people -- more collaborative, more curious, more sincere and more trustworthy. This process is only going to work if we hold ourselves to this higher standard.

Now, historically, alchemy led to three things -- failure, medicine and gunpowder. I'm hoping that we end up with medicine, but you never know.

The ultimate goal is to produce functional conversation tools that work for all the volunteers.

Marshall: This is not just about “fixing Talkpages”, but really how to improve communication in general.

Corey: +1 prefer broadening the concept to "Communication".

Flow strategy

Danny: The strategy was to enable Flow on the smaller and mid-sized wikis. Improve it on wikis where people want it, in the hope that it would become so good that everyone (e.g., established editors at enwiki and dewiki) wanted it. This strategy is not working.

Two problems in Flow's early development that hurt its acceptance among established editors:

The design and development was done in office, with some users involved but not in public. For many users, it appeared out of nowhere, with a "big reveal" at Wikimania.
It was supposed to be one tool that fit every use case, but it clearly didn't, the way it was designed. People saw it as something that had to be stopped at all costs, before it bulldozed all of their workflows.

The 2019 talk pages consultation

When: Starting in February 2019. Likely to require multiple years to develop any products.

Who we want to reach, in addition to the existing core communities members:

Volunteers who work at Teahouse and other new-user forums
- We'd like to hear from newcomers themselves, but that group has historically been difficult to reach, so we are hoping that established volunteers who work with them will represent the needs of newcomers.
Movement organizers, e.g., people who organize edit-a-thons
Editors who use off-wiki communication systems, such as IRC, Facebook, or Whatsapp, in preference to existing talk pages
People from as many languages as possible
People who use mobile devices
Contributors on the non-Wikipedia sister projects, including Commons, Wikisource and Wikidata

Open and participatory product process.

The central project page will be on MediaWiki.org, but this will inevitably involve discussions on other wiki pages and off wiki.

Traceability of decision points. Discussions will be in public and traceable. We'll post notes from any offline discussions (including this meeting).

Possible outcomes: All options are on the table. Editors might need multiple, separate solutions. For example, huge complicated discussions might need a different solution from one-on-one conversations.

Corey: Software-wise, the goals are consistent with some old work: threads, topics, summaries, etc. Are we saying we might throw the existing stuff away?

Danny: We could build something on top of existing talk pages. We could build something new. We could build something with parts of Flow, or turn Flow into something. One idea that's appealing to me is that we might find we need different tools to match different use cases.

Roan: I feel good about redoing the software. It's got some bad code, and it wasn't designed with incrementalism as a priority. We do need to be able to migrate Flow/LQT discussions to anything new, but everything else is possible.

Product process

1). Define the problem you want to solve

What are the problems with existing talk pages that we want to solve?
What are the good features of talk pages that we want to keep?

2). Important use cases

Researching the existing off-wiki channels might produce useful information about desired features.

Kaldari: The key is understanding the use cases. It will become obvious what the solutions should or could be.

Rita: Looking at how other wikis are using offwiki comms channels, we should research how effective they find using it. Opportunity to also connect and speak with newer editors via these places (especially for user testing of whatever it is we end up trying).

Quiddity: Partial semi-related list at bottom of m:Social media plugins#Solutions

Danny: We're not going to be better than Facebook at getting people to talk, but we can definitely learn from them and be better than we are.

Corey: Are we planning more market research? We don't necessarily want to invent a brand-new thing. We should learn about existing external platforms.

Rita: +1 not just Market research, but if we could also do user research or pull data on how these off-wiki comms channels have impacted contribution and communication on the language wikis that are actively using these other tools (e.g., the Arabic Wikipedia, which uses Twitter, Facebook, and Instagram).

Corey: One difficulty of anyone trying to communicate is finding all the existing comms channels.

Communication has immediate components ("Let's do this today") as well as long-term documentation benefits ("Why did they do that, last year?").

Marshall: Social media thoughts: People aren't just using social media because regular talk pages are difficult, but also because "that's where people already are". Maybe in the far-future there's some way to take advantage of somehow integrating, or leveraging the fact these systems and communities exist, as opposed to trying to compete with them. Don't write off any options.

Amir: Historic search is crucial part of why onwiki is key.

Danny: Consultation steps will be making lists of good and bad points of existing talk pages.

Kaldari: Talk pages are not just for discussions, they are also repositories of information. Making things searchable, archivable. Workflows for facilitating other workflows. We might end up with various tools, with parts that are needed in different situations.

Joe: Possibly, some things used on talk pages make move to other systems (e.g. WikiProject categorization). Right now WikiProjects and other classifications are on the talk page. Is that a good thing?

Roan: No, it’s a bad thing! It breaks things like searching for content using WikiProject categories, because you’re searching the talk pages instead.

WhatamIdoing: There's been a proposal for years to move WikiProject categories and similar data into real metadata. There’s a trade-off with temporary pain at the larger wikis as some tools (such as the ArticleAlerts bot at enwiki) will need to be rewritten, but some of these changes will be better for everybody in the end.

David: Wikidata puts Property documentation on the talk page: https://www.wikidata.org/wiki/Property_talk:P18#Documentation

Trevor: Relationship between on-wiki and off-wiki is always difficult for anti-harassment. Whatever we build, day 1, should have a report/take-down/moderation feature(s), absolutely critical.

Stephane: Past research on existing uses of talk pages shows significant diversity.

Quiddity: See Flow/Index#Workflows and other research and especially Flow/Use cases.

Marshall: We can start with listing what people use talk pages for, those can turn into user stories. For example, people use talk pages to display the status of their project, but that doesn’t fundamentally require talk pages.

Danny: I want the communities to tell us what they need from talk pages.

Amir: One area we can learn from is analyzing which templates are frequently used on talk pages, commonly not the same as the ones used on articles. I don’t know how much template use was analyzed when designing Flow – I think at least some on some wikis, but not very comprehensively.

Quiddity: The idea for Flow was initially for it to be a modular system that the communities could use to build workflows. It would have semi-standard components for cross-wiki familiarity and maintenance, but with local adjustments for their particular setups/needs. Smaller wikis could recreate the complex systems that large wikis have, and large wikis could upgrade from community-maintained custom and duplicative tools to something WMF-supported and easier/faster for non-technical people to use and build upon. Small wikis wouldn’t need a technical person, because they would use a wizard to create a wizard. The blocker on enwiki was that it was never gonna be accepted until all workflows that were already taking place on talk pages worked in it.

3). Talk about possible solutions & tradeoffs

For example, an existing solution might use filling in a template or a category, but that could be replaced by a built-in specialty tool. Does everything need to be supported from the beginning, or could a partial approach improve communication now, and more features be added later?

Joe: this sounds like a really big project that would unfold over multiple years. Just maintaining Flow, without adding any new features, took up like a third of a development team’s bandwidth. Because it's an alternative wiki system, it needs to be made to work with all existing systems.

Roan: This is like building the Sagrada Familia cathedral. People are still building that cathedral, but they're using it now. One problem with Flow was it didn't have a plan compatible with incrementalism. Like boiling the ocean: none of it will start boiling until reached boiling point. Flow had to reach all use-cases before it could be welcomed by some communities.

Joe: Thinking about incrementalism, one of the types of things we could think about is Flow modules that can be inserted into existing wikitext talkpages. A hybrid approach.

Gergo: There are other communication channels wiki editors use (IRC, email) to cover different use cases like real-time communication. There is no inherent reason those have to happen outside the wiki, they just do because current talk pages are so hopelessly bad at supporting them. Are those use cases within scope?

Danny: As our communities scale up, it's like New York City: it's a ton of people, and they all have different interaction methods and different needs and priorities. They have their own ways of dealing with the deluge of information, in every office and street corner.

Do we need mostly small changes or mostly large changes?

Roan: Here's a concrete example of iterative way of fixing wikitext discussion: phab:T128535: make pings work with parser-function instead of template-based mentions. I hope this stokes your imagination about the things we could think of doing.

How much emphasis to put on intra-wiki vs inter-wiki communication?

Corey: We have all the different projects, everything is so separate and it's hard to get cross-wiki communication.

Quiddity: Cross-wiki topics was an original goal of Flow. Social complexities were the main reason it didn't get past the idea stage, i.e. how (each community would want) to handle the case of someone blocked on wiki A who tried to comment in a discussion that was shared between both wiki A and wiki B.

Corey: Content supported by groups like WikiProject Medicine would benefit from better cross-wiki communication support.

WhatamIdoing: If we had cross-wiki communication, we wouldn't have needed the Commons Deletion Notification Bot wishlist item. Commons deletion discussions could have been posted simultaneously to all affected wikis in full.

Things we need to figure out

Trevor: I'm thinking about the movement strategy. There were people where all they did was own the documentation process. (Tech writers?) Practically, how should we organize all this information?

Joe: We need dedicated notetakers and organizers and ambassadors and facilitators. Will there be contractor budget? Is there money for translation?

Danny: Maybe it will be possible to hire contractors. We would need a clear plan in order to ask for such budget.

WhatamIdoing: Will there be a specific product manager who will work on this full time?

Danny: I'm leading the consultation, hopefully with support from a number of people. When it comes to designing and experimenting with a new feature, it'll have a team, probably starting around July, if things are going well.

Marshall: Along lines of planning, good to consider upcoming community events where we can bring this up? All Hands?

Kaldari: Most of us don’t perform complex workflows like archiving discussions or AFD, we have to listen to those who do.

Next steps

Danny: Step 1 of consultation is clear: Getting lists of pros/cons from all the diverse stakeholders. Step 2 is finding possible directions. We need clearer ideas about how to structure Step 2. We can look at the Movement Strategy process, but how else?

Joe: Need to make lists of use-cases at same time as problems to solve.

Danny: Let's make a list of people to talk to, things to read, or questions to consider. Just quickly add whatever you think needs to be done.

(The following list was a "free-writing" exercise for the last five minutes of the meeting. People were asked to write down ideas for next steps, and questions to consider.)

Document existing workflows/use cases to get a full picture of the “problem”
Determine organization structure of documentation, decision making and traceability
Consider which scenarios for the use of discussion pages could be moved away from talk pages and unified (at least partly) among different projects (even though the policies may be different at the moment): Page deletion, Administrators appointment, RFC, etc.
Do usertesting.com tests with people who are new to wikis and ask them to use a talk page. This will generate clear indicators of where newcomers struggle with talk pages.
And check what we already did, e.g. https://www.mediawiki.org/wiki/Flow/Moderated_Testing,_November,_2014:_talk_pages_and_Flow
Meditate on the notion that some experienced editors will reject all innovation in any case :)
Brainstorm ideas for small interventions to the current talk pages in the meantime? E.g. turning {{ping}} into a parser function.
Make detailed list of all key stakeholders, how to reach them, and who will be main contact facilitators.
Post-mortem of Flow, to elucidate what we've learned from it so far.
Post-mortem of LQT, to detail what we learned from it back then.
Start getting the project pages translated and try to get into a rhythm of translating them regularly.
Staff can list some of our community contacts who would have good perspective on this from multiple communities.
Ask Audiences staff to list the ways in which changes to Talk pages or communication would affect their team’s features and work.
Move the page to MediaWiki.org (and disable Flow on its talk page). Done :)
Contextualise how this project would fit into existing 3-5yr planning and annual plan outcomes – this would be helpful in determining resourcing and assigning to it to a properly staffed project team
Decide on a mobile-first policy for all design and development around this project.
Consider making good mobile support one of the central goals for the project.
Draft a design/product planning document outlining the goals, use cases, and users for whom we want this new “Fix Talk Pages/Improve onwiki communications” project to solve. Something we can share with the communities relatively early so they can collaborate on it (e.g., help identify use cases we’ve missed).
Propose as priority topics for the Design Research Strategy team (Margeigh et al) to research - items listed above such as: post-mortems on past projects, user testing, reviewing off-wiki comms usage on other language wikis, etc.
Re-analyze the difficulty of importing existing wikitext talkpages into a half-structured system. - The prospect of Tabula Rasa was a nightmare for the larger wikis. Even if it's only "1 thread into 1 comment", that's better than nothing.

Closing

Danny: It's really nice to see everyone being optimistic and excited about this project. It's going to be big and weird and difficult, and I'm really glad to see that people want to be involved.

November 29, 2018

Attending: Danny, Trevor, Quiddity, WhatamIdoing, Kosta Harlan, Alex Hollender, Marshall, Benoît Evellin, Erica Litrenta, Ryan Kaldari, Janna Layton (notes), Rita Ho, Gergo Tisza

Goals:

Discuss medium-term goals for the project
Define staffing for the project
Begin to plan documentation and project pages
Identify people we can partner with

Danny: Last meeting was an intro to the whole program and give the opportunity for questions. It was helpful and productive.

Medium-term goals

Danny: By the end of June, we are aiming for a definition of what we’re building. Are we adding something new to something that already exists, building something completely new, something specialized for large discussions, etc.? Then a product team and designers can work with that.

Trevor: Can we define “definition”? Does that include consensus with wider Wikimedia community?

Danny: Rough consensus (i.e., are we building one thing or two?), have enough consensus from enough people that we are confident we are proceeding in the right direction.

Marshall & Danny: Be able to define in one sentence what we're planning. Process will be very transparent.

Benoît: Wants more definition of “rough consensus,” when to move forward regardless? Do you plan to step in and say “we are responsible of the development and the maintenance”? Will we follow the 2030 strategy to prioritize some points over others, for instance?

Ryan: Having the majority of people in the community not hate the idea, fair degree of buy-in, we can plausibly say we have support.

WhatamIdoing: The perception of consensus is a slippery, changeable thing. VisualEditor, which was one of the main outcomes of the original 2009–2010 strategy process, in which hundreds (thousands?) of people participated, was not perceived by English Wikipedians as ever having had consensus in 2013.

Trevor: Also define what we’re not doing, use cases

Danny: Communication with community is more about getting information than buy-in, because we need the community input to make the right decision. This information and confidence in the right direction is what we want by the end of June. This is all hand-wavy and feelings-based.

Benoît: Will this project be aligned with WMF values and the strategy “knowledge equity” and the goal of having more people involved on wikis, especially in emerging communities?

Danny: Yes

Ryan & Danny: Balance between better function without killing what people like now

Alex: (continuing to think about “definition”, wasn’t said out-loud)... having the pros & cons of the solution defined, and having some kind of plan for how to test it as a hypothesis. Like, in June, are we saying here’s the solution, and we’re going to build it, or are we saying here’s our hypothesis and how we’re going to test it? We hope it’s right, but it’s more about setting up a framework for learning (or maybe that’s more what’s happening between now and then?).

Staffing

Danny has talked to Toby and Maggie, and there is some funding for this. Danny has a list of functions we need someone to do. For this list, sessions = conversations, phases = questions answered

We need multiple people to…

Define the framework for documentation (Danny: documentation is hugely important)
Organize and update documentation
Document existing workflows and use cases
Define stakeholders, how to reach them
Draft a design/product planning document that we can share early on with communities
Organize translation, and outreach to various languages
Lead retrospectives on wikitext talk pages, Flow and LiquidThreads, summarize and discuss with stakeholders. What do people like and dislike about each?
Comparative study of other systems (Wikia, Reddit, Quora, Stack Overflow, etc.)
Know why some people/communities use alternate discussion tools (Facebook, Telegram, Google Docs…)

+1 : ‘Competitor/comparative’ research on other tools is something proposed that we could ask Design Strategy Research to work on.

An aspect of the research could be to evaluate how effective these other off-wiki comms channels have been for other language wikis (increase in new editors, contributions, etc.).
Know the impact of off-wiki interactions on the on-wiki life
Collect community-written viewpoints
Collect non-community members viewpoints
Collect newcomers' viewpoints
Collect and synthesize recent research on talk pages -- substantial research was done a few years ago, when designing Flow.
Some user testing of existing solutions?
Define and lead user research
How new users understand/use talk pages on desktop + mobile web + mobile apps
Define the structure of sessions and phases, how they fit together
Schedule and coordinate sessions
Follow-up after sessions
Summarize sessions, discuss summaries
Track activity
Respond to questions on wiki, etc.
Outreach on social media

(The above are multiple people, not one role.)

WhatamIdoing: There is a long history of off-wiki communication, ranging from everyday help questions on IRC to coordinated harassment campaigns. At the English Wikipedia, the problems eventually resulted in rules on consensus-forming discussions about article content being held on wiki and rules about off-wiki harassment, but other communities currently take other approaches, and all of them took other approaches in Wikipedia's early days.

Benoît: We need to know why people use non-wiki discussion tools.

Danny: this discussion will need to be part of our sessions

Trevor: It is a fool’s errand to think that once this is made, no one will use non-wiki tools for wiki projects discussion ever again. We’re not building social chat space.

Erica: Question about having people assisting in community discussions, like it has been done for Strategy. There is funding, but getting people enrolled takes 2 to 3 months. It is compatible with your timeline?

Danny: IDK.

Danny: Toby suggested some of the people already trained might be able to jump in.

Erica (in writing): they may, but still at a cost. This process needs to start ASAP if you want people with us by March.

Kosta: In the comparative review, reference to Google Docs. Are we looking at in-editor communication? Or are we just looking at Talk Pages?

Danny: We can look at this.

Nick: Workflows – original idea was that Flow would be used for workflows – which primarily take place on discussion pages – replacing/upgrading some of the bots+categories+templates systems that are built by a few communities. Hence the first module built for Flow was for 'discussion'. Are we still thinking about aiming for this?

Danny: Yes, we are looking at this. We need to look at the use cases, metadata, address all things people use Talk Pages for.

Rita: Link to Notifications, esp. expectations when posting on mobile to receiving push notifications to devices

Gergo: Will we be aiming to build one thing that works for all communities? WMF focus is supporting emerging communities, who might have different needs. But the folks who turn up for the consultation are statistically likely to be primarily from the largest wikis. How can we ensure that emerging communities have a voice?

Danny: We cannot ignore either side of this. We can't build a feature that every wiki uses except for the three biggest wikis. That being said, we're going to have to work extra hard to make sure that emerging communities are an important part of the consultation. We might build differently for different scale communities.

Danny: Is there something else we need somebody to do?

Ryan: Comparative review of non-WMF discussion systems

Benoît: Will we do user testing of these?

Benoît (precision in the notes): there are websites where you can find people to test something. It may help.

Nick: There was some earlier research on wikitext talkpages and Flow and also https://www.mediawiki.org/wiki/Flow/Research/User_Test_Data and also https://meta.wikimedia.org/wiki/Collaboration/Flow_satisfaction_survey/Report (and a few more in https://www.mediawiki.org/wiki/Flow/Index ).

Trevor: one difficulty is how we engage for those who are not yet using these systems.

Trevor: We would need to check with Legal re: the whole project, need to be proactive with that

Rita: the Design team has used Usertesting.com for evaluation, as well as recruiting via Wikipedia’s Social Media for less active editors/community members to participate

Erica: Need to state which tasks happen when and their duration. Try to organize list based on what comes first, and what happens throughout the project. Proper user research, for instance, takes a long time, and we shouldn’t start discussion without having any research. Also, do we need external help or are we sharing these tasks among ourselves? Do we have an idea of how many people can be hired?

Danny: next step is to write a written plan to share with team and Toby and Maggie

Nick: +1 to Erica's comment about needing more research summarized before major discussion begins.

Erica: Of course the community consultation in a sense is a research result for us :)

Ryan: Community consultation doesn’t have to be “what should we do?” We will have research that doesn’t involve community having to understand entire project.

Danny: Whole project is a research project

Ryan: Half our project is for people to become experts in how our community communicates, how Talk Pages are used

Erica: “What should we do?” is the usual question Erica had in mind for community ;) Using a different angle is fine, but then we channel quite heavily the conversation if so, and so it is not as open as we initially said?

Rita: Useful to see comparison of power users versus newbies perception of Talk Pages, Abbey recently did this with Mobile. Social media has been useful for getting newbies’ ideas

Danny: Hope that this will be useful in getting those who want to contribute to Wiki but hasn’t yet

Benoît: Those who have made a few edits and then feel blocked by communication

Danny: work made by Growth team is interesting about that aspect.

Danny: Starting to get idea about what kind of people we need for different tasks

Marshall: Growth Team knows that asking questions for newbies is difficult. The goal of the Help panel is to provide access to facilitate posting on Help desks even though using talk pages is difficult.

Benoît: it is a shortcut to post a message, not an alternate way to discuss.

Ryan: See how discussions work on the Teahouse, which is a mix of software, gadgets

Danny: there are several places like that across Wikipedias, like the Forum des nouveaux on French.

Marshall: some communities use Flow for help desks, even though they don’t use Flow for anything else.

Gergo: other collaborative projects like reddit and stackoverflow also have done extensive research on discussion systems (stackoverflow has been fighting their own editor decline problem for a while). It would be great to involve people who have significant experience with those systems.

Ryan: Do we want a system where there are multiple views of the same discussion possible? Different interface system for different level of users?

Danny: Like “training wheels”?

Trevor: This is going to have to work for both desktop and mobile. Need to separate structure from presentation.

Danny: Idea that I’m workshopping: We’re going to need to do a retrospective and post-mortem on Flow and LiquidThreads, and it’s going to be hard to have a single, neutral version of what happened in that story without re-fighting those fights. Quim suggested: the WMF’s version and allow other people’s versions, allow people to say what they want to say, even if tirade, this will be part of archive

Trevor: People want to be heard.

Benoît: Wikitext Talk Pages have their issues too.

WhatamIdoing: Will this review of systems that we (volunteers) have used include the offwiki discussion systems that were the primary discussion systems in the early years of what are now the big wikis, e.g., mailing lists and IRC?

Danny: What the end result of the post-mortem will be

Kosta: Retrospective should include technical side?

Erica (in writing) I am curious at to what the expected outcome here should be. Facts matter a lot to us, and that's why we should be able to come up with the reconciled WMF side of the story. It sounds like we don't believe that a neutral history is ever possible, and we think that dozens personal stories are going to be effective in replacing that. I don't see how.

Danny (in writing, response to Erica): I may be overthinking this aspect. I'm a little worried that a post-mortem on Flow & LiquidThreads will mean that people end up re-fighting old fights, and then that drags down the whole project. It'll probably be fine.

Danny: will take these ideas and incorporate into plan. Are there people here in this room who want to work with Danny on how we structure documentation, framework.

Trevor, Nick, Benoît volunteer

Danny: Thank you!