Topic on Talk:Growth/Positive reinforcement/2023

Hello @HLHJ @Mike Peel @Czar @John Cummings @T Cells @Pigsonthewing @Sdkb @John Broughton @Pelagic @NickK @LittlePuppers @Zoozaz1 @Nick Moyes @Galendalia @Barkeep49 @NGC 54 @ValeJappo @Ата --

I wanted to invite you all to a discussion on the feature set the Growth team is working on next. This "positive reinforcement" effort is our next big project following "add an image", which many of you helped guide. The idea behind "positive reinforcement" is that newcomer tasks (like "add a link" and "add an image") are doing a good job of helping newcomers make their first edits -- but they aren't having a clear impact on getting newcomers to come back another time to make more edits. In other words, they are helping with "activation", but not with "retention". We think the "positive reinforcement" project could increase retention by showing newcomers the impact they are having and encouraging them to continue. Also, when we discussed newcomer tasks with communities, many of you wanted to make sure that newcomers wouldn't get stuck doing easy tasks, and that they would progress to more valuable kinds of tasks. With positive reinforcement, we can encourage that progression. But we know that the wikis are not like other apps and website -- we don't want to motivate people to make bad edits just to rack up points and awards. So we definitely need community input to figure out how to motivate newcomers in the right way that fits our communities.

I also want to tell you about an important transition and introduce you to @KStoller-WMF. I'm moving into a new role at WMF as the manager over the Growth, Editing, and Web teams. Kirsten will be the new product manager for the Growth team, and we're excited about her background as a librarian and as a product manager for a website that also has an active community. Going forward, you'll mostly be interacting with Kirsten around Growth product work, but I will still be working closely with her on strategy and tying projects together across multiple teams. Thank you all for your help building the Growth team features over the previous years, and Kirsten and I are looking forward to continuing to learn from you as we keep making the wikis a better place for newcomers!

Back to positive reinforcement: it would be great if you can check out the background and some of the design ideas on the project page, and post any of your thoughts, concerns, and questions in this thread or in a new one. Here are some prompts to think about:

• What has worked well on your wiki for motivating newcomers?
• Which of the ideas on the project page do you think have the most promise? The least promise?
• What could go wrong and what should we try to avoid?

I'm fairly certain that the WMF tried, maybe five or so years ago, thanking new contributors (on their User Talk page, I think). I believe this was done randomly, and I think the conclusion was (a bit more nuanced that this, but basically) that it made no difference. So, if my memory is right [sorry, unable to find a page to point to, regarding that effort/project], personal praise isn't a magic bullet.

Thanks for the feedback @John Broughton! I’m not sure which study you are thinking of, but you are right, personalized praise definitely won’t be a magic bullet. But based on several past studies we think it’s reasonable to expect this idea to be a step in the right direction.  

Here are a few of studies we looked at that found that newcomers’ editing activity is increased (at least in the short term) when they receive thanks and other positive messages about their work from an experienced editor or mentor.  Here are few of the studies we looked at:

• WMF Research:Understanding_thanks  
• Citizen and Tech study: Effects of Saying Thanks on Wikipedia
• Zhu, Haiyi & Zhang, Amy & He, Jiping & Kraut, Robert & Kittur, Aniket. (2013). Effects of peer feedback on contribution: A field experiment in Wikipedia.

I think one of the major challenges this idea faces is that reviewing and thanking newcomers requires time from contributors. We are still brainstorming ways to surface praise-worthy edits to mentors or other contributors. Do you have any feedback? How do you think we should flag contributions that might be worthy of thanks?

Hello Kirsten - always good to have more library folk on board! And good luck in your new role, Marshall.

We show editors a "your edit has been saved" notice each time that happens. Perhaps for new accounts, we could mae use of that? Something like "Your edit has been saved - you're doing great work - why not do another one"?

Thanks for the warm welcome, @Pigsonthewing!  

Nice, it sounds like we are thinking along the same lines as you.  Here’s a mockup of one idea we are considering, which displays an encouraging prompt after a newcomer edit is saved. Is this similar to what you are thinking? Do you have any ideas for improving this idea?

Generally yes, but I'm not so keen on the "1/5", "5/5", "daily goal" part. Is that goal set by the user? We shouldn't be discouraging people who might otherwise make more than five edits, nor encouraging people to make unnecessary edits just to hit an arbitrary target.

Great, thanks for the feedback! I also have some of the same hesitations regarding the daily goal. I assume allowing users to set and customize their own goals would likely be more motivating. Customization would also add to the complexity of implementation, but it's certainly worth considering. We'll keep this in mind as we take another look at the designs.

As the project page says, "We believe that these reinforcements from real people would be more effective than automated ones coming from the system."

A general suggestion: I think it's worth learning from other sites how they do it. I think that often involves, e.g., an email a day later saying thank you for your first contribution, have you considered coming back to do more, and here's some suggestions? This is perhaps more what newbies will expect, and less what experienced Wikimedians would want? But I'd also suggest contacting people that work actively on this (I don't), I know @WereSpielChequers has worked on this, and there's lots of on-wiki projects that would have relevant expertise (new page patrol, Teahouse, etc.) - and always good to ask new editors who're just past the first edit stage, what kept them going?

Thanks Mike, I spent a lot of time on Quora in recent years, and there there was much more automated feedback on things like the number of people who'd read what you'd written. Obviously not worth mentioning if figures are low, but good to send people feedback if what they've done is read by a lot of people, we do this with the DYK process but could do it more widely. As far as Wikipedia is concerned, I think there is a big potential in tailored invites - people's first edits usually indicate some of their interests, and we could auto generate invites from relevant WikiProjects. Even if the WikiProject concerned is not currently active, it is likely to contain a bunch of resources and list some relevant and potentially tempting tasks.

Thanks, @Mike Peel and @WereSpielChequers. Agreed, it's fairly standard to receive a series of welcome emails when joining a new site. We've recently started some work in that direction, and plan to closely measure the impact: Engagement emails

One of the Positive Reinforcement ideas we are discussing includes Personalized Praise which could result in an email depending on the implementation method we pursue (along with user preferences and if the new user has an email address associated with their account). I think the main challenge we will face with that project is finding ways to encourage mentors and other contributors to spend time providing praise to newcomers who make high-quality initial edits. How do you think we should flag contributions from newcomers that might be worthy of thanks?

I think there is a big potential in tailored invites - people's first edits usually indicate some of their interests, and we could auto generate invites from relevant WikiProjects.

Currently the Growth team is using the Welcome survey to provide some personalization on the newcomer homepage. But certainly there is more we could do here. I love the idea of connecting users to the topics and WikiProjects that would appeal to them. Newcomers have so much to learn and read when they get started, it's easy for them to get lost and never find the right project or task that appeals to them. I wonder if we could use the topics a newcomer selects as interests as a way to also suggest a WikiProject that relates to that topic. Is that what you were thinking?

Do you have any other feedback or concerns about the Positive Reinforcement ideas and designs?

Hi KStoller, I was thinking in terms of the Wikiprojects tagged on talkpages of articles newbies have edited or that are relevant to categories those articles are in. This has the drawback that some tags will be completely irrelevant to the user, however transparancy could be maintained if you send hem an invite that contains links to all the relevant Wikiprojects - they can then hone in on the ones they think are relevant to them. Hi <insert name hare> based on your first ten edits, we thought you might be interested in some of the following Wikiprojects Vulcanology, mountaineering, Washington state and skiing - WikiProjects are groups of Wikipedians like you who have an interest in a certain area. OK they may have edited that article on a mountain because of the botany or abandoned radar base, but they will have picked up the principle that Wikiprojects exist and there might be ones of interest to them. As for encouraging praise of newbies, I'd add some software to the welcoming process, enable those who welcome newbies to get feedback on those newbies such as their 100th or 500th edit, with an easy opt out such as these are newbies who you welcomed and whose userpage you still watch. You could also create a page on WikiProjects that enables members to give thanks to new editors who have begun to be active in that area, though I suspect that has been covered by watchlisting etc. Perhaps a better route would be to send messages to active Wikipedians who haven't yet issued thanks or barnstars explaining how to do so now that you've done your thousandth edit, have you considered encouraging Wikipedians who you see doing good edits, either by using the thank button or giving them a barnstar? WereSpielChequers (talk) 09:07, 19 May 2022 (UTC)

Thank you for detailed feedback, @WereSpielChequers.

I agree, there is more we could do to personalize suggestions based on what a newcomer has edited. The newcomer homepage currently suggests edits based on topics the individual has selected as interests, but certainly there is more personalization we should consider as we further onboard newcomers.

And thanks for all of the ideas related to encouraging personalized praise of newcomers. As you've noted, there are definitely many different routes we could consider. Many of the ideas we have discussed involve mentors, but I like that your ideas aren't limited to just that small (and likely already very busy) subset of contributors.

Thanks KStoller, this business of newbies selecting topics of interest is new to me. Unless we are talking about userpages, either it didn't exist in 2007 or I've forgotten about it. Can you tell me what proportion of new editors indicate such interests? If it is the great majority then this might be a more effective route than the one I detailed. WereSpielChequers (talk) 10:32, 20 May 2022 (UTC)

Hi @WereSpielChequers. This is a relatively new feature, so not something you would have encountered in 2007.

As part of the newcomer Homepage and newcomer tasks, we allow newcomers to select interests from about 40 topics. Here's an example screenshot. We then use that to personalize the articles / edits we suggest.

I'll have to dig into metrics to see how many newcomers complete this step of the onboarding funnel and select interests. Parsing the data is slightly complex since the percentage of new accounts which have the newcomer homepage and newcomer tasks enabled by default varies by wiki. But certainly for users who have self-selected interests, it's likely to be a fairly accurate way to understand which topics or WikiProjects they might find appealing.

If you are interested in checking out the newcomer Homepage, here's info in enabling the Homepage. And if you have any feedback, please let me know. Thanks again for the feedback about the Positive Reinforcement project!

Thanks, that's interesting. I see a disconnect in that forty topics is too few to match with the Wikiprojects. So Sports and Biography are in the forty, but Wikiprojects function more at the level of professional wrestling or Baseball. Sports is an umbrella. WereSpielChequers (talk) 00:14, 21 May 2022 (UTC)

You are totally right, the topics are very general, and the Wikiprojects can get very specific. It's also worth considering that the smaller and medium wikis will have fewer Wikiprojects than English. And I believe that some small wikis have no Wikiprojects, so it's perhaps not an idea that would work for all wikis.

But I still love the underlying idea: that as newcomers start advancing and trying more difficult tasks, eventually we should expose them to more advanced ways to contribute and involve themselves in the community. Learning to contribute as a newcomer feels like a labyrinth of technical, conceptual, and cultural challenges, and there's so much more we can do to help ease newcomers along the path to becoming successful contributors.

Sorry, yes I was thinking of English. Taking Georgian as an example, it is a few years since I went to a Tbilisi meetup, but they had a monthly focus project that a lot of them would participate on. If that's still their modus operandi, for that wiki you'd likely promote the idea of a monthly topic. What seems obvious to me, is that this requires asking the various wikis what process we should promote to newbies. Currently because of our software limitations we don't have autosign on talkpages, so we have to promote the idea of signing with four tildas in all welcomes to Newbies. Fix that problem and we have a lot of communication bandwidth that could be repurposed for a message more positive than "Our software is so klutzy you have to use this symbol you've never heard of to sign your posts".

Thanks for the feedback and example, @WereSpielChequers. Your Georgian example is in line with a concept the Growth team wants to focus on more: ensuring Growth features are extensible and customizable to better fit the unique needs of each wiki.

And you are right, manually signing talk pages is definitely not intuitive to newcomers. I know the Editing team has many Talk Page improvements in the works that should benefit newcomers. I really hope we see an improvement in new editor retention as we start to remove some of these common stumbling blocks and the Growth team works on features to ensure newcomers receive positive reinforcement along the way.

Thanks again for taking the time to chat through these ideas!

The Growth team completed user testing of Positive Reinforcement designs and wrapped up some other projects, and we are finally more focused on moving the Positive Reinforcement work forward! In an effort to share early and often, I wanted to post an update:

Check out the initial improvements to the Impact module in Beta. The Impact module (the "Your impact" section) is still incomplete and missing the article list, but you can see some of the initial work on adding in more personalized info and data visualization.

If anyone has any initial feedback, I would love to hear it! I'm especially interested in:

Should the Impact module's "Thanks received" count link the Thanks log? Example Thanks log for Trizek (WMF).

I think we would only want to do that if the Thanks number was more than zero. Our designer is concerned that the Thanks log is likely too confusing for new editors though. Which I think might be true, but also not too many new users actually get Thanked, and maybe once a newcomer is Thanked then learning about the Thanks log is OK?

Ok, thanks again for all the previous feedback and guidance, we are definitely continuing to use that feedback to decide how to iterate on designs and scope our work! And here's a shameless plug to watch the Growth Updates page if you are interested in Growth team work. I post updates most weeks on what the Growth team is working on. Thanks!

Hello @John Broughton, @Pigsonthewing, @Mike Peel, @WereSpielChequers @Sdkb @Barkeep49 @TheDJ @NGC 54

Thank you all for previously providing feedback on the Growth team's Positive Reinforcement plans!

Impact module comparison

If you are interested, you can test the first part of Positive Reinforcement: the new impact module. You can opt in and view the new impact module by adding &new-impact=1 to your homepage URL.

Example for English Wikipedia:

https://en.wikipedia.org/w/index.php?title=Special:Homepage&new-impact=1

We are releasing this change to Growth pilot wikis (Arabic, Czech, Bengali, and Spanish) and we will complete a month long A/B test comparing the old impact module to the new impact module. If results look promising, we will release to more wikis once we've also addressed any issues or pain points surfaced by our pilot wikis and anyone who chimes in here.

If you have a chance to test, let me know what you think. Do you think newcomers will benefit from the new layout? Do you notice any bugs or aspects of the feature that are confusing? Thanks!

@KStoller-WMF: It looks good to me, but it is annoying that some lines span on 2 rows (at least in Romanian). P.S. The new version says that I have made only 1,000 edits, but the old (current) version says the truth (36,754 edits).

Thanks for testing!

Yes I see what you mean about some of the translated text spanning 2 rows for certain browser window widths. It's tricky to find a perfect solution there due to translations varying in length and since the module adjusts depending on the window size.

Thanks for noticing the 1,000 edit issue, hopefully we can find a reasonable solution there. T324286

I like the idea, but, as always, things could be improved:

• On the first page, the "1 of 2" would be better placed to the right of page title
• Showing "1 of 7,216" (or "1 of [an even larger number]) seems unduly daunting. Why not just "1 of 50" or "1 of 100"?
• When the first article is shown, the right arrow works (goes to the second), but the left arrow doesn't. Can you hide the left arrow?
• Showing the email address at the top of the page is, at least for more experienced edits, weird. Communicating by email is strongly, strongly discouraged, because it isn't consistent with the "everything should happen on-wiki" philosophy of wiki transparency. It's also unnecessary, unless I'm missing something major here.
• I'd prefer that clicking on an article open that article in a new window or tab, at least if not in mobile mode. That way, when I'm finished editing, I can close the window or tab, and return to where I was.
• It would be nice to be able to sort by the number of visits in the past 60 days, and/ or the date when the article was marked as needing copyediting. Or, perhaps better, the software should use a combination of the two of them to prioritize the articles that are initially present. (More visits is better; an older flag is worse.)
• I'm not convinced that copy edits "help articles be more professional and trustworthy". I'd replace that with "make articles more readable, and - if done after checking with sources - more accurate."
• The information "i" at the top right has two subheadings, one of which is "Filter by topic" But what is being done is NOT filtering. The heading should be "Select by topic". Filtering **reduces** the number of items shown. What happens when the user selects an additional topic is that the number of items shown (listed) **increases**.

Thanks for the feedback, @John Broughton!

It looks like this feedback is mainly about the Suggested edits feed, not the "Your impact" module, but this is all very valuable feedback that I'll review with our designer. We hope to have time to continue to iterate and improve on existing features in the coming year, so I appreciate the thorough breakdown of improvement ideas!

Showing the email address at the top of the page is, at least for more experienced edits, weird.

This is prominent to encourage newcomers to add an email address (as a way to improve new editor retention). However, I can see how it might seem odd to continue to display that so prominently once an email address has been added.

When the first article is shown, the right arrow works (goes to the second), but the left arrow doesn't. Can you hide the left arrow?

I've noticed this too. The left arrow is lighter to show that it's "greyed out" and inactive, but I don't think the contrast difference between those two arrows is enough to make that obvious. I'll chat with our designer about this and the other ideas you've mentioned. Thanks again!

Yeah the upper right panel is clearly not working right. For instance I have definitely had a longer streak than the 6 days it says. That said I think the layout is strong. I look forward to seeing what data is gained from the pilot wikis but I am hopeful.

Thanks for taking a look, @Barkeep49! We've definitely made some compromises in this initial version to ensure the data can be computed efficiently. Each section is only evaluating the last 1,000 edits: which works great for newcomers, but is problematic for established editors.

I'll work with the team to see if we can fix that Total edit count T324286.

And I think we'll need to add in some clarification to the "Longest streak" that for editors with over 1,000 edits, that the Longest streak will only reflect the last 1,000 edits. We could increase that number, but the larger the number the more likely we will run into performance issues.

Thanks again for taking a look and for the quick feedback! I'll be sure to follow up in this thread again once we have some data from Growth pilot wikis.

Is it only looking at mainspace edits to calculate those numbers?

@Barkeep49 Correct, we are limiting the metrics to what can be derived from the user's last 1,000 mainspace edits.

Overall, the module looks good!

Thanks everyone who provided feedback on the new Impact module, we made some changes to the new Impact module edit count based on your feedback. We don't have experiment results yet, so the new Impact module is still only released as an A/B test on Growth pilot wikis (Arabic, Bengali, Czech, and Spanish), but hopefully I'll have more information to share soon.

Post-edit message T322386

Newcomer notification T322435

In the meantime, the Growth team is shifting focus to the next part of the Positive Reinforcement project: the Leveling up features. These features are meant to help engage new editors and encourage them to try new types of newcomer-friendly suggested edits. Like all major Growth features, we will conduct A/B tests to carefully monitor the impact of this work.

After our first round of community feedback several months ago, we shifted our approach and decided to descope goal-based incentives, including the daily goal idea. Our focus is now more on encouraging users to try more difficult task-types as they are successful with easier tasks.

We also plan to test two types of notifications to see if they help reengage new editors and improve newcomer retention. If you want more details you can read more about current Leveling up designs and plans here, or you can view all of the associated Leveling up Phabricator tasks here.

@Pigsonthewing @NGC 54, @Barkeep49, @TheDJ - Previously you all mentioned concerns around some of the goal or badge-based incentives we were considering. You can see that we have cut that out of the Leveling up project. Are you more comfortable with this plan now? Do you have any further feedback?

Do you think post-edit messages and notifications will encourage newcomers?

If the experiment shows that these features are helping with newcomer retention, then we'll plan to make these features community configurable via the Growth configuration, so communities can control if the features are enabled and how they work. Do you have any thoughts on what you would want to configure?

Can you think of anyone else who might want to offer feedback on this project?

(Pinging @BusterD, @Sdkb, and @Clovermoss since we've been discussing new editor challenges recently).

This does address my previous concerns and I'm pretty excited to see what the A/B test results will be. My expectation will be modest but meaningful improvements (versus the larger improvements goal/badge might have achieved at a higher moderation cost)