Croissance/Renforcement positif

Groupe :	Croissance
Début :	2022-02-25
Membres de l'équipe :	Gergő Tisza (software engineer), Benoît Evellin (movement communications specialist), Elena Tonkovidova (QA engineer), Sergio Gimeno (software engineer), Mew Ophaswongse (software engineer), Morten Warncke-Wang (data analyst), Rita Ho (senior user experience designer), Mariya Shilova (program manager)
Direction :	Kosta Harlan
Gestion :	Kirsten Stoller (product)
Mises à jour :	Most updates posted here.

This page is a translated version of the page Growth/Positive reinforcement and the translation is 22% complete.

Suivi dans Phabricator
Tâche T248330

Cette page décrit le travail sur le "renforcement positif" dans le cadre de l'ensemble des fonctionnalités de Croissance. Cette page contient les principales ressources, les designs, les questions ouvertes et les décisions.

La plupart des mises à jour progressives seront publiées sur la page des mises à jour de l'équipe Growth, et certaines mises à jour importantes ou détaillées seront publiées ici.

Statut actuel

2021-03-01: Page du projet créée
2022-02-25: Début du projet et discussions avec l'équipe
2022-03-01: Page du projet développée
2022-05-11: discussion avec la communauté
2022-08-12: user testing complete
2022-11-24: current designs and measurement and experiment plan added
2022-12-01: new impact module released to pilot wikis
2023-02-07: Leveling up and Personalized praise work started & second community discussion started
2023-02-14: published Newcomer task milestone analysis which will help guide Leveling up work
2023-03-22: Leveling up features released as an A/B test at Growth pilot wikis
2023-03-24: published Thanks Usage analysis
2023-05-25: released Personalized praise module on Growth pilot wikis
2023-10-01: released new Impact module on all Wikipedias & published Leveling up experiment results
Next: release Personalized praise on all Wikipedias

Résumé

L'équipe Croissance s'est attachée à créer une « expérience cohérente pour les nouveaux arrivants » qui donne accès aux éléments dont les nouvelles personnes ont besoin pour rejoindre la communauté éditrice de Wikipédia. Par exemple, avec les tâches pour récemment arrivés, nous leur avons donnés accès à des occasions de participer, et avec le module de mentorat, leur avons donnés accès à des mentors. Les éditions suggérées ont permis à davantage de nouveaux arrivants de réaliser leurs premières éditions. Avec ce succès, nous voulons prendre des mesures pour encourager les nouveaux à continuer à faire plus d'éditions. Cela attire notre attention sur un élément non développé auquel les nouveaux arrivants doivent avoir accès : l'évaluation des performances. Nous appelons à ce projet « renforcement positif ».

Nous voulons que les nouveaux comprennent qu'il y a une progression et une valeur aux contributions suivies sur Wikipédia, en augmentant la rétention de ces utilisateurs qui ont fait le premier pas en éditant

Notre grande question est : Comment nous pouvons encourager les novices qui ont visité la page d'accueil des nouveaux arrivants, et qui ont essayé nos fonctions, à continuer d'éditer, et rendre pérenne de cette première impulsion ?

Contexte

Lorsque la page d'accueil des nouveaux arrivants a été déployée en 2019, elle contenait un « module d'impact » basique, qui répertoriait le nombre de pages vues pour les pages que le nouvel arrivant avait éditées. C'est la seule partie des fonctionnalités de Croissance qui donne au nouvel arrivant une idée de son impact, et nous ne l'avons pas améliorée depuis son premier déploiement.

Sur cette base de départ, nous avons réuni quelques apprentissages importants concernant le reforcement positif :

Les membres de la communauté nous ont transmis leurs impressions positives concernant ce module, et les contributeurs expérimentés disent qu'il est intéressant et qu'il leur rend service.
Il a été démontré que l'appréciation d'autres utilisateurs augmente la rétention, comme dans le cas des « remerciements » (ici et ici) et dans une expérience sur la Wikipédia allemande. Nous pensons que ces renforcements provenant de personnes réelles seraient plus efficaces que ceux automatisés provenant du système.
Les membres de la communauté ont expliqué qu'il est prioritaire pour les nouveaux arrivants de passer à des tâches plus importantes après avoir commencé par des tâches faciles, plutôt que de s'enliser dans des tâches faciles.
Autres plateformes, comme Google, Duolingo et Github, utilisent des nombreux mécanismes de renforcement positif, comme des insignes à acquérir et des objectifs à atteindre.
Les communautés sont réticentes à encourager des modification de faible valeur. Nous avons vu que lorsque les concours d'édition offrent des prix en espèces, ou simplement lorsque des rôles utiles tels que autopatrolled dépendent du nombre d'éditions, cela peut inciter les gens à faire de nombreuses éditions problématiques.

Personnage-utilisateur

Il existe de nombreuses étapes du parcours des nouveaux arrivants dans lesquelles nous pourrions tenter d'augmenter la rétention. Nous pourrions nous concentrer sur les nouveaux arrivants qui ont arrêté d'éditer après seulement une ou quelques éditions, ou nous pourrions nous concentrer plus loin dans le parcours sur les nouveaux arrivants qui ont arrêté d'éditer après des semaines d'activité. Pour ce projet, nous avons décidé de nous concentrer sur les nouveaux arrivants qui ont terminé leur première session d'édition et que nous souhaitons voir revenir pour une deuxième session. Le diagramme les illustre par une étoile jaune.

Nous voulons nous concentrer sur les nouveaux arrivants à ce stade, car il s'agit de l'étape suivante de l'entonnoir des éditeurs, au cours de laquelle nous pouvons contribuer à améliorer la rétention. C'est également là que nous constatons un taux d'attrition très important actuellement, donc si nous pouvons aider à retenir les nouveaux arrivants à ce stade, cela devrait avoir une augmentation significative de la croissance des éditeurs au fil du temps.

Recherche et design

Des recherches ont été menées sur les différents mécanismes qui ont été employés pour encourager les gens à contribuer aux contenus des wikis, à la fois directement sur les wikis mais aussi en dehors. Voici quelques-uns des points clés provenant de cette recherche :

Les motivations des éditeurs de Wikipédia sont multiples et changent avec le temps et l'expérience. Les nouveaux éditeurs tendent à se laisser porter plus par la curiosité et la connexion sociale que par l'idéologie.
Les projets internes se centrent en des stimulants intrinsèques, font appel à des motivations altruistes et ne s'appliquent pas systématiquement.
Élargir les motivations au-delà des idéologies seules se peut améliorer la diversité des éditeurs fidélisés sur Wikipédia.
Les messages positifs des utilisateurs expérimentés et les mentors ont démontré leur efficacité dans la rétention à court terme.

Pour avoir un résumé des idées actuelles de design sur le renforcement positif, merdi de consulter ce Rapport de design. Nos designs continueront à évoluer grâce aux commentaires de la communauté et à divers cycles de tests-utilisateurs.

Idées

Nous avons trois idées principales pour le renfort positif. Il est possible que nous poursuivions avec plusieurs idées tant que nous travaillons dans ce projet.

Impact

Impact : une révision du module d'impact basée sur l'incorporation de stats, graphiques et d'autres informations sur les contributions. Le module d'impact révisé fournirait aux nouveaux éditeurs plus contexte quant à leur impact, tout en les encourageant à continuer à contribuer. Les pistes d'exploration sont :
- Jalons du nombre de modifications suggérées, pour encourager aux utilisateurs à essayer les éditions suggérées.
- Statistiques concernant le numbre de modifications effectuées dans le temps (similaire à ce que propose X Tools).
- Décompte des « remerciements reçus », pour souligner la capacité à recevoir de la reconnaissance venant de la communauté.
- Activité d'édition récente - y compris les jours d'affilée où les nouveaux arrivants ont édité (« séries ») pour encourager un engagement continu, ou rappeler aux gens de reprendre leurs contributions.
- Voir l'activité de la consultation des articles que les nouveaux arrivants ont édité au fil du temps (similaire à l'info présentée sur Wikipedia:Pageview_statistics).

Module d'impact, design A - plus d'importance accordée à l'impact d'un utilisateur sur les autres (lecteurs et éditeurs)
Module d'impact, design B - met l'accent sur l'activité d'édition récente d'un utilisateur

Changer de niveau

Monter de niveau: il est important pour les communautés que les contributeurs récemment arrivés progressent vers des tâches de plus forte valeur. Pour ceux effectuant de nombreuses tâches faciles, nous souhaitons les inciter à essayer des tâches plus difficiles. Ceci pourrait arriver après avoir complété un nombre déterminé de tâches simples, ou via un encouragement sur leur page d'accueil. Les pistes d'exploration incluent :
- Le nouveau venu verra des messages de réussite après l'édition qui le motiveront à faire d'autres éditions de même niveau de difficulté ou de niveau différent.
- Le module d'éditions suggérées suggère la possibilité de réaliser des modifications plus difficiles, pour que les nouveaux puissent devenir des éditeurs plus expérimentés.
- Dans le module d'impact, ajouter un compteur de jalons ou de récompenses.
- Sur la page d'accueil, ajouter un module présentant des challenges pour atteindre une récompense (badge, certificat).
- Ajouter des notification encourageant les nouveaux à essayer des tâches plus difficiles.

Idée de design incorporand un objectif d'édition quotidien
Les nouveaux peuvent monter en niveau pour des tâches plus difficiles et recevoir une reconnaissance
Un module de reconnaissance sur la page d'accueil des nouveaux arrivants
Newcomers making poor quality “fast edits” could receive guidance
Newcomers who complete a challenge could receive a sharable "Skilled newcomer" award

Éloge personnalisé

Éloges personnalisés : les recherches démontrent que les reconnaissances et stimulations venant d'autres utilisateurs augmentent la rétention de ceux récemment arrivés. Nous voulons réfléchir à comment encourager les utilisateurs expérimentés à remercier et décerner un prix aux récemment arrivés pour leurs bonnes contributions. Peut-être les mentors pourrait être encouragés à faire cela sur leur page de suivi (Mentor Dashboard) ou via les notifications. Nous pouvons utiliser les mécanismes de communication existants qui, selon des études antérieures, ont démontré un certain degré d'effet positif. Les pistes d'exploration sont :
- Un message personnel du mentor tel qu'il apparaît sur la page d'accueil d'un nouvel utilisateur.
- Une notification (Echo) venant d'un mentor ou de l'équipe de Croissance.
- « Remerciement » sur une modification spécifique.
- Un nouvel insigne de jalon attribué par le mentor ou l'équipe de croissance de Wikimedia en relation avec une modification spécifique.

Affichage des remerciements sur la page d'accueil
Affichage de Wikilove sur la page d'accueil

Discussion avec la communauté

We discussed the Positive Reinforcement project with community members from Wikipédia en arabe, Wikipédia en bengali, Wikipédia en tchèque et Wikipédia en français, and here on mediawiki.org.

We received direct feedback about the three main ideas, along with many other ideas for improving new editor retention.

Below is a summary of the main themes from the feedback, along with how we plan to iterate based on the feedback.

Impact


We heard...	Plans to iterate based on feedback
😊 Looks good!	This idea seems the least controversial and most supported. We will plan to start development on this first, and allow for more time to refine other ideas.
😐 The impact module would be more effective if it scaled with editors as they gained experience.	We plan to focus on newcomers for now, but the new impact module will be built in an extensible way to accommodate improvements in the future.

Leveling up


We heard...	Plans to iterate based on feedback
😊 Leveling up ensures newcomers don't get "stuck" on easy tasks	Once users have a certain number of unreverted edits of one type, we should suggest they try more difficult tasks.
😊 Newcomers are often eager for awards	If we give awards they will need to feel meaningful to newcomers, and ideally are sharable either on-wiki (on their user page) or off-wiki.
❌ Goal-based incentives might be problematic, and may result in low-quality edits	Incentives that include a time-based element (similar to service awards) might be an effective approach as they factor in not only number of edits, but length of time registered. Certain "quality gates" could help slow down and guide newcomers if they are making edits that are getting reverted. We plan to reduce the scope on the award-side of "Leveling Up" for now, and focus more on encouraging users to try more difficult task-types as they are successful with easier tasks.
❌ Daily goals might be stressful and demotivating for some people	We will review this idea further and likely allow for goal customization if we pursue this idea.

Personalized praise


We heard...	Plans to iterate based on feedback
😊 Spreading praise and positivity might help increase newcomer retention.	We are still refining designs for how to encourage more Thanks and personalized paise of newcomers, but hope to have further design ideas to present soon.
😐 Scaling personalized praise might be a challenge as it takes more time for experienced editors.	Mentors are already busy, so we hope to find a way to surface "praise-worthy" mentees. We will also brainstorm other ideas that don't rely on just mentors.
😐 We should use existing systems (Thanks, WikiLove, etc.)	Plans aren't finalized, but we definitely plan to take advantage of existing systems.

Other ideas:

Community members suggested several other ideas for improving newcomer engagement and retention. We think these are all valuable ideas (some of which we are already exploring or want to work on in the future) but the following ideas won't fit within the scope of the current project:

Send newcomers onboarding and welcome emails (the Growth team is actually currently exploring engagement emails in collaboration with the Marketing and the Fundraising teams).
Expose newcomers to Wikiprojects that relate to their interests.
Include a customizable widget on the newcomer homepage to allow wikis to promote certain newcomer tasks or events.
Send notifications to users who welcome newcomers once the newcomer reaches certain editing milestones (to help prompt the user to offer Thanks or Wikilove).

Second community consultation:

In February 2023, we completed a community consultation in which we reviewed the most recent Leveling up designs with the Growth Pilot wikis. This consultation was completed in English on MediaWiki, and at Arabic Wikipedia, Bengali Wikipedia, Czech Wikipedia, and Spanish Wikipedia. (T328356) In general, feedback was quite positive. These two tasks help address feedback mentioned by those that responded to our questions:

Leveling up: Community configuration (T328386)
Leveling up: Second design iteration of "Try a new task" dialog (T330543)

In March 2023, we completed a community consultation in which we reviewed the most recent Personalized praise designs with the Growth Pilot wikis. This consultation was completed on English Wikipedia, Arabic Wikipedia, Bengali Wikipedia, Czech Wikipedia, French Wikipedia, Spanish Wikipedia, and at MediaWiki in English. (T328356) Most feedback was supportive of Personalized praise features, but several further improvements were requested. We've created Phabricator tasks to address these further improvements.

On Arabic Wikipedia, and other wikis with Flagged Revisions, mentors want to see not only the number of edits a user had completed, but more details on the review status of edits (T333035)
Mentors want to be able to view the number or percentage of reverts their mentee has, and customize how many reverts a newcomer can have to be considered praiseworthy (T333036)
Mentors would appreciate knowing which edit a mentee is Thanked for (T51087)

User testing

Along with community discussion, we wanted to validate and add to our initial designs and hypothesis by testing designs with readers and editors from several countries. So our design research team conducted Positive Reinforcement user testing aimed to better understand the project's impact on newcomer contribution across several different languages.

We tested several static Positive Reinforcement designs with Wikipedia readers and editors in Arabic, Spanish, and English. Along with testing Positive Reinforcement designs we introduced data visualizations from xtools as a way to better understand how these data visualizations are perceived by newcomers.

User testing results

Make impact data actionable: Impact data was a compelling feature for participants with more experience editing, which several related to their interest in data—an unsurprising quality for a Wikipedian. For those new to editing, impact data, beyond views and basic editing activity, may be more compelling if linked to goal-setting and optimizing impact.
Evaluate the ideal editing interval: Across features, daily intervals seemed likely to be overly ambitious for new and casual editors. Participants also reflected on ignoring similar mechanisms on other platforms when they were unrealistic. Consider consulting usage analytics to identify “natural” intervals for new and casual editors to make goals more attainable.
Ensure credibility of assessments: Novice editor participants were interested in the assurance of their skills and progress the quality score, article assessment, and badges offer. Some hoped that badges could lend credibility to their work reviewed by more experienced editors. With that potential, it could be valuable to evaluate that the assessments are meaningful measures of skill and further explore how best to leverage them to garner community trust of newcomers.
Reward quality and collaboration over quantity: Both editor and reader participants from esWiki were more interested in recognition of their knowledge or expertise (quality) than the number of edits they have made (quantity). Similarly, some Arabic and English editors are motivated by their professional interests and skill development to edit. Orienting goals and rewards to other indicators of skilled edits, such as adding references or topical contributions, and collaboration or community involvement may also help mitigate concerns about competition overtaking collaboration.
Prioritize human recognition: While scores and badges via Growth tasks is potentially valued, recognition from other editors appears to be more motivational. Features which promote giving, receiving, and revisiting thanks seemed most compelling, and editors may benefit from selecting impact data which demonstrates engagement with readers or editors most compelling to them.
Experiment with playfulness of designs: While some positive reinforcement features can be seen as the product of “gamification”, some participants (primarily from EsWiki) felt that simple, fun designs were overly childish or playful for the seriousness of Wikipedia. Consider experimenting with visual designs that vary in levels of playfulness to evaluate broader reactions to “fun” on Wikipedia.

Design

Below are the current designs for Positive Reinforcement. We have refined the three main ideas outlined above, but the scope of plans and the actual designs have evolved based on feedback from community discussions and user testing.

Impact

Suivi dans Phabricator
Tâche T222310

The revised impact module provides new editors with more context about their impact. The new design includes far more personalized info and data visualizations than the previous design. This new design is fairly similar to the design we shared previously when discussing this feature with communities. You can view the current engineering progress at beta wiki, and we hope to release this feature to Growth pilot wikis soon.

Leveling up

Suivi dans Phabricator
Tâche T317813

The Leveling up features focus on encouraging newcomers to progress to more valuable tasks. Ideas also include some prompts for new editors to try suggested edits, since structured tasks have been shown to improve newcomer activation and retention.

“Level up” post-edit dialog message: A new post-edit dialog message type is added to encourage newcomers to try a new task type. We hope this will encourage some users to learn new editing skills as they progress to different, more challenging tasks.
Post-edit dialog for non-suggested edits: Introduce newcomers who complete ‘normal’ edits to suggested edits. We plan to experiment by showing newcomers a prompt post 3rd and 7th edit. Desktop users who click through to try a suggested edit will also see their Impact module, which we hope helps engage newcomers and provides a small degree of automated positive reinforcement. We will carefully measure this experiment, and ensure there aren't any unintentional negative effects.
New notifications: New echo notifications to encourage newcomers to start or continue suggested edits. This acts as a proxy to “win-back” emails for those who have an email address and settings on to receive email notifications.

“Level up” post-edit dialog message
Post-edit dialog for non-suggested edits
New notification

Personalized praise

Suivi dans Phabricator
Tâche T317817

Personalized praise features are based on research results that show that encouragement and thanks from other users increases editor retention.

Encouragement from Mentors: We will add a new module to the Mentor dashboard, that is designed to encourage Mentors to send personalized messages to newcomers who meet certain criteria. We will allow Mentors to customize and control how and when "praise-worthy" mentees are surfaced.
Increasing Thanks across the wiki: We plan to fulfill the community wishlist item to Enable Thanks Button by default in Watchlists and Recent Changes (T51541, T90404). We hope this will increase Thanks and positivity across the wikis, and hopefully newcomers will benefit from this directly or indirectly.

Design for a new Mentor dashboard module
Design of the settings view of the new dashboard module

Measurement and results

Hypotheses

The Positive Reinforcement features aim to provide or improve the tools available to newcomers and mentors in three specific areas that will be described in more detail below. Our hypothesis is that once a newcomer has made a contribution (say by making a structured task edit), these features will help create a positive feedback cycle that increases newcomer motivation.

Below are the specific hypotheses that we seek to validate across the newcomer population. We will also have hypotheses for each of the three sets of features that the team plans to develop. These hypotheses drive the specifics for what data we will collect and how we will analyse that data.

The Positive Reinforcement features increase our core metrics of retention and productivity.
Since the Positive Reinforcement features do not feature a call to action that asks newcomers to make edits, we will see no difference in our activation core metric.
Newcomers who get the Positive Reinforcement features are able to determine that making un-reverted edits is desirable, and we will see a decrease in the proportion of reverted edits.
The positive feedback cycle created by the Positive Reinforcement features will lead to a significantly higher proportion of "highly active" newcomers.
The Positive Reinforcement features increase the number of Daily Active Users of Suggested edits.
The average number of edit sessions during the newcomer period (first 15 days) increases.
"Personalized praise" will increase mentor’s proactive communication with their mentees, which will lead to increase in retention and productivity.

Experiment plan

Similarly as we have done for previous Growth team projects, we want to test our hypotheses through controlled experiments (also called "A/B tests"). This will allow us to establish a causal relationship (e.g. "The Leveling Up features cause an increase in retention of xx%"), and it will allow us to detect smaller effects than if we were to give it to everyone and analyze the effects pre/post deployment.

In this controlled experiment, a randomly selected half of users will get access to Positive Reinforcement features (the "treatment" group), and the other randomly selected half will instead get the current (September 2022) Growth feature experience (the "control" group). In previous experiments, the control group has not gotten access to the Growth features. The team has decided to move away from that (T320876), which means that the current set of features is the new baseline for a control group.

The Personalized Praise feature is focused on mentors. There is a limited number of mentors on every wiki, whereas when it comes to newcomers the number increases steadily every day as new users register on the wikis. While we could run experiments with the mentors, we are likely to run into two key challenges. First, the limited number of mentors could mean that the experiments would need to run for a long time. Second, and more importantly, mentors are well integrated into the community and communicate with each other, meaning they are likely to figure out if some have access to features that others do not. We will therefore give the Personalized Praise features to all mentors and examine activity and effects on newcomers pre/post deployment in order to understand the feature’s effectiveness.

In summary, this means we are looking to run two consecutive experiments with the Impact and Leveling up features, followed by a deployment of the Personalized Praise features to all mentors. These experiments will first run on the pilot wikis. We can extend this to additional wikis if we find a need to do that, but it would only happen after we have analyzed the leading indicators and found no concerns.

Each experiment will run for approximately one month, and for each experiment we will have an accompanying set of leading indicators that we will analyze two weeks after deployment. The list below shows what the planned experiments will be:

Impact: treatment group gets the updated Impact module.
Leveling up: treatment group gets both the updated Impact module and the Leveling up features.
Personalized praise: all mentors get the Personalized praise features.

Leading indicators and plan of action

While we believe that the features we develop are not detrimental to the wiki communities, we want to make sure we are careful when experimenting with them. It is good practice to define a set of leading indicators together with plans of what action to take based if a leading indicator suggests something isn't going the way it should. We have done this for all our past experiments and do so again for the experiments we plan to run as part of this project.

Impact

Impact of the Impact module - results published on Jan 24, 2023.
Indicator	Expected result	Plan of action	Results
Impact module interactions	No difference or increase	If Impact module interactions decrease, then this suggests that we might have performance or compatibility issues with the new Impact module. If the proportion of newcomers who interact with the new Impact module is significantly lower than the old module we investigate the cause, reverting back to the old module if necessary.	Significant decrease
Mentor module interactions	No difference	The new Impact module takes up more screen real estate than the old module, which might lead to newcomers not finding the Mentor module as easily as before. If the number of newcomers who interact with the Mentor module is significantly lower for those who get the new Impact module, we investigate the need for design changes.	No signifiant difference
Mentor module questions	No difference	Similar concerns as for interactions with the Mentor module, if the number of questions asked to mentors is significantly lower for newcomers who get the new Impact module, we investigate the need for design changes.	No signifiant difference
Edits and revert rate	No difference in both edits and reverts, or an increase in edits and a decrease in revert rate	If there is an increase in the revert rate, this may suggest that newcomers are making unconstructive edits in order to inflate their edit or streak count. If the revert rate of newcomers who get the new Impact module is significantly higher than the old, we investigate their edits and decide whether changes are needed.	No signifiant difference (once outliers are removed)

Suivi dans Phabricator
Tâche T323788

Impact module interactions: We find that the proportion of newcomers who interact with the old module (6,1%) is significantly higher than for the new module (5,0%): $χ^{2} = 17.5, d f = 1, p ≪ 0.001$ This difference showed up early on in the experiment, and we have examined the data more closely understand what is happening. One issue we identified early on was that not all interaction events were instrumented, which we subsequently resolved. Examining further, we find that many of those who get the old module click on links to the articles or the pageviews. In the new module, a graph of the pageviews is available, thus removing some of the need for visiting the pageview tool. As a result, we decided that no changes were needed.

Mentor module interactions: We find no significant difference in the proportion of newcomers who interact with the Mentor module. The proportion for newcomers who get the old module is 2,4%, for those who get the new module it's 2,2%. A Chi-square test finds this difference not significant: $χ^{2} = 1.5, d f = 1, p = 0.219$

Mentor module questions: We do not see a substantial difference in the number of questions asked between the old module (269 edits) and the new module (281 edits). The proportion of newcomers who asks their mentor a question is also the same for both groups, at 1,5%.

Edits and revert rate: We do not see a substantial difference in the number of edits nor in the revert rate between the two groups measured on a per-user average basis. There are differences between the groups, but these are driven by some highly prolific editors, particularly on the mobile platform.

Levelling up

Leading indicators for the Levelling up experiment
Indicator	Expected result	Plan of action	Results
Levelling up post-edit dialog: interactions	No difference or increase	The percentage of users who click / tap on a Levelling up post-edit dialog should be similar or higher than the percentage of users who click / tap on the standard post-edit dialog. If there is a decrease, then we need to investigate what causes this difference.	Higher on mobile, no difference on desktop
Levelling up post-edit dialog: "Try a suggested edit" click through	>10% click through to suggested edits	If the "try a suggested edit" dialog isn't resulting in more newcomers exploring suggested edits, then this notice is just extra noise for newcomers and we should investigate or consider removing the feature.	Significantly higher than 10%
Levelling up post-edit dialog: "Increase your skill level" click through	>10% click through to Try new task	If the "increase your skill level" isn't resulting in more newcomers trying more difficult tasks, then this notice is just extra noise for newcomers and we should investigate or consider removing the feature.	Significantly higher than 10%
Levelling up notifications: "get started" click through	>5% of users who view this notification click on it	We don't have a great baseline to compare this to, but if this number is too low we should investigate if there are technical issues or an issue with the language used.	More than 5% on desktop, less than 5% on mobile
Levelling up notifications: "keep going" click through	>5% of users who view this notification click on it	We don't have a great baseline to compare this to, but if this number is too low we should investigate if there are technical issues or an issue with the language used.	More than 5% on desktop, less than 5% on mobile
Activation	No difference or increase	If we see a significant decrease in the treatment group, similar to what we discovered for the New Impact Module experiment, then we examine monitoring and event data to try to identify a cause of this difference.	Decrease

Suivi dans Phabricator
Tâche T328055

Levelling up post-edit dialog interactions: We find a higher proportion of newcomers interacting with the post-edit dialog in the Levelling Up group (90,8%) compared to the standard post-edit dialog (86,5%). This is largely driven by mobile where the Levelling Up interaction proportion (88,0%) is a lot higher than the other group (81,6%). The proportion is still higher for the Levelling Up group on desktop (93,6%) compared to the control (92,2%), but we regard it as "virtually identical" because the high proportion in the control group means there is little room for an increase.

Try a suggested edit click through rates: 21,9% of newcomers who see the "Try a suggested edit" post-edit dialogue chooses to click through, which is significantly higher than the threshold set. The proportion is higher on desktop (24,0%) than on mobile (19,7%), but in neither case is there a reason for concern.

Increase your skill level click through rates: We find that 73,1% of newcomers who see the "increase your skill level" dialog click through to see the new task, which is a lot higher than our expected threshold of less than 10%. Proportions are high on both desktop (71,1%) and mobile (77,3%).

Get started click through rates: 3,8% of newcomers who get the "Get started" notification clicks through to the Homepage. Users who registered on desktop are more likely to click the notification (5,5%) than those on mobile (2,5%). Because the threshold of 5% is met, we are investigating further to understand this difference between desktop and mobile behaviour, particularly to understand if our 5% threshold is reasonable.

Keep going click through rates: We find that 9,6% of users who get the "Keep going" notification clicks through to the Homepage. Similarly as we do for the "Get started" notifications, we find a much higher proportion on desktop (16,2%) compared to mobile (4,7%). Our investigations into differences in notification behaviour by platform will hopefully give us more insight into this difference.

Activation: We find a decrease in constructive article activation (making a non-reverted article edit within 24 hours of registration) of 27,0% compared to 27,7%. As soon as we noticed this we opened T334411 to investigate the issue, with a focus on patterns in geography (countries and wikis) and technology (devices and browsers). We did not find clear patterns explaining the issue. The investigation of this decrease in activation will be investigated further: T337320.

Personalized praise

Leading indicators for the Personalized praise experiment
Indicator	Expected result	Plan of action	Results
Personalized praise notification click through	At least 10% of Mentors who view a Personalised praise notification click on it	If this number is much lower than the click through on other notifications, then we should investigate if there are technical issues or an issue with the language used.	73% of Mentors who received a notification clicked on it
Personalized praise mentor dashboard module click through	At least 10% of Mentors who view a Personalised praise suggestion on their Mentor dashboard end up clicking through to send praise	If this threshold is met then we should investigate if there are technical issues or an issue with how Mentors are interpreting this call to action.	27.5% of Mentors who view a Personalized praise list click through

Suivi dans Phabricator
Tâche T337069

Data was gathered on 2023-06-13, from the four pilot wikis where the feature is deployed (Arabic Wikipedia, Bengali Wikipedia, Czech Wikipedia, and Spanish Wikipedia).

Personalized praise notification click through: Although this is still a relatively small sample, results seem healthy and show that Mentors are indeed receiving notifications and clicking through to view their praise-worthy mentees.

Personalized praise mentor dashboard module click through: Only 27.5% of Mentors are clicking through to a mentee's talk page, however it's to be expected that some of the mentees we are surfacing aren't deserving of praise. Based on this data and feedback from Mentors, the Growth team will pursue the following tasks to help improve this feature:

Add revert scorecard to Personalized praise module on Mentor dashboard (T337510)
Exclude blocked accounts from the Personalized praise suggestions (T338525)

Experiment Results

Many of the experiments that the Growth team runs will focus on the same set of key metrics (commonly referred to as KPIs), and this includes all of the Positive Reinforcement experiments. The key metrics are defined as follows:

Constructive activation is defined as a newcomer making their first edit within 24 hours of registration, and that edit not being reverted within 48 hours.
- Activation is similarly defined as constructive activation, but without the non-revert requirement.
Constructive retention is defined as a newcomer coming back on a different day in the two weeks after constructive activation and making another edit, with said edit also not being reverted within 48 hours.
- Retention is similarly defined as constructive retention, but without the non-revert requirements.
Constructive edit volume is the overall count of edits made in a user's first two weeks, with edits that were reverted within 48 hours removed.
Revert rate is the proportion of edits that were reverted within 48 hours out of all edits made. This is by definition 0% for users who made no edits, and we generally exclude these users from the analysis.

Impact module experiment results

The New Impact module reduced activation for mobile web newcomers

We initially found a significant decrease in constructive activation for newcomers who registered on mobile web and got the New Impact module.

There was no difference in activation for newcomers who registered on desktop.

This was quite surprising as the empty state for the old Impact module was nearly identical to the empty state of the new Impact module.

First-day activity correlates strongly with later activity, and as a result we also found a significant decrease in edit volume for mobile web users.

Again, there was no difference for desktop users.

We found no difference in retention rates and revert rates. While there are features in the New Impact module that focuses on staying active and making good contributions, such as the number of thanks received and the streak counter, we often do not see significant impacts on metrics unless there's a clear call to action or we are able to isolate a specific subgroup motivated by the feature.

As soon as we learned about the decrease in activation we started investigations into probable causes of this in T330614. Unfortunately we could not identify a specific reason and we also found that the issue was not replicated in another dataset.

We decided to add activation as a leading indicator to the Levelling Up experiment so that we could take action more quickly. When we noticed that the issue persisted, we started a new investigation in T334411 and created an "epic" task that connects all relevant subtasks: T342150. We restarted experiment data collection after making several small changes, and we now see that activation is identical between the experiment and control group, which is what we would expect.

Although we are pleased that we have received positive feedback from new editors regarding the new Impact module, we have found that the Impact module alone hasn't resulted in significant changes in newcomer retention, edit volume, or revert rates. Our next experiment will combine the new Impact module with the Leveling up features. We hope that this combination of Positive Reinforcement features will lead to substantial improvements in activation, retention, and edit volume. We will soon publish a detailed report that highlights the outcomes of this experiment.

Levelling up experiment results

For this experiment, we completed both an analysis of the overall effects across the whole newcomer population, and individually for each of the four components of the Levelling up features. These consist of the two notifications sent to newcomers 48 hours after registration, and two post-edit dialogues. The notifications are based on the number of suggested edits a newcomer might have done. If the newcomer has not made any suggested edits they get the "Get started" notification, and if they have made one to four suggested edits they get the "Keep going" notification. Newcomers who have made five or more suggested edits do not get any notifications.

The post-edit dialogues are shown after completed edits to articles based on certain criteria. If a newcomer has made three or eight article edits and not yet made any suggested edits, they get the "Try suggested edits" dialogue asking them if they want to try that feature. If a newcomer has completed five suggested edits of a specific task type, they get the "Try new task" dialogue suggesting a different type of task.

The Get Started notification increased editing within one week after receiving it.

Our overall analysis did not find any significant effects on the team's key metrics (described above), and so we focus instead on the individual components. For the "Get started" notification, we find that this is sent to the vast majority of newcomers as making suggested edits is fairly uncommon. In our dataset, more than 97% of newcomers got this notification. We find that the notification leads to a significant increase in newcomer activity in the week following the notifications being sent. Newcomers are more likely to return and make an edit, which also increases the average number of edits made during that week. We also find that this effect is lower for those who registered on mobile web, and reduced or negative for highly active newcomers. Based on this, we decided to introduce a threshold so that those who make ten or more edits will not receive the notification. (T342819)

The Keep Going notification increased editing within one week for desktop users.

When it comes to the "Keep going" notification, we again find a significant increase in newcomer activity in the week following notifications being sent for those who registered on the desktop platform. For users who registered on mobile web, we find that it does not increase their probability of returning to edit but does increase the average number of edits made.

For the "Try suggested edits" dialogue, our analysis finds that while it has a reasonably high click-through rate it does not lead to newcomers successfully completing suggested edits. In our leading indicators report above, the click-through rate was 21.9%, and in a dataset from late July 2023 we found the rate to be higher at 25.3%. Using event data, we find that few newcomers find a task they are interested in, and subsequently only a fraction of newcomers go through and complete an edit. We plan to make a few improvements to this "Try suggested edits" dialog to see if we can increase the percentage of editors who click through and go on to complete an edit. (T348205)

For the "Try new task" dialogue, which is shown to users who complete five suggested edits of a given task type, we find both high click-through rates and a reasonably high rate of completed edits. We reported a click-through rate of 73.1% in our leading indicators, and in our more recent dataset from late July 2023 the rate is 81.9%. Our analysis of subsequent edits shows that 33.3% of desktop users and 20.0% of mobile web users go through and complete a suggested edit of the new task type. One thing to keep in mind is that this dialogue is not shown to a large number of newcomers, and we therefore cannot draw conclusions about whether there are meaningful differences between platforms. What we can conclude, is that this dialogue is successful in introducing new task types. In order to show the dialogue to a larger number of newcomers, we decided to reduce the number of edits needed to see it from five to three. (T348814)

Personalized praise experiment results

For this experiment, we focused on the effect of praise on newcomer retention and productivity. Since praise is a response to editing activity, it means there will be some time period between registration and receiving a praise message. We therefore started with an analysis of the time between registration and a mentor clicking the "Send praise" button. In that analysis, we found that most newcomers get it within 30 days of registration. This led us to redefine the time period for retention and productivity to also use this 30-day period (instead of our default of 14 days).

The Personalized praise feature was deployed to the Arabic, Bangla, Czech, and Spanish Wikipedias in late May 2023. We analyzed the Spanish Wikipedia separately from the other three because on the Spanish Wikipedia 50% of newcomers are randomly assigned a mentor, which means the feature is part of a controlled experiment. All newcomers are assigned a mentor on the other three Wikipedias.

Using a Difference-in-Differences analysis approach, we compared a three-month period prior to deployment (January through March) with a similar period after deployment (June through August), and compared data from 2023 with data from 2022 and 2018. We use two comparison time periods as a robustness check since 2022 was affected by the COVID pandemic.

For the Arabic, Bangla, and Czech Wikipedia, we found no significant impact of Personalized praise on neither retention nor productivity. Digging further into this we found that usage of the feature was limited (we're not releasing specific counts in accordance with our data publication guidelines). In discussions with wiki ambassadors we learned that sending praise is a time-consuming process as the mentors need to check a mentee's edits, thus explaining why the feature isn't more widely used.

Personalized praise increased number of non-reverted edits made within 30 days of registration on Spanish Wikipedia.

When it comes to the Spanish Wikipedia, we found the feature has been more widely used. While we again found no significant impact on retention, we found a significant positive impact on newcomer productivity. This finding is encouraging since our preliminary analysis of mentorship found conflicting results of none or a negative impact.

As these results were not positive enough to justify the time investment from Mentors, we have decided to start conversations with our ambassadors and communities and consider further improvements before releasing the feature more widely. We will consider improvements related to reducing the amount of work needed by Mentors, potential design improvements, and improvements to how newcomers are selected to be displayed in the Personalized praise module.