Growth/Personalized first day/Structured tasks/Add an image/Experiment analysis, March 2024
 | This page in a nutshell: This analysis finds that the Add an Image structured task leads to an increase in newcomer participation on the mobile web platform, particularly by making constructive (non-reverted) article edits:
|
|
2022 Add an Image Experiment Analysis
Results from controlled experiment testing the efficacy of the Add an Image structured task
|
In late November 2021, the Growth team added the Add an Image structured task to the "Suggested Edits" module on the Newcomer Homepage. Structured tasks are suggested edits that can be broken down into step-by-step workflows with simple steps that make sense to newcomers, are easy to do on mobile devices, and may be assisted by machine learning. Similarly as when we released the Add a Link task earlier the same year, our hypotheses were that users would be likely to complete the Add an Image task and that this would lead to increases in our core metrics without decreasing edit quality.
To learn more about the impact of Add an Image, we deployed the feature in a controlled experiment. Users in this experiment were randomly assigned to one of three groups:
- A control group that did not get access to any of the Growth team features (20%)
- A group that got access to the Growth team features with Add an Image as the default task in the Suggested Edits module (40%)
- A group that got access to the Growth features and where the default task in the Suggested Edits module was Add a Link (40%)
This report analyzed data from early July through October 2022, where the feature was available on seven Wikipedia wikis: Arabic, Bangla, Czech, French, Persian, Portuguese, and Turkish. For more details about the experiment and the dataset, see the Methodology section below.
Summary of findings
[edit]In general, the analysis finds that the Add an Image structured tasks improves outcomes for newcomers on the mobile web platform compared to those who did not have access to the Growth team features. On that platform, the feature either performs as well as or better than Add a Link, while on the desktop platform the results are mixed.
- Newcomers on mobile web who get the Add an Image structured task are more likely to be activated (i.e. make a constructive first article edit) compared to the baseline, and equally likely as those who get the Add a Link structured task. On the desktop platform, Add an Image does not significantly increase activation over the baseline.
- They are more likely to be retained (i.e. come back and make another constructive article edit on a different day) relative to the baseline, and similarly likely as those who get the Add a Link task, if they are mobile web users. On the desktop platform there are no significant differences.
- Newcomers with Add an Image see an increased edit volume (i.e. number of constructive Article & Talk namespace edit over the first couple of weeks), while at the same time reducing revert rates. Those who get the Add a Link task see a similar increase in edit volume and reduction in revert rates.
Glossary
[edit]- We analyze data from seven Wikipedia wikis: Arabic, Bangla, Czech, French, Persian, Portuguese, and Turkish.
- Not all newcomers received the Add an Image structured task: 20% of users were randomly chosen to get the default newcomer experience, which does not have access to any of the Growth features. We refer to this group as the control group. 40% were randomly chosen to get the Growth features with the default task in the Suggested Edits module set to the Add an Image structured task, and we call that group the Add an Image group. The remaining 40% were also given the Growth features but with the Add a Link structured task as the default task in the Suggested Edits module. We refer to this group as the Add a Link group.
- Constructive activation is defined as a newcomer making their first edit within 24 hours of registration, and that edit not being reverted within 48 hours. The baseline constructive activation rate is the rate of constructive activation for the control group.
- Activation is similarly defined as constructive activation, but without the non-revert requirement.
- Constructive retention is defined as a newcomer coming back on a different day in the two weeks after constructive activation and making another edit, with said edit also not being reverted within 48 hours.
- Retention is similarly defined as constructive retention, but without the non-revert requirements.
- Constructive edit volume is the overall count of edits made in a user's first two weeks, with edits that were reverted within 48 hours removed. The baseline constructive edit volume is the count for users in the control group.
- Revert rate is the proportion of edits that were reverted within 48 hours out of all edits made. This is by definition 0% for users who made no edits, and we generally exclude these users from the analysis.
Detailed findings
[edit]Below are the specific impacts estimated from the controlled experiment. These are based on observing 148,412 new accounts on the seven Wikipedia wikis between July and October 2022. For more specifics on the methodology, see the Methodology section below.
Activation
[edit]In this analysis, we focus on the Article and Talk namespaces because the Add an Image task asks users to edit articles, and because we've seen in previous analyses that the Growth features can have a positive effect on activation.
Activation
[edit]Newcomers who register on the mobile web platform and get the Add an Image structured task are 11.2% more likely to activate compared to the baseline in the control group. Users who get the Add a Link structured task activate at a rate similar to that of the Add an Image group. On the desktop platform, there is no change in activation for users who get the Add an Image task compared to the control group, while users who get the Add a Link structured task are more likely to activate. When we add data from the French Wikipedia and compare the two structured tasks, we find no difference in activation rates between them, indicating that the effectiveness of the task for desktop users on that wiki is different to the overall average.
| Platform | Experiment group | Activation rate | Difference | Relative difference |
|---|---|---|---|---|
| Desktop | Control | 26.5% | ||
| Add an Image | 27.1% | +0.6pp | +2.2% | |
| Add a Link | 27.8% | +1.3pp | +4.8% | |
| Mobile web | Control | 24.8% | ||
| Add an Image | 27.6% | +2.8pp | +11.2% | |
| Add a Link | 27.1% | +2.2pp | +8.9% |
When we add the data from the French Wikipedia, the activation rates for the Add an Image group changes to 31.9% on desktop and 27.0% on mobile web. For the Add a Link group the activation rates are 32.4% and 26.8%, respectively. These differences are not statistically significant.
Constructive activation
[edit]
We find a larger effect of Add an Image when considering non-reverted edits in the Article & Talk namespaces compared to counting all edits. Users who registered on the mobile web platform and get the Add an Image task are 17.0% more likely to activate compared to the baseline in the control group. On the desktop platform, we again find no change in activation between users in the Add an Image and control groups, but we find an increase in activation for those who get the Add a Link structured task. When we add data from the French Wikipedia and compare only Add an Image and Add a Link, we again find no difference in the activation rates between them.
The statistics for activation rates, rounded to the nearest decimal, are shown in Table 2 below and in the related figures. The absolute and relative differences are calculated compared to the control group on each platform, and also rounded to the nearest decimal.
| Platform | Experiment group | Activation rate | Difference | Relative difference |
|---|---|---|---|---|
| Desktop | Control | 21.6% | ||
| Add an Image | 22.0% | +0.4pp | +1.9% | |
| Add a Link | 22.9% | +1.3pp | +6.1% | |
| Mobile web | Control | 16.6% | ||
| Add an Image | 19.4% | +2.8pp | +17.0% | |
| Add a Link | 18.9% | +2.2pp | +13.5% |
When we add users from the French Wikipedia, the constructive activation rates for the Add an Image group changes to 26.2% on desktop and 19.3% on mobile. For the Add a Link group the constructive activation rates are 26.8% and 19.1%. These between-group differences are not statistically significant.
Retention
[edit]Similarly as we did for activation, we focus our analysis of retention on the Article & Talk namespaces because Add an Image asks newcomers to edit articles.
Retention
[edit]Newcomers who register on the mobile web platform and get the Add an Image structured task are 15.9% more likely to be retained compared to the baseline. Users who get the Add a Link structured task are retained at a rate similar to those who get the Add an Image task. Newcomers on the desktop platform are retained at a rate similar to the baseline if they get the Add an Image task, while those who get the Add a Link task are retained at a slightly higher rate. When we add data from the French Wikipedia and compare the two structured tasks, we find no difference in retention rates between them.
The statistics for retention rates, rounded to the nearest decimal, are shown in Table 3 below. The absolute and relative differences are calculated compared to the control group on each platform, and also rounded to the nearest decimal.
| Platform | Experiment group | Retention rate | Difference | Relative difference |
|---|---|---|---|---|
| Desktop | Control | 3.5% | ||
| Add an Image | 3.6% | +0.1pp | +1.6% | |
| Add a Link | 3.7% | +0.2pp | +6.7% | |
| Mobile web | Control | 2.8% | ||
| Add an Image | 3.2% | +0.4pp | +15.9% | |
| Add a Link | 3.2% | +0.5pp | +16.5% |
When we add the data from the French Wikipedia, the retention rates for the Add an Image group changes to 4.3% on desktop and 3.1% on mobile web. For the Add a Link group the retention rates are the same as the Add an Image group.
Constructive retention
[edit]
Similarly as we did for activation, we find a larger effect of the structured tasks when considering non-reverted edits to the Article & Talk namespaces compared to when we use all edits. Users who registered on the mobile web platform and get the Add an Image structured task are 24.3% more likely to be retained compared to the baseline in the control group. Those who got the Add a Link structured task were retained at a similar but slightly lower rate (23.7% relative to the baseline). On the desktop platform, we again find those who got the Add an Image task were retained at a rate similar to the baseline (a 1.7% relative increase that is not significant). Users who got the Add a Link task were also retained at a similar rate (a 7.4% relative increase that is also not significant). When we add data from the French Wikipedia and compare the two structured tasks to each other, we find no significant difference in the retention rates between them.
The statistics for retention rates, rounded to the nearest decimal, are shown in Table 4 below and in the related figures. The absolute and relative differences are calculated compared to the control group on each platform, and also rounded to the nearest decimal.
| Platform | Experiment group | Retention rate | Difference | Relative difference |
|---|---|---|---|---|
| Desktop | Control | 2.9% | ||
| Add an Image | 3.0% | +0.1pp | +1.7% | |
| Add a Link | 3.2% | +0.2pp | +7.4% | |
| Mobile web | Control | 1.9% | ||
| Add an Image | 2.3% | +0.5pp | +24.3% | |
| Add a Link | 2.3% | +0.4pp | +23.7% |
When we add the data from the French Wikipedia, the constructive retention rates for the Add an Image group changes to 3.7% on desktop and 2.3% on mobile web. For the Add a Link group the constructive retention rates are 3.6% and 2.2%, respectively. As mentioned above these minor differences are not statistically significant.
Controlling for first-day activity
[edit]In our analyses of retention we typically control for the amount of activity (i.e. the number of edits) the newcomer makes on the first day. We find, perhaps unsurprisingly, that higher first-day activity is strongly associated with returning to make additional edits, a finding that's consistent with all previous analyses we've done. In addition, we also find that with that control in place, the differences in retention between experiment groups disappear. This is also consistent with previous analyses, such as our Add a Link experiment from 2021. We have previously hypothesized that the Growth team features, such as structured tasks, have a positive impact on activation, and that this impact sustains through the retention period. The results from this experiment adds additional support for that hypothesis.
Edit volume
[edit]
Like we did for activation and retention, we also focus our analysis of edit volume on the Article & Talk namespaces. In addition, we limit this analysis to only counting constructive (i.e. non-reverted) edits. We do this partly because fitting the models for edit volume takes a long time (24–48 hours), and also because the activation and retention analyses were showing positive signals for these types of edits.
We find that the constructive edit volume during the first two weeks after registration for newcomers who registered on the desktop platform and get the Add an Image structured task increases by 3.0% compared to the baseline in the control group. For newcomers who registered on the mobile web platform, their edit volume increases by 21.8% compared to the control group baseline. Newcomers who get the Add a Link structured tasks make more edits on desktop than both of the other groups with a 6.6% increase relative to the control group's baseline, while those on the mobile web platform make slightly fewer edits than the Add an Image task at a 18.0% increase compared to the control group.
| Platform | Experiment group | Average number of edits | Difference | Relative difference |
|---|---|---|---|---|
| Desktop | Control | 0.324 | ||
| Add an Image | 0.334 | +0.010 | +3.0% | |
| Add a Link | 0.345 | +0.021 | +6.6% | |
| Mobile web | Control | 0.233 | ||
| Add an Image | 0.284 | +0.051 | +21.8% | |
| Add a Link | 0.275 | +0.042 | +18.0% |
On the desktop platform, this corresponds to the following counts per 1,000 registration:
- 1,000 newcomers without Growth features would make 324 constructive article edits.
- 1,000 newcomers with Add an Image would make 334 constructive article edits.
- 1,000 newcomers with Add a Link would make 345 constructive article edits.
For newcomers who register on the mobile web platform, we get the following counts:
- 1,000 newcomers without Growth features would make 233 constructive article edits.
- 1,000 newcomers with Add an Image would make 284 constructive article edits.
- 1,000 newcomers with Add a Link would make 275 constructive article edits.
These increases for Add an Image and Add a Link reflect that these tasks increase both the likelihood that a newcomer makes an initial constructive article edit and that some newcomers go on to make multiple edits.
When we add data from the French Wikipedia and compare the two structured tasks to each other, we find comparable edit volumes for both groups as shown in the figure.
Revert rate
[edit]
When it comes to reverts, we again focus on the Article & Talk namespaces like we've done for the other measurements for exactly the same reasons. We also only measure revert rates for users who make at least one edit during the first two weeks after registration, because it does not make sense to talk about revert rates for an account that hasn't edited. Lastly, we do not separate this analysis out by platform of registration. Unlike the other analyses where we found significant differences in result between the desktop and mobile web platforms, our models did not identify a similar difference for revert rate. One possible explanation for this is that our models control for the number of edits made, a factor that is strongly connected with revert rates (e.g. those who have edited more tend to have fewer reverts).
We find that newcomers who get the Add an Image structured tasks have a revert rate that is 3.3% lower than the baseline. Newcomers in the control group have a baseline revert rate of 29.9%, while those in the Add an Image group have a revert rate of 28.9%. This difference of -1.0% corresponds to a relative decrease of -3.3%. Those who get the Add a Link task have an even lower revert rate of 28.5%, a difference of -1.4% that corresponds to a relative decrease of -4.7%. That the Add a Link group has a lower revert rate correlates well with the findings from our 2021 Add a Link experiment about edit volume and revert rates.
When we add data from the French Wikipedia and compare the two structured tasks, we find no significant difference in revert rates between them, as shown in the figure.
Next steps
[edit]- Disseminate experiment analyses to highlight that structured tasks serve a broader purpose beyond individual edits. Their goal is to enhance the influx of new editors and encourage their sustained participation. Engaging a new cohort of editors is crucial to make Wikipedia and Wikimedia projects multigenerational in their scope (T362129).
- Invest in a more scalable framework for Edit Check and structured tasks to help provide a variety of community configurable tasks for new editors (T360591)
- Investigate new ways of offering people edit suggestions within Visual Editor (T360489).
- Investigate how to relieve some of the patrolling pressure of structured tasks for experienced editors reviewing newcomer edits (T315732).
- Consider creating other structured tasks, like a Copy edit structured task (T315096).
- Explore enhancements to complementary features such as Mentorship (Mentorship Project).
- Explore the creation of additional pathways for newer editors to Level up and advance to more impactful tasks (T316699).
Methodology
[edit]The Growth team deployed Add an Image to three Wikipedia wikis on November 29, 2021: Arabic, Bangla, and Czech. At time of deployment, the feature was only available on the mobile web platform. In late January 2022, the feature was also deployed on the desktop web platform. Four additional Wikipedia wikis got the feature in March the same year: French, Persian, Portuguese, and Turkish. These seven wikis are the ones features in our dataset used in this analysis.
While the feature was deployed in November 2021, our data gathering covers registrations from July 4 through October 31, 2022. This is because we discovered a bug in May 2022 that had been around since March the same year (T309152), and another bug in early July that we caught early (T309152). The presence of these bugs led us to decide that we would restart the experiment and let it run for a few months in order to get good data on the efficacy of the feature.
During the experiment, users were randomly assigned to one of three experiment groups: Control (20%), Add an Image (40%), and Add a Link (40%). In the Control group, newcomers receive no access to any of the Growth features. Users in the Add an Image and Add a Link groups receive access to the Growth features, but differ in what type of tasks they have available in the Suggested Edits module. Users in the Add an Image group have that as the only default task, while users in the Add a Link group has that and the "copy edit" task as their default tasks. Users in the two groups cannot change the type of structured task they have access to, but they can turn on or off the tasks that are available (e.g. copy edits, adding references).
Users can turn the Growth features on and off in their user preferences at any point. If we find indications that they've done so, we exclude them from analysis. We also exclude known test accounts, users who registered through the API (these are mainly app registrations), bot accounts, and accounts that are autocreated.
The dataset for this analysis contains 148,412 accounts registered between July 4 and October 31, 2022. Of these, 22,011 (14.8%) are in the Control group, 62,973 (42.4%) are in the Add an Image group, and 63,428 (42.7%) are in the Add a Link group. These percentages are different from the 20/40/40 split described above because the French Wikipedia did not have a control group during the experiment, meaning that users there had 50% probability of being in the Add an Image group, with the other 50% being assigned to the Add a Link group. We chose to incorporate this lack of control group on French Wikipedia by separately analyzing the two structured tasks, meaning we added the data from that wiki to that from the other wikis with their respective control groups removed.
Our analysis makes extensive use of multilevel (hierarchical) regression models, using the wiki as the grouping variable. This allows us to account for differences between the wikis in our analysis. For example, our activation models are multilevel logistic regression models, which means that they account for the inherent differences in activation rate between the wikis. We also know that editing activity follows a long tail distribution, and therefore model number of edits made using a zero-inflated negative binomial distribution. This model is also multilevel to allow both zero-inflation and the negative binomial distribution to vary by wiki. Lastly, our revert rate analysis uses a zero-one-inflated beta distribution. This is because revert rates calculated across a time window tends to fall into one of three categories: 1) the user has all of their edits reverted (one-inflation), 2) the user has none of their edits reverted (zero-inflation), and 3) the user has some of their edits reverted (resulting in a beta distribution). We again use a multilevel model so that these are estimated per wiki.