Crecimiento/Primer día personalizado/Tareas para recién llegados/Análisis de experimento, Noviembre de 2020

Crecimiento

Resumen de funcionalidades

Usar las herramientas

For communities

Actualizaciones

Equipo

Proyectos de la Fundación Wikimedia Newcomer tasks experiment analysis Results from controlled experiment testing efficacy of Growth features. Grupo: Crecimiento Inicio: 2019-11-21 Fin: 2020-05-14 Líder: Morten Warncke-Wang (Staff Data Scientist)

Seguido en Phabricator
Task T230174

En noviembre de 2019, el equipo de crecimiento añadió las "tareas para recién llegados" a la página de inicio para nuevos usuarios. Las tareas para recién llegados proporcionan un feed de artículos sugeridos para editar, adaptados a su área de interés. El objetivo era ofrecer a los recién llegados ediciones fáciles que les interesen cuando lleguen por primera vez a la wiki. Nuestra hipótesis era que las herramientas harían más probable que los recién llegados empezaran a editar, aprendieran las habilidades de edición, vieran su impacto y luego continuaran editando.

Para conocer el impacto de las funcionalidades, las desplegamos en un experimento controlado: El 76% de los recién llegados recibieron las herramientas y el 24% no. El experimento duró seis meses y recogimos datos de las wikipedias árabe, vietnamita, checa y coreana.

Resumen de los resultados

En general, el análisis mostró que las funcionalidades de Crecimiento mejoran los resultados para los recién llegados. A continuación se exponen los puntos más importantes.

• Los recién llegados que obtienen las funciones de Crecimiento tienen más probabilidades de "activarse" (es decir, de hacer una primera edición del artículo).
• Creemos que también son más propensos a ser retenidos (es decir, a volver y hacer otra edición del artículo otro día).
• Las características también aumentan el volumen de edición (es decir, el número de ediciones) sin dejar de ser constructivas' (por ejemplo, porque las ediciones se reviertan).

Creemos que estos resultados confirman que las funcionalidades de Crecimiento, en particular las tareas para recién llegados, hacen que éstos editen más y que permanezcan más tiempo en el wiki.

Debido a estos resultados, creemos que todas las Wikipedias deberían considerar la implementación de estas herramientas.

También creemos que estos resultados validan que el equipo de Crecimiento debe seguir trabajando en tareas estructuradas, para crear nuevos tipos de flujos de trabajo de edición fáciles para los recién llegados.

Glosario

• Desde noviembre de 2020, diez wikis cuentan con las funcionalidades Growth. Sin embargo, en nuestro experimento, analizamos cuatro wikis piloto: las Wikipedias árabe, vietnamita, checa y coreana.
• No todos los recién llegados reciben las herramientas de Crecimiento; el 20% elegido al azar sigue recibiendo la experiencia por defecto. El grupo con las funcionalidades es el grupo de tratamiento y el grupo con la experiencia por defecto es el grupo de control'. Los números que provienen de la experiencia por defecto se denominan números de referencia.
• Activación se define como el hecho de que un recién llegado haga su primera edición dentro de las 24 horas siguientes al registro. La tasa de activación de referencia es la tasa de activación con las características por defecto, no con las opciones de Crecimiento.
• La activación constructiva se define como el hecho de que un recién llegado haga su primera edición dentro de las 24 horas siguientes al registro, y que esa edición no sea revertida dentro de las 48 horas siguientes. La tasa constructiva de referencia es la tasa de los usuarios con las características por defecto, no las de Crecimiento.
• La retención se define como el hecho de que un recién llegado vuelva en un día diferente en las dos semanas siguientes a la activación y haga otra edición. The baseline retention rate is the rate for users with the default features, not the Growth features.
• Edit volume is the overall count of edits made in a user's first two weeks. The baseline edit volume is the count for users with the default features, not the Growth features.

Detailed findings

Below are the specific impacts we estimated from the controlled experiment. These are all based on observing 97 755 new accounts on the pilot wikis, between November 2019 and May 2020. For more specifics on the methodology, see "Methodology" below.

Growth features increased activation (making a first article edit) and constructive activation (making a first unreverted article edit)

Activation

For this analysis, we focused on edits to the Article and Article Talk namespaces.

• Activation: newcomers with Growth features are 11,6% more likely to make a first article edit. On our four pilot wikis, the baseline activation rate is 21,6%. The Growth features are estimated to increase activation to 24,1%, which is an 11,6% increase over the baseline.
• Constructive activation: the effect is larger when looking only at constructive activation. Newcomers with Growth features are 26,7% more likely to make a first unreverted article edit. On our four pilot wikis, the baseline constructive activation rate is 16,1%. The Growth features are estimated to increase this to 20,4%, which is a 26,7% increase over the baseline.

This graph shows how we believe the increase in activation translates into an increase in retention. It shows how the Growth features increase activation, and then that causes a similar increase in retention.

Retention

Because retention is much more rare than activation, it is harder to detect changes. In this experiment, we did not detect any changes directly. Instead, we estimate that retention is increased to a similar degree that activation is increased, i.e. by about 11,6%. This comes from the idea that activity during the first day affects activity on the following days, something we account for in our statistical models. Since the Growth features are found to increase the number of users who are active on their first day, and we find no change in the probability that activated users are retained, it follows that we can expect the increase in activation to translate into a similar increase in retention. In other words, the Growth features appear to lead to an increase in retention caused by the increase in activation: some of the users that the Growth features activated would naturally go on to being retained.

The baseline retention rate across the four wikis in the experiment is 3,2%. We estimate that the Growth features increase this to 3,6%.

Growth features substantially increased edit volume by newcomers (total number of article edits)

Edit volume

The Growth features lead to an 22% increase in the number of article edits by newcomers in their first two weeks.  On our four pilot wikis, the baseline estimated edit volume is 1,4, which means that the average newcomer is estimated to make 1,4 edits. Newcomers with the Growth features are estimated to make an average of 1,7 article edits.

In other words:

• 1000 newcomers without the Growth features would make 1400 article edits.
• 1000 newcomers with the Growth features would make 1700 article edits.

This increase reflects both that the Growth features increase the likelihood that a newcomer makes an article edit and that some newcomers make many suggested edits quickly. Some of them even make over 100 edits within two weeks of registration.

Other metrics

We also looked at several other metrics, with less significant findings.

• Reverts: we looked at whether newcomers with Growth features were more or less likely to have their edits reverted. This analysis did not show large or clear results.
• Highly active newcomers: our results have shown that Growth features cause more newcomers to become active and to make more edits. We also wanted to see whether the features lead to more newcomers becoming highly active. We defined them as users making 50 edits in their first 30 days. This analysis did not show differences resulting from the Growth features.
• Thanks: we looked at whether newcomers with Growth features receive more “thanks” than other newcomers. We found similar results to the retention analysis in that we expect that Growth features do lead to more thanks received, but that this is only because they cause more edits. This is not because the features cause newcomers to make edits that are more likely to attract thanks.
• Differences between wikis and platforms: we compared the wikis and platforms (mobile vs desktop). We did not find significant differences in the effect of the Growth features.

Takeaways and next steps

Takeaways

• The features work: the Growth team features work to increase newcomer engagement. This is especially true for the "newcomer tasks" component, which suggests easy edits.
• Confidence in building structured tasks: this gives us confidence that our current work to build more kinds of newcomer tasks, such as the "add a link" task, will increase impact.
• Need for positive reinforcement: the results showed that the Growth features primarily impact activation – getting newcomers to make their first edit – as opposed to retention. The features only seemed to increase retention because they increased activation. The Growth team should think about what we can add to the features to encourage newcomers to return after making their first edits. Thus, we are planning work on "positive reinforcement" this year. This will add milestones and statistics, so that newcomers can get excited about their progress and impact.

Next steps

• Spread the word: we now have increased confidence in the value of the features. Therefore, the Growth team will encourage more wikis to read results, and consider deploying the features.
• Continue the work: this year, we'll continue to focus on adding new types of tasks and providing positive reinforcement when newcomers complete tasks.
• Extend the analysis: now that we have completed this analysis, we're able to more easily run it again in the future. We'll be able to look at how the features impact more wikis, and see how improvements alter their impact.

Methodology

The Growth Team deployed the newcomer tasks module to the Homepage on Czech, Korean, Vietnamese, and Arabic Wikipedias on November 21, 2019. During the experiment, users were randomly assigned to either a treatment or control group. In the treatment group, users received all Growth features (homepage, newcomer tasks, help panel, etc.), while users in the control group received none.

From November 21 until December 12, 2019, the chance of being in the treatment group was 50%. This changed to 80% on December 12, when the team started an A/B test of two variants of the newcomer tasks module.

Users can turn the Growth features on or off in their user preferences at any point. If doing so, they are excluded from this analysis. We also exclude known test accounts, users who registered through the API (these are mainly app accounts), and accounts that are autocreated.

The dataset for this analysis contains 97 755 accounts registered between the start of the experiment and May 14, 2020. Of these, 23 529 (24,1%) are in the control group and 74 226 (75,9%) are in the treatment group.

Our analysis makes extensive use of multilevel (hierarchical) regression models, using the wiki as the grouping variable. This allows us to account for differences between the wikis in our analysis. For example, our activation models are multilevel logistic regression models, which means that they account for the inherent differences in activation rate between the wikis. We also know that editing activity follows a long tail distribution, and therefore model number of edits made using a zero-inflated negative binomial distribution (again using a multilevel model).