Growth/Growth 2014/2014-15 Goals
The following document is an overview of the 2014-15 Annual Plan target and roadmap for the Growth team, a part of Wikimedia Engineering.
Current team members (as of June 2014):
- Matt Flaschen (Software Engineer)
- Aaron Halfaker (Research Scientist)
- Kaity Hammerstein (Associate Interaction Designer)
- Rob Moen (Software Engineer)
- Sam Smith, (Software Engineer)
- Moiz Syed (Interaction Designer)
- Steven Walling (Product Manager)
Summary of 2013-14
[edit]The Growth team's 2013-14 Annual Plan target was developed by extrapolating from an early A/B test related to onboarding new Wikipedians. We then aimed for ambitious increases in the rate at which new editors reached 1+ and 5+ article edits per month. Spread across multiple products, these otherwise ambitious gains seemed more reasonable.
Early in the fiscal year, the team's capacity was severely limited as we were reduced to one full-time engineer. In Q3 the team hired an additional engineer, but only by Q4 was the team up to full capacity again. Despite these limitations, we completed much of our product roadmap for 2013-14 fiscal year.
The purpose of a target is ultimately to provide a measuring stick that can motivate the team to help reach organizational goals. In this regard, aiming for a raw number of additional total active editors gained per month was a poor goal. Our product metrics are most accurate and motivating at the weekly or biweekly level, since controlled tests typically run for a week. Waiting for a monthly metric to roll over and see if products had the desired impact was far too slow to be useful. Additionally, translating between gains measured in a controlled experiment and new editors gained after a product launch is difficult. Outside of an experiment, it is harder to say with confidence that new editors were gained solely through use of a particular feature.
In short, we accomplished a great deal considering we were severely limited in engineering capacity until early 2014. However, we did not clearly meet the target, and likely chose a target that was too high-level for the team to focus on as a motivator.
How we created the 2014-15 targets
[edit]Our team's goal is to attract valuable new contributors to Wikimedia projects, particularly Wikipedia. For us, this translates to a focus on arresting the year-over-year decline in total active editors. However, instead of focusing on a single monthly goal, we have broken up our 2014-15 targets in to increases in three conversion rate increases:
- new user acquisition, or the number of new registrations for all Wikimedia projects.
- new editor activation, defined as the rate at which new registrations reach 5+ edits in 30 days.
- and new editor retention, which is the rate at which new editors remain active in the next month.
Read more at Research:Modeling monthly active editors.
Splitting in to these three targets has many advantages for the team:
- We can measure these three targets at a much faster pace than total active editors, which is measured on a monthly basis. If we want, we can look at them on a daily basis, or even in real time.
- We can measure these targets in controlled settings (i.e. via A/B or multivariate tests). This means that we more clearly know if our changes to the user experience actually caused an increase in any metric.
- Last, and perhaps most importantly, we can develop the exact increases by precisely measuring how much a particular metric would need to be improved to arrest the decline in total active editors. This is the case whether we look at each metric individually, with the others held constant, or taken as a whole.
2014-15 Annual Plan targets
[edit]We have selected the following targets to be met by June 2015.
Ambitious case
[edit]The following targets are the most ambitious scenario, in that any of these conversion rate increases will alone lead to a full reversal of the year-over-year decline in total active editors. If all three targets are met, then there will be significant growth in the number of total active editors.
- Acquisition: we will increase (compared to a control) new registrations across Wikimedia projects by 23%.
- Activation: we will increase (compared to a control) the rate at which new registrations become active editors by 23%.
- Retention: we will increase (compared to a control) the survival rate of active editors on Wikimedia projects by 87%.
Mid case
[edit]If we average our gains across all three targets, only a 7.6% increase in acquisition and activation are required to bring total active editors flat, along with a 29% increase in retention. This middle scenario is the most likely outcome this year, though it is likely that acquisition and activation will remain easier to move. Early data from the anonymous editor acquisition work indicates it is easy to exceed a 20% gain in acquisition without any negative impact on activation.
Conservative case
[edit]The most conservative scenario for the year would be based on small gains on top of what we have already achieved. In the past several years, we have been able to achieve up to a 4% gain in acquisitions, measured via our previous account creation user experience tests. For activation, the onboarding new Wikipedians project increased first-time edits by 2%. We have not yet seen any significant gains in month-to-month retention, so we would not predict any gains there without further experimentation.
Together, these are three of our most conservative estimates for impact of Growth team work. Cumulatively, these conversion rate changes would reduce the year-over-year decline from -5% to -3%.