Growth/Growth 2014/Quarterly Planning

The following are notes from the Quarterly Product Planning (Janurary-March 2013) for the Editor Engagement Experiments team. This is not a roadmap, but is rather a description of the team's thinking at the beginning of the quarter. Since our last intensive planning session was in the fall, it was a particularly good time to stop and look farther forward in to the future than the next few development cycles.

Background

This summary of the quarterly plan for E3 was developed by the team in early January. Partially in preparation for the Quarterly review, we collaborated on a followup to our last round of prioritization, which led to the development of our current work.

January-March 2013

Features

Subject to the results of data collected and the associated performance metrics for each project, E3 currently plans on tackling the following features development over the next three months.

Account creation

Rationale: This is a launch and infrastructure project based on A/B testing of account creation that is now complete. The most successful test version has been expanded to be given to all English Wikipedia users for several months, but making a permanent version of the user interface and analytical modifications is a necessity. In addition to the user-facing portions of account creation, we have found a great deal of use in the ability to tag users who come from particular funnel or event.

Breakdown:

• Launch of the new Agora-style UI
• I18n and l10n
• An associated login redesign
• Campaign support, which is three broad parts:
  • User tagging
  • Bucketing
  • MediaWiki tags

Guided tours

Rationale: Guided tours is a general purpose tool for educating users about Wikipedia interfaces and community processes. In particular, it is able to easily be adapted to support other key E3 initiatives like the onboarding process, helping teach users about the mechanics of editing or show them where to find additional tasks.

Breakdown:

• Productization and l10n
• A guided tour for GettingStarted
• Other guided tours in support of onboarding, TBD
• Support for community-created tours and other feature tours

Onboarding (GettingStarted)

Rationale: Onboarding (GettingStarted) is the primary user-facing project of Editor Engagement Experiments. By directly serving potential new editors compelling tasks, this is our most direct route to increasing the base of new editors on Wikipedia.

Breakdown:

• Post-edit task assignment, including any necessary integration with Echo and adding a way for users to navigate back to GettingStarted
• Special:GettingStarted optimization, including...
  • SuggestBot productization (i.e. a recommender system and task repository)

Related features work deferred for now

These ideas came up as options during discussion, but were deferred to either another quarter or depend on data to suggest whether they are good avenues or not.

• Developing onboarding support for key alternate user flows, such as article creation and editing of semi-protected pages
• Support for custom signup experiences and post-registration landing pages via campaigns
• Redesign of GettingStarted in to an editor dashboard
• Work on the wikipedia.org portal to drive signups and aid returning users

Infrastructure, analytics, and metrics

Rationale: the following two pieces of infrastructure are critical to the team's ability to measure the effectiveness of individual products, in relation to the active editors target.

EventLogging

• More robust campaign support (e.g. ?campaign=mobile_signups)
• Real-time monitoring/alerts
• Preliminary support for funnel analysis/automated statistical testing
• Documentation (seriously!)

Metrics API

• Cohort metric visualization (with analytics team support)
• First public release/announcement
• User tag repository redesign and tag update jobs

April-June 2013

Features

The following are our priorities for the user-facing portions of E3's work.

Account creation and login

Current state of the project: making our redesign of account creation and login is a goal leftover from the previous quarter. We expect to soft launch the new versions of both in core before April is over. This soft launch will be via URL parameter for a week or so of testing, then we will make them the defaults pending no critical bugs that arise.

Product and experimentation roadmap: our primary goal for this quarter is to wrap up launch of our redesign and move the product to maintenance mode, while documenting further work that could be done on the form, additional outstanding feature requests, etc. The major items in this list are, in loose order of priority:

1. Respond to any incoming bugs during the soft launch and first default launch
2. Measure the impact of the community statistics we replaced the benefits to signing up with (documentation). The simplest experiment here would be to split test whether they stats are visible or not to viewers of the account creation page. Proper internationalization and localization required that redesign our original take on a benefits list, but we should still quantify the effec of our current choice on signup conversion rates.
3. Measure the impact of the CAPTCHA on (human) users attempting to sign up. (Specification)
4. Support rebuilding the form using HTMLForm, on top of Tyler's login API change. Code review and testing for community efforts here will pay off in the long run for end users.
5. Restore client-side validation as a follow up to the enhacements made by Brion et al. to the account creation API. Our testing proved that client-side validation prior to submission of the form significantly improved conversions and decreased the rate of errors after submission, but not so much that we made it a requirement blocking any rollout of the redesign. (More about this.)

Guided tours

Current state of the project: guided tours has largely reached stability in its feature set, code is being contributed back upstream to the main guiders.js library, and our first major experiment applying a tour to Getting Started was a clear success.

Product and experimentation roadmap:

1. Test applying a generic "how to make your first edit" tour to all newly-registered users when they visit an editable article. We know that the majority of successful contributors complete their first edit within 24 hours of registration. Now that we have proved that tours can increase first time edits in the area of Getting Started, it's time to test this hypothesis. If successful, we could easily roll out a permanent "make your first edit" tour to all Wikipedias where localization is complete, separate from the more difficult problem of Getting Started localization.
2. Simplify the design and user experience. We took a strong first pass at the functionality for stopping, starting, and moving through tours, but there may be ways we can simplify the tour user interface still.
3. Continuing development of Getting Started tours or other new editor-focused tours, as needed

Getting Started

Current state of the project: we've successfully implemented a working replacement for SuggestBot to recommend articles, and launched a second iteration on the landing page design. However, this new landing page with additional choices of task types and articles associated with them proved to be more confusing to first time editors, and reduced our conversion rates from 0→1 article edit in 24 hours.

Product and experimentation roadmap: we responded to the above results by prototyping a new landing page and navigation system for GettingStarted, which should be launched early in the new quarter. We plan to continue protoyping before launch more aggressively, to learn what decision-making models are best for the 0→1 and 1→5 stages in the user lifecycle.

1. Implement current prototype, including the new persistent navigation bar.
2. Design and test other prototypes, with the goal of solidfying the landing page design by the end of the quarter.
3. Quantify the impact of the Echo notifications (already deployed) on new and returning GettingStarted editors.
4. Design and test light gamification aspects, such as progress bars, presenting users with an editing goal, notification reminders regarding their progress etc.
5. Explore personalization of Special:GettingStarted and the tasks recommended, dependent on finalized landing page design and results of other tests aimed at the 1→5 stage of the user lifecycle. Related idea, but likely too large for the scope of the quarter, is exploring a basic task recommendation system for editors outside the GettingStarted funnel.
6. Design and test tools for reactivating previous GettingStarted editors, or directing them to new tasks post completion of one and five edits.
7. Specify localization and internationalization requirements for rolling out GettingStarted to other Wikipedias.
8. Explore designs for a mobile Web version of the GettingStarted workflow.

Other features and experiments

Acquisition experiments: as GettingStarted and GuidedTours reach further stability, and our new account creation experience is rolled out permanently across Wikimedia projects, we should consider returning to experiments that aim to acquire new editors by drawing additional cohorts of users in to the signup flow.

Ideas for targeted acquisition experiments include:

1. Optimizing the signup call to action on the login page. See bug 47392.
2. Semi-protected pages (e.g. transitioning from "View source" to "Edit" and a subsequent call to action)
3. The edit window (e.g testing optimizations for the edit notice call to action)
4. Anonymous editors post-edit call to registration
5. Main Wikipedia.org portal improvements, including login/signup calls to action and search improvements

Infrastructure, analytics, and metrics

We estimate we need the following backend infrastructure and analytics support to accomplish our current and future experiments.

1. Re-implement campaign support for account creation, for tracking the source of signups via URL parameters (see Extension:Campaigns). We disabled this feature in prep for the soft launch of login and account creation, and will need it for future experiments. Next steps may be usertagging with campaign identifiers and support for custom landing pages post-registration
2. Funnel analysis and support in EventLogging. (Potentially very useful but not immediate pressing need.) We need to reinvestigate the viability of funnel analysis, since the reliability of data collection at each funnel step is necessary for making this useful.
3. Error logging alerts. If a theoretically impossible situation should be logged (such as test impressions from a control group), generate an error message via the EventLogging mailing list.
4. iPython Notebook development and support.
5. Agora improvements as needed. For example:

• SASS support in ResourceLoader (we use Compass to process Sass .scss files to generate the Agora CSS). This would make prototyping easier. We should also help finalize a LESS vs. SASS decision in support of the design team.
• Applying Agora to other forms and interfaces, like the edit window, preferences, password recovery, etc.