Growth/Analytics updates/Help Panel experiment plan
The goal of the help panel is to provide users with easier ways to get help without them having to leave the editing context. This can then enable them to complete their tasks while potentially also meeting the Wikipedia community through questions and answers at the community’s help desk. Helping users complete their tasks can lead to an increase editor activation (the proportion of new users who make edits), and potentially also editor retention (the proportion of new users who return to edit a second time), the latter being the Growth Team's overarching goal.
In order to understand the help panel's impact on editor activation and retention, we propose a six month A/B test. During that test, 50% of new registrations on target wikis will have the help panel enabled by default, and 50% will have it disabled. We are likely to be running other experiments on the target wikis at the same time, for example testing variants of our welcome survey. If those experiments require stratified sampling, we will make sure our sampling strategies are modified accordingly.
For more information on the questions we intend to pursue with the help panel, see this section on the project page. For more information on exactly what data we will be recording, see this EventLogging schema.
Variants
[edit]During those six months we also envision testing variants of the help panel to understand how specific interface elements positively affect behaviors inside the workflow of seeking help. We will remember that the stronger our hypothesis that an altered interface will affect activation or retention, the more a test on that interface confounds our longer term experiment on activation and retention.
We will need to prioritize which of the variants to test first, because we will only test one at a time. We will also not run any of the variants until about a month of the larger experiment has run without any of these smaller ones nested inside. This is so that we can learn clearly whether the help panel itself has an effect on new user activation rate.
Leading indicators and plans of action
[edit]The duration of the A/B test is six months because it is impossible to detect changes to new editor retention on mid-size wikis in less time than that (unless we drastically impact retention, but we see that as somewhat unlikely). While we wait for our results we want to be able to take action if we suspect that something is amiss. Below, we sketch out a set of scenarios based on the data described in the instrumentation strategy above. For each scenario, we propose a plan of action to take to remedy the situation.
| Indicator | Plans of action |
|---|---|
| Not opening the help panel | If >75% of users in the target group who see the help panel (i.e. they open the editor) also do not open the panel, we prioritize testing the variant of the panel with increased affordance.
|
| Not using the panel | If >50% of users who open the panel also close the panel without interacting with it, we consider developing a guided tour that introduces the panel after it is opened.
|
| Turning off the panel | If >10% of the users turn off the panel, we cross-reference with Welcome Survey data to see if those are users who report having edited before or if they skipped the survey. If not, we re-evaluate the design. |
| Not clicking help links | If one of the links get <10% of the clicks (amongst the various help links), we consult with community ambassadors on whether the link should be deleted or replaced with something else. |
| Not asking questions | If >25% of the users who open the help panel (without using any of the help links) also do not start on the path to asking a question, we consider developing a guided tour to introduce the question path.
|
| Low quality questions | If >50% of the questions are under 30 characters (or language-appropriate equivalent), we think about whether the design can encourage more detailed questions. |
| Unanswered questions | If >10% of the questions asked through the help panel go unanswered, we reach out to our ambassadors to learn why, and consider a system to ping editors to respond. Or we consider whether help desks are overwhelmed.
If the median time to getting a response to a question goes above 24 hours, we reach out to our ambassadors to learn why. |
| Users do not return to read the answer | If <35% of users return to read the responses to their questions, we talk about better systems for capturing email addresses and notifying users. |
| Does not attempt to edit again after receiving an answer | If >25% of users who interact with the help panel and ask a question and get an answer also returns to the site but does not attempt to edit again, we investigate a random sample of these users and find out what their question was in order to understand if it was unreasonable to expected them to return and edit the second time. |
Status of leading indicators after one month
[edit]The Help Panel was deployed to Czech and Korean Wikipedias on January 11, 2019. One month later we gathered data for all registrations for both wikis up until that point so that we can evaluate our leading indicators and determine whether any of the feature's behavior was concerning. In short, while the evaluation exposes some areas for improvement, we think the help panel's behavior so far is healthy and that it is not having a negative impact on the wikis.
Known test accounts were removed, as were users who turned the Help Panel on or off in their preferences because self-selection into or out of the treatment group violates the equal expectation resulting from random assignment to groups. However, as we will see below, very few users changed their preferences.
| Indicator | Threshold | Czech Wikipedia | Korean Wikipedia |
|---|---|---|---|
| Not opening the help panel | >75% of users in the target group who see the help panel (i.e. they open the editor) also do not open the panel. | 84.7% | 77.4% |
| Not using the panel | >50% of users who open the panel also close the panel without interacting with it. | 36.8% | 39.4% |
| Turning off the panel | >10% of the users turn off the panel. | 0.5% | 0.3% |
| Not clicking help links | One of the links get <10% of the clicks (amongst the various help links). | 3 links | None |
| Not asking questions | >25% of the users who open the help panel (without using any of the help links) also do not start on the path to asking a question. | 69.2% | 93.7% |
| >15% of users who start the question path do not complete it. | 21.7% | 37.5% | |
| Low quality questions | >50% of the questions are under 30 characters (or language-appropriate equivalent). | 30.0% | 16.7% |
| Users do not return to read the answer | <35% of users return to read the responses to their questions. | 36.8% | 50.0% |
There are four thresholds in the table above that are cause for concern, and the list below explains how we're thinking about them.
- Not opening the help panel: for both wikis, the number is somewhat, but not alarmingly higher than the threshold. This makes us feel like we have healthy open rates that have room for improvement. In that vein, we have started to display the help panel in more places, so that users have more opportunities to open it up. The help panel is now being displayed in reading mode in the Help, Wikipedia, and User namespaces. This work was tracked in T215664 and completed by March 6, 2019.
- Not clicking help links: since the analysis showed that three Czech links were getting low traffic, we removed the link to more information about notability and replaced it with a link about how to add an image, as the latter was the most frequent question posted to the Help Desk and the most frequently clicked link in Korean. We also relabelled the link to the guide to be labelled "Quick tutorial", because that is what is used in the Korean Wikipedia, where that is the second most frequently clicked link. This work was tracked in T217391 and completed by March 6, 2019.
- Not asking questions: in retrospect, our expectation that 75% of users who open the help panel (and don't click links) would ask a question was likely too ambitious. We feel comfortable with the rate of questions being asked in Czech, and we are learning that perhaps the paradigm of asking public questions is not a great fit for Korean Wikipedia, given the low rate of questions there. This is one of the reasons we added "search" to the help panel, so that users would have different ways to find help that might fit their own preferences of how to find it. This work was tracked in T209301 and completed by February 25, 2019.
- Starting the question path and not completing it: the absolute numbers for this metric are still low enough that it is hard to say with confidence whether we are notably higher than the threshold. At the time of this analysis, only about 25 people had attempted asking a question. We will therefore revisit this indicator at a later stage.