What, specifically, are the goals of the New Editors project? And what are the numerical measurements that will help us track success on those goals? Daisy, Pau and I met today to try to hash out that latter question in particular—partly because putting metrics tests in place takes time.
Nailing down the specific goals of the New Editors project is not as obvious as it might seem at first. After a lot of discussion, we’d like to posit that that the goal of the New Editors project should be to help new editors progress from a state of wiki ignorance to some (definable) state of wiki experience.
In other words, it’s not our brief to make people experts editors, or to ensure that they contribute over the long term—as valuable as those goals may be. The interventions we pick should be aimed at getting users through their initial period of vulnerability, over the hump, out of the danger zone—to successfully deliver them, in effect, OUT of the state of being “new editors” and into some level of mastery (where they don’t know all the answers but they know how to figure things out). If we agree on that general goal, the questions become: what level defines the upper limit of “newness” for an editor, and how do we measure progress toward that limit?
Here are some ideas for measurements that will help us track users’ progress from ignorance to basic mastery (based on Google’s HEART framework). We're not suggesting any level of improvement on these as a “goal” yet; they are simply indicators of success worth tracking.
Engagement
- % of registrants who make their first edit (within a defined timeframe?)
- # of editors monthly who make it past their first 10 (non-reverted?) edits
- # of new editors monthly who make it past their first 50 (non-reverted?) edits
- # of new editors monthly who make it past their first 100 (non-reverted?) edits
- # of accounts created (not our focus at this point, but still relevant)
- # of edits a new user makes in the talk (or user talk?) namespaces (not a goal itself, but a good diagnostic tool, since communication has been shown to correlate with success)
Retention
- % of editors who are active in the second 30 days after their first edit
- And/or, the overall # of editors who are active in the second 30 days after their first edit
Task success
- % of edits in the users’ first 30 days that are not reverted.
- % of edits in users’ second 30 days that are not reverted.
These are just first proposals for general new user metrics. I’m sure there will be lots of discussion, and other metrics will need to be established that measure particular interventions we embark on.
And we didn’t talk about the other two elements of Google’s HEART system—”Happiness” and “Adoption.” The latter corresponds with recruitment, which we’ve agreed shouldn’t be our focus until a later point in this process.
Happiness is something that we probably should create performance indicators for, but these will likely be assessed by other means than site metrics. E.g., it would be quite useful to track users' answers over time to questions like: “When you try a new task on wiki, do you feel you know how to find out what the policies around that activity are?” But we can save that for another post. Thoughts?