성장 팀/첫 날 이해하기
|
Understanding first day
 Gather information about new editors' first 24 hours to understand how they prepare to edit
|
이 문서는 성장 팀의 "첫 날 이해하기" 프로젝트의 일부로써, "편집자여행"으로도 알려진 작업의 목표, 계획, 세부 결정사항과 결과들을 나열하고 있습니다. 대부분의 순차적인 업데이트는 일반 성장 팀 업데이트 페이지에 게시될 예정이며, 대규모 업데이트가 이곳에 기록됩니다.
이 프로젝트는 전적으로 연구개발과 측정에 맞추어져 있으므로, 법무, 보안, 분석, 제품 분석 팀 등의 다양한 부서가 연관되어 있습니다. 사용자의 프라이버시를 보호하면서 동시에 새 사용자의 경험을 개선시키기 위해 필요한 정보를 알아내는 일은 매우 중요한 일입니다. 이러한 목적을 달성하기 위해 나아가면서 이 문서에 적혀있는 내용이 바뀔 수 있습니다.
현재 상태
- 2018-11-19: "EditorJourney" 스키마가 한국어 위키백과와 체코어 위키백과에 배포되었습니다.
- 2019-01-15: EditorJourney 데이터의 최초 보고서가 발행되었습니다.
- 2019-01-16: EditorJourney 스키마가 베트남어 위키백과에 배포되었습니다.
- 2019-07-01: EditorJourney schema deployed on Arabic Wikipedia.
- 2020-07-16: we have decided to stop deploying EditorJourney to new wikis because we are finished with analyzing data that it generates, and we don't want to store more user data than necessary.
요약
새 계정을 만드는 사용자의 절대다수는 편집을 하지 않습니다. 그렇지만 편집을 하는 대부분의 사용자는 가입 첫 날에 편집을 합니다. 어떤 새 사용자들이 첫 날에 무엇을 하는지에 대해서는 전혀 알려져 있지 않습니다. 도움말 문서를 읽는지, 편집을 시도하지만 저장은 안 하는지, 아니면 우리가 모르는 다른 일을 하는지요. 새 사용자들이 어떤 일을 하는지 알기 위해서, 팀에서는 새 사용자 경험을 통해 다음 질문에 대한 답을 알아보려고 시도했습니다. We have little knowledge of what new account holders do on that first day after they create their accounts - whether they read help content, attempt edits that they do not publish, or something else. Knowing more about these initial sessions could help us make the experience better, and lead to more of those new account holders making edits on their first day. To understand what new account holders do on their first day, we have instrumented the new user experience in Czech and Korean Wikipedias to answer these questions:
- 새 사용자들이 계정을 만든 직후 24시간동안 하려고 하는 공통적인 작업 순서가 있나요?
- 얼마나 많은 새 사용자들이 이러한 작업 순서를 따르나요?
- 어떤 작업 순서가 편집을 하도록 유도하고, 어떤 작업 순서가 편집을 하지 않도록 하나요?
- What percent of those new account holders go through each of those workflows?
- Which workflows do and do not tend to lead to edits?
성장 팀은 "이벤트로깅"이라는 기록 도구를 한국어, 체코어, 베트남어 위키백과의 (계정 생성 직후 24시간 이내의) 새 사용자를 대상으로 활성화하여 답을 찾고자 하였습니다. 새 사용자의 경험을 정확하게 알아내기 위하여, 알림, 시각편집기, "가이드 투어"와 같은 다른 도구의 데이터도 사용하기 위한 계획이 준비 중입니다. 이 기능은 2018년 11월에 체코어 위키백과와 한국어 위키백과에 적용되었으며, 최초 보고서가 2019년 1월에 발행되었습니다. To develop a clear view of the new user experience, we also plan connect to and use data from other existing EventLogging efforts around features like Echo, Visual Editor, and Guided Tour. This capability was first deployed in November 2018 to Czech and Korean Wikipedias, and an initial report was published in January 2019.
이 프로젝트에 대한 최초의 공동체 토론은 여기에서 읽어 볼 수 있습니다.
Why this project is prioritized
Our research has taught us a lot about the motivations and needs of new editors, and one of the most important things we learned is that a new editor's very first experiences with editing can be the deciding factor for whether they continue to edit. Most people who create accounts in Wikipedia never make an edit, but of those who do, that edit tends to happen on the day they create their account. In conjunction with the "Personalized first day" project, this project is critical for continuing our learning on those first moments of attempting to edit. We need to know more about how new editors approach their first edits, so we know where to plug in and help. For instance, if we see a lot of new editors reading help content, but then not attempting an edit, we might conclude that help content is important but not helpful as currently written and presented. Or if we see a lot of new editors going straight to opening the editor, but then leaving before saving an edit, we might conclude that the most important place to provide help is inside the editing experience.
We prioritized this project because:
- It helps us increase our learning at this early stage of our team's work with new editors.
- Community members were positive in their feedback on the idea.
- It lays the groundwork for learning whether changes we make to the new editor's experience change their behavior.
- It will be easy to translate and apply to other wikis in the future.
Specifications
This section contains the evolving plans for measurement.
Overall parameters
- Will apply to all new accounts created in Czech and Korean Wikipedias, but will exclude auto-created accounts from other wikis.
- Will be active for the first 24 hours of the existence of a new account.
- At the user level, will record visits to pages in namespaces where new editors might be seeking help, such as the Help, Wikipedia, or User namespaces (and their associated Talk spaces).
- For pages in more sensitive namespaces, such as Article, Draft, or Portal, will record only that the namespace was visited, not the specific page title.
- Data will only be available to people with NDA access.
- Data will be anonymized, deleted, and/or aggregated after 90 days.
Specific questions
The outline below lays out the specific questions we want to answer with this instrumentation effort. Most of these questions will be answered with the new EditorJourney EventLogging schema being built for this project. Some questions, especially those under #4 ("After the editing experience begins") will be answered by connecting with existing EventLogging schemas built to measure existing features.
- Context: How often do accounts get created from the different possible account creation contexts?
- Homepage
- Reading experience
- Editing experience
- Survey: When shown the “Personalized first day” survey right after account registration, do users respond to one or more questions in the survey, or skip the survey altogether and go back to what they were doing prior to account registration?
- After account creation, what are the various common workflows that new account holders go through before making an edit (or before never making an edit)? We want to count the frequency of workflows such as, but not limited to, the following. The reason it’s “not limited to” is that we don’t yet know which workflows we will discover.
- Reading articles first: reading many pages in the Article namespace and then either leaving or editing.
- Learning first: Consuming some sort of learning content and then either leaving or editing. This content is found in namespaces other than Article namespace, or through certain actions that may not be captured in page views or existing schemas:
- Viewing content in the Help, Wikipedia, or User namespaces (among others), including help desks
- Clicking on a link in a welcome message on their own user talk page
- Opening and reading notifications
- Opening and updating account settings/preferences
- Verifying or adding/updating email address.
- Straight to editing: going straight to editing without reading many articles or any learning content
- This can either happen because the account was created from the editing experience, or the reader opened the editor from the reading experience soon after account creation.
- Is this the creation of a new page?
- When opening the editor, some wikis display a GuidedTour or GettingStarted. Did the user click on anything in GuidedTour or GettingStarted?
- Any combination of the above, such as a workflow in which users read some articles, followed by reading a help page, followed by starting and abandoning an edits, followed by a successful edit.
- After the editing experience begins, what percent of users successfully save an edit? And for those who abort their edits, what do they do in the editor before aborting?
- On what page is the attempted edit happening?
- How often do users quickly exit the editor without actually interacting with the page?
- How often do users do a substantial amount of interaction with the page before aborting?
- How often do users switch the type of editor?
- If the edit was saved, how many bytes changed in the edit?
- Was the resulting edit reverted or thanked?
- After saving or aborting an edit, what happens next? (Return to Step 3).
Other schemas
This is a list of some of the other EventLogging schemas we may use to enrich the picture of what new account holders do in their first hours on the wiki:
- ServerSideAccountCreation
- Echo
- EchoInteraction
- PrefUpdate
- EditAttemptStep
- VisualEditorFeatureUse
- Page creation
- GettingStartedRedirectImpression
- GuidedTourButtonClick
- GuidedTourExited
- GuidedTourExternalLinkActivation
- GuidedTourGuiderHidden
- GuidedTourGuiderImpression
- GuidedTourInternalLinkActivation
User flows
In planning out this project's measurement and analysis needs, our team's designer assembled some visual user flows that help us chart out the journey of new users during their first hours in the wiki. As it says in the slides linked below:
"The following user flows document the various pathways newly created Wikipedia accounts can take to become a newly 'activated' editor (after making a first edit), to being retained (2nd edit), through to 'survival' (3rd edit within first 6 weeks). The diagrams note the desired data to be captured at each user interaction, with the intention to both help the Growth team identify the various points for instrumentation, as well as helping visualize the funnel of New Editor retention."
Though the flows extend our thinking to multiple edits in a new user's journey, we have since decided to constrain our measurement to just their first day.
The full slide deck is here, and also linked on this Phabricator task. At right is the image of the full user flow, which is the main slide from the slide deck. Please note that these are presented as an artifact of our team's thinking, but are no longer being updated as our approach continues to evolve. The Specifications section above reflects the current approach.
Analysis
Because the EditorJourney data records much detail on what newcomers do on their first day, we'll be able to ask and answer many questions using the dataset. In this sense, our analysis will never be complete, and we can always use the dataset to answer additional questions about newcomers. This section lists the analyses that have been completed. Community members should feel free to translate any of these reports into their languages. Future analyses will continue to address the list of questions in the "Specifications" section above.
- EditorJourney initial report: the first report using the EditorJourney dataset, addressing the questions of what context newcomers create their accounts from, and how many newcomers do which activities at some point on their first day. Some important toplines:
- This data substantially clarifies our understanding of newcomers: many of them notice outreach, look for help content, and a majority of them open an editor. It's good news that so many are engaged and trying to succeed, and the clarity gives us opportunities to meet them where they are, so that when they go looking for something helpful, they find it.
- Large numbers of users view help or policy pages on their first day: 41.5% in Czech and 27.8% in Korean.
- Large numbers of users view their own User or User Talk page on their first day: 33.8% in Czech and 39.3% in Korean.
- A majority of new users open an editor on their first day – but about a quarter of them do not go on to save an edit during that time.