Wikimedia Apps/Team/Android/Machine Assisted Article Descriptions/th
Project background
Since 2017, the Wikipedia Android app has offered pathways for adding short descriptions. In 2019, the team released a short description adding tool to app users that are logged in. You can read more about the current state of the Suggested Edits tool for adding Short Descriptions here.
In March 2022, the Android team created and increased a gate for editors to be eligible for editing short descriptions, as a result of feedback received from English Wikipedia users regarding a desire for users, especially new users, to improve the quality of their article short descriptions.
In May 2022, the Research arm of the Swiss Institute, also known as EPFL, built a machine model called Descartes to suggest short descriptions for Wikipedia articles. After testing the Descartes model independently, EPFL reached out to the Wikimedia Foundation offering the model to Wikipedians to aid editors in the creation of article short descriptions. With consideration of requests to improve the quality of short descriptions, especially from new editors, the Android team determined that it could offer machine suggestions using the Descartes model to support users, if and only if the Descartes tool yielded similar promising results with Wikipedians that it did in EPFL’s initial testing.
In January 2023-April 2023 the team built the UI and client side changes to the Descartes model, while putting in guardrails for risks, then deployed a modified version of the model mid-May 2023 through mid-June 2023. The version of the model users saw in the Android app included quality controls. Additionally, suggestions were concealed behind an affordance where users had to actively click to view machine suggestions then decide to leverage one or manually type out their short description.
By the end of June the feature was removed from the app. In Mid June thru the end of July, the Android team conducted outreach to various language communities inviting them to serve as graders and patrollers for the edits produced through the tool during the experiment duration. Volunteers that responded to the call for support via onwiki email participated in the experiment as evaluators. Experienced editors evaluated what app editors published during the experiment time with machine suggestions, in conjunction with human generated short descriptions. Volunteers were encouraged to revert errors. Volunteer grading ended in early August 2023.
After an initial review of the data, the team had enough insights to suggest migrating the Descartes model from a temporary holding space on Cloud VPS to LiftWing as a permanent host space. After more in-depth data analysis from our team’s data analyst, our research team and the EPFL research team, we felt confident enough in the results to offer the a modified version of the model permanently in the app as suggestions to human editors, only after approval from language communities and having sessions to understand if further changes could be made to improve the model to meet its intended purpose of supporting editors with writing higher quality short descriptions.
Current Status
As of August 2024, we’ve posted the results of the experiment, what modifications app side changes were made to the cloud hosted version of the model for quality purposes, and offered suggestions of if the language community should consider adopting the feature. We will be conducting outreach in September 2024 through November 2024 to inform the Android product team of what further improvements can be made and if communities would like to adopt the feature. While we have recommendations of if the feature might be useful based on data, adoption is a decision left solely to language community members. We recognize short descriptions are handled differently across language communities and steps towards ensuring articles have short descriptions over the years have evolved, which is why we are taking a community by community outreach approach.