Wikimedia Apps/Team/Android/Image Recommendations

Background

In May 2021 the Android team built an minimum viable product of Image recommendations. The goal of the MVP was to determine if it would be a good task for newcomers, especially on mobile. The results were generally positive resulting in the Growth team introducing the feature on Web (Desktop and Mobile) on select language wikis. The Structured Data team also released a version of an image suggesting tool targeted at experienced users across all Wikis on web via notifications. Due to both tools having generally positive results, and to meet requests of bringing the feature back to the app, we are re-releasing Image Recommendations. Instead of the tool validating the accuracy of the suggestion like the original MVP, this version will enable experienced editors to place suggested images into Wikipedia articles.

Feature Requirements

Essentials for Version 1

The entry point for users should be located both on the 'Suggested Edits' screen and within an article.

'Image Recommendations' task should be positioned immediately below 'Article Descriptions'.
A new task indicator should be available for users who have already completed at least one suggested edit. This indicator should disappear upon their second visit.
Users should be able to add captions and alt text
Users must have an accessible avenue to report feature-related issues, which will lead them to our support email.
A link to the FAQ page should be included within the task.
Measures should be in place to prevent users from repeatedly confirming edits within short intervals (less than 5 seconds) to guard against misuse.
The first time a user opens the task they will be guided through an onboarding process for the feature.
Users should have the ability to access the full article and image metadata.
Users cannot skip tasks without providing some form of feedback (Yes, No, Not Sure).
Users must be able to zoom/pinch on the image
Provisions need to be made for scenarios when images run out of tasks or tasks are unavailable in certain languages.
Suggested Edit should not be available to users with less than 50 edits

Nice to Haves

Enabling users to filter tasks by topics. Incorporation of subtle positive reinforcement messages (displayed in dark mode).

Out of scope

Addition of section images.

Target Wikis

While we welcome feedback from everyone, we are especially interested in hearing from:

Spanish and Portuguese Wikipedia editors within Latin American and Caribbean countries.
Hindi Wikipedia editors residing in India.
Persian Wikipedia editors distributed across the diaspora.

In our study, we are committed to fostering a balanced and diverse group of testers. To that end, we aim for a broad spectrum of gender and age representation, ensuring that all perspectives are well-captured and accounted for.

User Stories

As a Wikipedia Android app user with a small screen and inconsistent internet, I would like to evaluate images and determine if they should go into an article, to contribute to Wikipedia articles that are in need of more content.
When I am using the Android app, I want to be able to add images with one hand to many articles on my mobile device, so that I can be productive while riding the bus in Bogotà, and listening to music.
When I am reading an article about Allameh Jafari Bridge, I want to learn that I can add an image to the Persian language article, so that I can enhance articles that I find interesting.
When I try to add an image to an article and fail, I want to learn how to do this task, so that I can build my confidence before going on to adding image that I want as my intended edit.
BONUS: When I am recommended an image for an article in Bengali and I don’t believe the image is a good fit and I am physically next to the subject of the article, I want to be made aware there is a Commons app, so I can upload the appropriate image using the Commons App

How will we know we are successful?

Validation

KR 1.1: 2000 articles have images in a 30 day period
KR 1.2: Average at least 8 edits per session per unique user
KR 1.3: 15% of Suggested Editors try image recommendations task
KR 1.4: 70% of those users complete the task again on a separate day in a 15 day period
KR 1.5: Reject and Accept rate does not deviate from Mobile Web or MVP by more than 10%
KR 1.6: DAU of Suggested Edits increase overall

Guardrails

KR 1.1: Feature does not worsen gender or geographic bias
KR 1.2: Less than 5% of users report NSFW or offensive content
KR 1.3: Users spend at least 10s evaluating a task before publishing it
KR 1.4: Bounce rate does not exceed 50%
KR 1.5: At least a 35% task completion rate
KR 1.6: Revert rate does not exceed 18%

Curiosities

KR 1.1: Do these numbers differ by language or user tenure
KR 1.2: If this is a user’s first suggested edit, do they go on to try others?
KR 1.3: Feature perception by geographically underrepresented groups on large language wiki
KR 1.4 At what point in workflow are most frequent dropoff events?

Designs

Below are initial designs for the feature:

Discovery of Image Recommendations
Image Recommendations
Onboarding to Image Recommendations
Contextual tooltips onboard the user to the task
Contextual tooltips onboard the user to Image Recommendations 2
Contextual tooltips onboard the user to Image Recommendations 3
Image Recommendations Feed
Image Recommendations Rejection dialog
Image Recommendations Overflow menu
Image Recommendations Sheet Collapsed
Image Recommendations Image Details
Image Recommendations File Page
Image Recommendations Image Detail Active
Image Recommendations Image Detail Filled
Image Recommendations Preview
Edit summary
Edit summary active
Edit summary loading

Next suggestion

Updates

June 2024

We added a check to ensure that images are only added automatically to infoboxes for English Wikipedia through the Image Recommendations suggested edit. T365252

We improved the image syntax in image recommendations so that it always follows the wiki’s default alignment. T365176

May 2024

We clarified the language on the image rejection reason screen in response to a Village Pump discussion T362935.

January 2024- Results after 30 days

Added the capability to copy and paste text within the image editing panel so users can reuse text between captions and other fields as long as text is under 50 characters.T335644

The Add an image suggested edit task has been available to users in the Android App as of September 2023. The findings from our 30-day Analysis for the period of 10 October 2023 to 09 November 2023 can be found below. Results have been positive overall, and we are currently working to implement this task in the iOS app as well.

Validation

KR 1.1: Goal: 2000 articles have images in a 30 day period
- Actual: 2118 articles had added images across 25 Wikis in a 30 day period

In descending order, the wikis with the most images added in a 30 day period
Language	Count of Edits
English	1124
German	361
Russian	113
French	102
Italian	75
Turkish	62
Chinese	54
Portuguese	51
Persian	26
Japanese	24

KR 1.2: Goal: Average at least 8 edits per session per unique user
- Actual: The average daily edit/editor was 6.5 average edits per editor in 30 day period. Overall the average is 13.8. The MVP had a higher edit per session per unique user rate, and we believe this is due to the positive reinforcement that was included in the MVP.
KR 1.3: Goal: 15% of Suggested Editors try image recommendations task
- Actual: 12.7% of eligible Suggested Editors tried image recommendations task.
KR 1.4: Goal: 70% of those users complete the task again on a separate day in a 15 day period
- Actual: 34.5% of those users completed the task again on a separate day in a 15 day period. 18.1% returned on 3 or more days. Although we did not meet our goal, this retention rate is quite high for mobile editing.
KR 1.5: Goal: Reject and Accept rate does not deviate from Mobile Web or MVP by more than 10 percentage points.
- Actual: Accept rate does not deviate from Mobile Web or MVP by more than 10 percentage points, however Android Reject Rate was significantly lower at 28.1% vs Growth’s average 44.2% Reject Rate. The Android Skip Rate is much higher than Growth's which may be attributed to the difference in language used (Skip vs Not Sure). It could also be a difference in audience, this task was for people with over 50 unreverted edits, whereas the Growth task is for newcomers. The MVP did not publish the image so users may have been more confident to accept the image.

Comparing Android Add an Image task with Web Add an Image task
Metric	Android	Growth Add an Image Mobile Web	Android MVP
Image Suggestion Accept Rate	15.9%	38.8%	64%
Reject Rate	4.2%	28.1%	20%
Skip/Not Sure Rate	60.2%	12.7%	14%

KR 1.6: Daily Active Users (DAU) of Suggested Edits increase overall
- Actual: Editor DAU increased by 4.8% compared to September 2023.

Guardrails

KR 1.1: Goal: Feature does not worsen gender or geographic bias
- Analysis forthcoming
KR 1.2: Goal: Less than 5% of users report NSFW or offensive content
- Actual: 0.6% of users reported NSFW or offensive content. Of 496 Rejection Reasons submitted, 1 was category 'Offensive', 0.2% percent of Reports
KR 1.3: Goal: Users spend at least 10s evaluating a task before publishing it
- Actual: Average time spent on the task was 150.8 seconds, and the median time spent was 68.7 seconds. There were no edits made in under 10 seconds.
KR 1.4: Goal: Bounce rate does not exceed 50%
- Actual: Bounce rate was 28.7% (Bounce rate defined as users that click begin the task, and then abandon the flow before publishing their edit)
KR 1.5: Goal: At least a 35% task completion rate
- Actual: 99.4% Task Completion rate (Defined as users that click on Add an image as a task, and then clicks Yes, No or Skip/Not sure (interact with the feature)
KR 1.6: Goal: Revert rate does not exceed 18%
- Actual: Revert rate did not exceed 3.8%, slightly higher than other suggested edits, and lower than Growth Add an Image task.

Edit Type	Revert rate
Android Add Image Suggested Edits	3.8%
Other Android Suggested Edits	2.2%
Growth Add an Image Mobile Web	13.10%
Growth Add an Image Desktop	8.30%

Curiosities

KR 1.1: Do these numbers differ by language or user tenure
- Actual: Broken out by Wiki, Revert Rate ranges between 27.3% (Bosnian) and 3.1% (English). It is noteworthy Bosnian Wikipedia only had 11 Image Recommendations edits at the time of this analysis.
KR 1.4: At what point in workflow are most frequent drop-off events?
- Actual: Caption Entry has the highest drop off rate (task steps not completed) at 32.6%

Funnel Step	Bounce Rate
Image Decision Task	0.6%
Caption Entry Task	32.6%
Edit Summary Task	2.1%

KR 1.5: How often are users adding caption to images?
- Actual: 73.5% of edits were captioned.
KR 1.5b: Learn: What was the revert reason, was it because of lack of caption? How often are reverted edits captioned?
- Actual: No direct relationship was demonstrated between reverts and captioned image status.

November 2023

After receiving feedback from German Wikipedia, we updated the restriction for seeing image recommendations on German Wikipedia to a minimum of 50 unreverted edits on German Wikipedia, instead of global edits. (T351275)

October 2023

Image Recommendations is now available to all editors with over 50 edits (T340668 and T336502).

We did a 15 day check on our data to ensure the feature is on the right track. Below our the results of our 15 day check:

15 Day Data
Goal	Actuals
Image Rejection Rate <29%	11.4%
Edit Over Acceptance Rate <35%	8.60%
Task Completion Rate is not Below 30%	60.8%
Revert Rate <18%	1.60%

Although 992 edits were completed during this period; the chart is based on 456 edits, with the exception of revert rate.

We encourage all editors with over 50 edits to give it a try and provide feedback via the overflow menu in the feature.

We have begun work on the Places (also known as Nearby) on Android feature. Updates about the feature can be found on the project page.

September 2023

Image Recommendations on Android is available in the Beta version of the app for users that over 50 edits. The feature can be accessed by experienced logged-in users by going to the edits tab in the main navigation bar.

August 2023

We expect this feature to be released by the end of September. We are currently code complete, however there are a few bugs that other teams need to work out before we can release.

June 2023

API Underpinnings and modifications to the Growth extension to enable the feature in the app was made available. We also completed the designs found in the section above.