Post-edit feedback

Wikimedia Foundation projects Post-edit feedback An experimental feature to test different kinds of feedback to give after an edit is completed Group: Editor Engagement Experiments Start: 2012-06 End: 2012-10 Team members: Steven Walling, Ori Livneh, Munaf Assaf Management: Terry Chay

See also Editor engagement experiments

This feature is a test of whether giving post-edit feedback to Wikipedia editors enhances their experience, and which kind of feedback is best at motivating them to continue editing.

User experience

Previous

Prior to this project, one of two things happened when a user saved an edit:

1. if the edit is successful, the page reloads in normal "read" mode
2. the edit fails due to an error such as an edit conflict or loss of session data

There was no system feedback which explicitly confirms that the edit was successful and is immediately live, though if they are aware of it, editors may check the revision history.

Tested

After an edit is saved successfully, a notification appeared on the page along with the read-mode of the page.

Confirmation and gratitude (test one)

In the first phase of this experiment, a sample of logged-in editors received a message every time they successfully save an edit. Depending on which experimental bucket they fell into, they received either a dry confirmation that their edit was saved, or a thank you message for making an edit. These notifications will disappear after three seconds and were be dismissable, with the intent that can editors simply scan the page after editing and use the presence of the notification as confirmation of their edit. Since this feature was primarily designed to support new editors unfamiliar with wiki editing, this confirmation message will only be delivered to editors who registered after the start of the experiment.

• Confirmation message: "Your edit was saved."
• Gratitude message: "Thank you for your edit!"

• 
  Confirmation message
• 
  Gratitude message

Results and future plans

Complete data analysis

The results from the test above showed that new editors who were told "your edit was saved" had a marginal/insignificant increase in the volume of edits made, with no significant increase in the burden they inflicted on experienced Wikipedians.

Based on this data, we built and deployed a confirmation message, via Extension:PostEdit, into MediaWiki, both for the current wikitext editor. We also recommended to the team building the upcoming VisualEditor that it incorporate a confirmation.

Historical feedback (test two)

In the second phase of this experiment, a sample of logged-in editors received a notification when they reach their first, tenth, 50th, and 100th edits. The purpose of this notification was to thank editors for reaching milestones in the contribution history, as an alternative to the current experience where they receive no information about how much work they have done, without looking it up using a specialized tool.

This notification looked slightly different than the previous one delivered confirming every edit, with the intent that newly-registered editors will feel they have accomplished something significant on Wikipedia. Since this feature is only designed to support new editors unfamiliar with wiki editing, it will only appear to those with less than 100 edits before the start of the experiment.

All messages will utilize the same general design, but the message text and edit count numbers will be specific to the milestone:

• First edit milestone: "Success! You made your first edit to Wikipedia."
• Fifth edit milestone: "You've made five edits. Keep up the good work!"
• 10th edit milestone: "That's your first 10 edits. Nice!"
• 25th edit milestone: "25 edits! You're doing great. "
• 50th edit milestone: "You’ve made 50 edits!"
• 100th edit milestone: "Congratulations! You've completed 100 edits."

• 
  Example of the first edit message. Text and the number in the icon will change according to the milestone reached.

Results and future plans

Complete data analysis

The results from the test above showed no statistically significant difference in the productivity or quality of editors who received the feedback message about editing milestones, compared to a control group. This was somewhat surprising given that experiments with wiki contributors outside Wikipedia concluded that historical feedback was one of the strongest motivators. Considering those prior results, the failure of this experiment may have been due to a variety of reasons, including that unlike the previous confirmation, the message appeared only intermittently and was easier to miss entirely.

In any case, we are not currently recommending that a feature to support this implementation of the feedback message be built. Edit counter-like notifications may eventually be incorporated in to the upcoming notifications system, but it is a low priority compared to many other types of notifications.

Regarding future tests, it may still be fruitful to test either more "sticky" implementations of feedback that would otherwise be intermittent, like the historical feedback previously tried, or other kinds such as social ranking.

Feature requirements

Confirmation and gratitude (test one)

• A message will be displayed to users in the test every time they successfully save an edit and the page is reloaded in read mode.
• Users will either see the confirmation message or the gratitude message, never both.
• The message will appear in a notification box that is overlaid on the page.
• The message will fade away after three seconds.
• The message will be visible even if the user is returned to a section via an anchor in the link.
• The message will be delivered in all namespaces.
• The message will only be delivered to users who registered after the start of the experiment.
• The message will not appear after page creation.

Historical feedback (test two)

• A message will be displayed to users in the test after they save their first, fifth, 10th, 50th, and 100th edits. (Including if an editing milestone is page creation.)
• The message for each milestone will be unique.
• The messages will appear in a notification box that is overlaid on the page.
• The messages will fade away after no more than five seconds.
• The messages will be dismissible by the user.
• The messages will be visible even if the user is returned to a section via an anchor in the link.
• The messages will be delivered in all namespaces.

Experiment methodology and analysis requirements

See Meta

Future work

A/B testing by the editor engagement experiments team compared delivering a confirmation post-save to a control group and a group which received a message of thanks for editing. This first round of post-edit feedback testing showed that a simple, unobtrusive confirmation message, "your edit was saved", led to a significant increase in the ability of new Wikipedia editors to contribute, without sacrificing quality.

Based on those results, and the fact that confirmation messages are a standard usability pattern, we're building the confirmation message into the editing experience of MediaWiki core. This confirmation message will also be adopted by VisualEditor.

MediaWiki core

We first built this into Extension:PostEdit, and based on the request at bug 48276, it was migrated to MediaWiki core.

Future versions of this may build upon or replace the mw.notify method (aka "bubble notifications"), with some enhancements either directly or in custom styling. See bug 40307 for a list of proposed design changes overall, including some blockers for meeting usability requirements.

UX requirements

The following requirements match the successful version A/B tested on English Wikipedia. Requirements and design for VisualEditor may vary in order be consistent with that project's design.

• A confirmation message declaring "Your edit was saved." will be displayed to users every time they successfully save all edits, and the page is reloaded in read mode. Since this enhancement was tested with new editors and may be an annoyance to those who expect no confirmation other than page reload, this feature will only be enabled for new or anonymous editors.
• The message will appear in a modal that is overlaid on the page, so that it will be visible even if the user is returned to a section via an anchor in the link.
• The message will fade away after approximately three seconds.
• The message will be dismissible by the user.

Alternative designs

The following ideas (please expand if you like!) are options for future improvements to test. They have not been priortized, but should be documented so we don't lose them.

Confirmation immediately after save/before page reload

Instead of delivering the confirmation after page reload, one idea to test is providing confirmation after successful submit, stalling the page reload function to let people decide to view or edit. The goal of this version would be to create a greater affordance for follow up editing (See mockup). We haven't explored the technical challenges of this approach yet, so keep that in mind.

Other designs for confirmation modals

Munaf Assaf, UX designer on the E3 team, produced a wide variety of options we considered (See gallery below). Ultimately we went with the fairly standard and international option of a green checkmark, but a future design review may bring up new needs in the context of icon standardization.

• Alt design mockups
• 
  Confirmation immediately post-save
• 
• 
• 
• 
• 
• 
• 
• 
•