Extension:CentralNotice/Design Research 2017/Results

When we created our categories and counted the number of themes mentioned, we were attempting to remove certain elements of personal bias and reduce the availability heuristic. We knew that our participants were not a perfect sample of the wikimedia movement in general. We mainly focused on experts in the tool, those associated with the tool or staff in other departments that CentralNotice overlaps with, either programmatically or  . With this in mind, we can only use these results and rough guidelines and not exact requests or mandates.

Also, in the counting of themes, we developed quantitative data and bar graphs showing the number of mentions of each theme. One might be tempted to assume that a theme that got 30 mentions is twice as important as a theme that got 15 mentions but this is not the case. These are not discrete topics or concepts, our personal judgement could still be an influence and this is not the best sample of participants. The following results are an interpretation of clusters of themes and some rough comparison within categories.

Within each category we also have some recommendations on engineering/design work as well as management needs. Engineering and design needs obviously require those skills. The management needs can be completed by product management, community liaisons and anyone in a coordination role in the movement. At the writing of this document that is just David Strine and Joseph Seddon but we are not opposed to more help here.  

Targeting

We knew that targeting was an important topic for many users. When we set up our interview protocol we put the targeting question at the very end. Our intent was to analyze how many times targeting was mentioned before this question and, separately, to analyze the responses from the direct question.

There was a strong indication that many participants understood the wide reach of CentralNotice. This is one of its advantages over a tool like SiteNotice or something like a mass email to public lists. That very advantage means that it is not suited for all communications. Some consider it too “loud” or a “last resort” for their messages. Also, some participants believed that standard ad systems were better at overall reach and specific targeting.

The most prominent feature requests were about increasing targeting around:

1.) specific user actions or user attributes

2.) specific content or wiki features.

These two themes were popular requests before we asked our targeting question. When we analyzed the targeting question, nearly all of the comments could be divided into these two requests.

“Targeting by user” covered the whole gamut of every demographic attribute knowable and their whole wiki presence: age, location, wiki account details, reading and editing histories etc. This was also seen as a way to make CN less loud and more effective for specific uses. This is also why some participants like more precise communication (i.e. talk pages). It was generally understood that CN shouldn’t interfere or usurp these channels but some improvement in targeting could help to fill the gap between .

“Targeting by content” was slightly less popular, but still a major topic. This included things like: Namespace, category, wiki project, wiki feature.

There is an interaction and trade off between Targeting, Data Availability, Performance and Privacy. Some participants mentioned this relationship on their own. There is a larger discussion on this below.

Engineering/Design projects

• We should check for any current bugs and address them.
• Targeting needs to be part of a consultation around data availability, performance and privacy.

Management projects

• We can offer training on current targeting features. This may help the perception the CN is too “Loud”
• We need to own the consultation for targeting, data availability, performance and privacy

Banner creation and functionality

The major themes that stood out in this category were anything related to new or inexperienced users. The two top request were:

1. Templates, design resources and widgets. Widgets were defined here as elements that can be used to create an interactive banner quickly.
2. Better previews for banners and/or WYSIWYG editing

In-banner surveys ranked highly too. Sometimes this was requested as widget-like function in the banner or it was simply requested that banner work well with other survey tools.

There were general comments about improving the banner and campaign UI. This was also in context to making it more approachable to new or lapsed users.

There were several smaller themes:

• Remind me later and branching, where the user takes and action and the banners are suppressed for some time, or it leads to whole separate flows.
• Clone campaign is a common topic among power users
• Banner version control
• Interaction with CN through an API

Engineering/Design projects

• Create templates, resources and widgets for banner and campaign creation
• Offer ability to preload code from a number of template when creating a banner
• Better previews for banners and/or WYSIWYG editing
  • Also syntax highlighting
• Research surveys in banners
• Clone campaign
• Banner version control
• Research CN api
• Fix CN admin interface header to show across all languages and show templates in header..

Management projects

• We can offer training for current creation tools and in creation
• Offer example banner code (github?)
• Provide examples of banners linking to survey tools (i.e. qualtrics)

Process

There was a lot of discussions about how to improve our processes. Many participants noted that Seddon is doing a great job in the absence of many tools. Some people feel like they ping Seddon, magic happens and their campaign is all setup. Also, along almost every theme, participants felt that other ad systems solved this fairly well. It would seem we have a lot of work to do here for scalability and usability.

The biggest topics were around data availability specifically:

• Campaign performance data available to Admins
• A way to easily understand what is enabled when and where.
  • This may be a master schedule
  • This may also be a way to understand what specific wiki reader should be experiencing at any given time.
• Campaign performance data available to the community

Again, when privacy issues were raised, participants understood that this had to be balanced with the need for campaign data.

Engineering/Design projects

• Data availability needs to be part of a consultation around targeting, performance and privacy.
• What is enabled where and when?

Management projects

• Look at wiki pages and flows
  • Do admins know how to help themselves or get help?
  • Do people know how to become admins?
  • Can community members effectively learn more about CN?
• Can we provide scheduling solutions while features are being built?  
• We need to own the consultation for targeting, data availability, performance and privacy

Other

As it sounds, this was our miscellaneous category. The items are not totally related.

There was a large interest in integration with other communication channels. There wasn’t much of a coherent request but an interest in how that might work.

There were some suggestions for users to have the ability to subscribe to specific types of CN content. So if a user opted out of certain types of content, they wouldn’t see it.

Some participants mentioned security. Some people feel like it might be too easy to vandalise CN banners or campaign. In general people seemed to lack knowledge around the degree of security around the system.

Also, as we expected, we encountered several general negative comments.

Targeting, Data Availability, Performance and Privacy

Targeting, Data Availability, Performance and Privacy are heavily interrelated. When discussed, most participants easily understood the tradeoffs.

Targeting and Performance are very heavily linked. Most implementations of hyper specific targeting affect the performance of Wikipedia readers. It’s not an insurmountable obstacle but at times there are tradeoffs.

Data availability, privacy and targeting are also heavily linked. In general, we must strive to anonymize users as much as possible. Raw CN data could allow someone to identify individuals and their activities fairly easily. This could be very dangerous depending on the type of content and the location of the reader. Even for WMF employees we tend to aggregate data and we have specific data retention guidelines for length of time we can store the data.

We may not be able to expose all the data on CN to all admins (let alone the public) for privacy reasons. We believe we could allow some admins more access depending on the type of process we set up. Regular community members and interested parties may get a heavily abstracted version of the data.

One complication to exposing data is targeting. If we target specific types of users or specific pages, it will be much easier to identify individuals. In this case we won’t be able to tell the campaign admin that any data has even been recorded.

We believe that features around these themes will require heavy collaboration and communication with interested admins and community members. If we develop too many features in one area (i.e. lots of targeting) we might inadvertently hurt efforts in other areas (i.e. performance and data availability)

Proto Engineering Backlog

Targeting, data availability and performance are all important but we need to engage the community in these talks, then design something before we can fully dedicate engineers. Also the “what is enabled where”/scheduling solution need some design attention too. We need to look at the next actionable projects

• Clone campaign
• Create templates, resources and widgets for banner and campaign creation
• Better previews for banners and/or WYSIWYG editing
  • Also syntax highlighting
• Research surveys in banners
• A way to easily understand what is enabled when and where.
  • This may be a master schedule
  • This may also be a way to understand what specific wiki readers should be experiencing at any given time.
• Banner version control
• Security/anti-vandalism  
• Targeting bug bash
• Targeting expansion
• Data availability
• Proper translation integration
• Integration with other platforms (I.E. mobile)
• Subscription

Proto Management Backlog

• We need to own the consultation for targeting, data availability, performance and privacy
• We need to design “what is enabled where”/scheduling
• We can offer training on current targeting features. This may help the perception the CN is too “Loud”
• Negotiate how to share the airways
• Establish criteria for admins to access data
• We can offer training for current creation tools
• Offer example banner code (github?)
• Provide examples of banners linking to survey tools (i.e. qualtrics)
• Look at wiki pages and flows
  • Do admins know how to help themselves or get help?
  • Do people know how to become admins?
  • Can community members effectively learn more about CN?
• Can we provide scheduling solutions while features are being built?  
• Investigate subscription