Wikimedia Research/Design Research/2015 workshop

Design Research Workshop

November 20, 2015

Links

Slides: LINK TO COMMONS
Youtube steam

First half (video quality issues): https://www.youtube.com/watch?v=7mhqtsMQQEs
Second half: https://www.youtube.com/watch?v=_YJFR0VW3SI

Interactive website: http://design-research-methods.wmflabs.org/

Comments

The referenced user testing for UploadWizard led to the Multimedia team writing an in-editor upload tool for VisualEditor (WikiEditor on the way...)
Another important consideration about sample size in usability testing is that if you have resources to put 15 people in front of your solution is it better to do 3 rounds of testing with 5 users each and improving your solution after each one rather than doing a big round with 15 users that allows to catch all issues but having no room to check if the solution to those was effective.
The exercise of completing the spreadsheet was really good! It's not only useful to plan and understand current workflows, but also, to better understand how to collaborate cross-teams (María).
Personas! Would be great if you can give us a guide to how to make our own personas if we don't see them in the list you have created.
http://design-research-methods.wmflabs.org/ is beautifully designed, and seems to contain useful information, but I doubt many people will remember the URL two weeks from now, and the content doesn't seem searchable or editable. Also, Labs doesn't seem a good place for publishing text in the long term (we'll still be interested in this stuff two years from now, right?). How about putting a copy on Meta-wiki or mediawiki.org?

Questions for design research

1. From Halfak: Is there documentation for persona research on MediaWiki/Meta? Where can I find it?

Answer: TODO add links to related resources to Meta/MW

2. what is understood by "the Global South"? And do you think one persona will represent "the Global South"?

Answer: not one persona for Global South (first part of question not really answered...)

3. what is the Ghana research? Can we access that?

Answer: analysis is ongoing, results will be published on wiki (TODO)

Building a product

	Understand	Concept generation	Concept evaluation	Develop	Review	Release	Maintain
Description	Wear your researcher hat. Discover personas, needs, challenges, and opportunities.	Generate many concepts and imagine possible futures.	Evaluate concepts against what was learned in "understand", business needs, technical resources other constraints.	Build, test and iterate on concept.	Review with stakeholders to decide whether to release.	Release user-friendly product that supports foundation's mission and meets our users' needs.	Make sure that our product continues working for our users, and measure impact.
Design researchers	Generative research Get to know users and their context Define problem & opportunities Analyze and synthesize (see patterns, gain insights)	Develop lots of concepts	Evaluate concepts against what we learned from users Evaluate concepts with users Combine similar concepts if possible Define success and how to measure it Match concepts with foundation's mission Consult / share with stakeholders	Build prototypes Refine and build MVP Usability testing / qualitative evaluation A/B testing or quantitative evaluation Iteration if needed	Consult with stakeholders regarding readiness of product If product is not ready, go back to “Develop” scoped amount of time to reach the decision for release	Publish the product Communicate about the release	Measurement of product Maintain product (bug fixes, etc.) Observe need for iteration or next feature set. Potentially retire something based on product performance and more user feedback Triage bugs
Designers	- Putting research hat on, listening to users and understanding audience. - - Look at community channels, feedback channels and figure out things to fix. - Guide questions being asked	- Sketch possible solutions to explore many different directions. -	- Make higher-fidelity representations (mockups/animations/prototypes) to communicate the design ideas internally and externally. - Get feedback from other designers	- Define scenarios and create prototypes to make ideas testable. -	- Check if product spec, interaction spec and visual design spec is done correctly - Get notified about patchsets without screenshots and trying to figure out what is going on.	- Help evaluate how the current solution works (in collab. with Research) - Work with communication team to craft presentation to public	- Respond to feeback and identify opportunities to improve the designs. - Continue to check for bugs
Product Managers	Identify problems to investigate, and user segments that we want to understand better. Prioritize goals to work on.	Brainstorm possible options to solve the problem, connect with possible stakeholders. Communicate concepts and plans to community team, developers, get feedback.	Create and communicate framework for how we evaluate potential solutions, keep process moving, decide which solution to build	Define spec for development. Evaluate tradeoffs between options. Keep the team focused on the most important pieces. Plan iterations so that every release adds value for the user.	Coordinate analysis of impact of solution, communicate results	Announce feature, gather feedback. Communicate learning with team for further development.	Prioritize bug fixes & feature updates into technical workflow.
Community Liaisons	Get clear understanding of the problem the product intends to solve. Define users that the product is being created for. Contact wide spread of communities (all areas of communities, ask for audiences.	Ask the question of "what problem does this solve?", and "whom is this for?" "What do you need to be able to do, as a user" Help users generate their own user stories. Create working groups of users to discuss needs (roubndtables). Conduct IRC office Hours and orther public meetings	Help product owners compare the new idea with similar existing ones, or previous trials that existed on wikis. Define MVP from audience perspective. Are there blockers? Are there power user tools we need to consider?	Get the community updated on planning, data, etc.+ help incorporate community in product development, as far as the process allows. Coordinate users for testing/feedback loops. Discuss data and impacts, review concerns.	Gather feedback and correlate accross different areas of product teams. Cast wide net for feedback, remind users of goals of product. Ensure community (readers or editors) 's feedback from previous step is incorporated. What is go-no go?	Discuss timing of release with different communities. Discuss data and how it correlates to decisions. Ensure a smooth roll out, that doesn't disturb any set of users, or block any other workflow.	Continue to work with the communities on features to add, bugs to fix, performance to enhance, etc + accomodate the needs to replicate the product in different languages, if it was originally done only in one wiki or in one platform.
Program Capacity & Learning	Who are the teams we are trying to serve? For example, we work a lot with program leaders, who run things like GLAM, Education Program, Affiliates, etc. These groups often need learning as well as technical products	We listen to the community. Then, we propose ideas, and we use a diverge/converge method (CPS); where generate a bunch of ideas and then build on those. Once we have lots of ideas, we begin to assess which ones are most feasible or most important	We look at our skills and capacity, contrast them with community members / communities' needs and understand what we are able to build.	We develop tools and resources, but we don't have a process for iteration. We are strong at ideation, and we don't have a proper feedback process. Some difficulties that could bring this up: we don't have enough resources / capacities; our projects are not based on a 2-week model. For example, we have conferences that we can only iterate on once per year.	We are going through this now. We are actually going to be inviting stakeholders who we serve, such as program leader in the community, affiliates, other staff, to feedback on our process and let us know if we are on the right track	Communications! We rely on having a communications plan for much of our work, that involves thinking of the concept or idea to promote, imagine possible responses, and what comms formats and release dates we look forward to having. We use a combination of social media, blogs, mailing lists and meta portal to outreach community members and generate engagement.	Our work has two aspects to maintain, tools that we create and documentation for learning. Much of the documentation might change from one year to the next.
Community Resources	What kind of support and resources to grantees and committees need and want? Define the two types of personas: grant applicants and committee members. Motivation and needs of both. How do goals of WMF and CR stakeholders intersect? Create a shared understanding of the needs and limitations of both groups.	-community consultations to better understand processes like grant applications; -have multiple converations with stakeholders about the meaning of impact; - future scenario: a world where we understand each region in its peculiarity and not as monoliths		Process evaluation after each grant report; survey to grantees. And after a few years we might do a big redesign of the grants programs overall to meet needs
Front-end Engineers	Understand what the feature is. Implement analytics to gather user data.	What tools (libraries, plugins...) do already exist? Can we use existing tools? How can we avoid wasting effort/duplicating work?	Helping understanding of non-devs technical constraints around concepts Maybe even extending ideas with small technical refinements, add-ons. What tools do we select? Select them	Write code, write tests, iterate	Coding conventions, coding guidelines, make sure implementation respects all the acceptance crtieria, verify security, verify performance	Ensuring documentation is reflecting released product	Fix bugs
UX Engineers	Understand the issue (need & challenge), look if issue has been come up elsewhere before. Look to gather all perspectives on it.	Work with design to make sure concepts are not too far from reality without stopping early concepts	Early addressing UI/UX/tech pitfalls, that might end up in product dead ends later. How can we build this prototype fast?	Build prototypes, AB testing and other evaluation	Consult with stakeholders, users, analytics, and determine what works and what doesn't	Announce features, results and reports on UX-related channels, listening to feeback, stepping back & then ensuring that product is documented well.	Iterate
Research and Data	Literature review; Exploratory analyses (scale & frequency); Ethnography (casual -- usually); Document on Meta.	Sketching; Story telling; Proposal writing; Exploratory analysis (correlations); Document on Meta;	Review of available technologies; Replication of past analyses; Design and review methodology; Discuss plans with other researchers, stakeholders, and specialists; Document on Meta;	Calibrate measurement strategy; Take measurements; Run simulations; Develop technological intervention; Design & run field experiment; Recruit participants; Perform offline analyses; Compare methodologies; Document on Meta;	Provide data and present findings in a way and in language that can support decision-making. Be heard. Iterate on explanations for observations. Propose future work. Perform followup analyses (proposed by stakeholders). Document on Meta.	Provide data and present findings in a way and in language that can support the release; Write up manuscript for future reference; Publish in peer-reviewed journals. Give talks to broad audiences -- encourage others to explore "Future work" for us. Upload datasets to public repositories; Document on Meta.	Record data and do longitudinal analysis; re-experiment as more data comes in and verify initial/preliminary findings; Summarize supported hypotheses into Theory. Support external researchers to replicate/extend our work. Document on Meta.
Communications	Help define audience segments Build understanding of audience media habits - What do they like to read/watch? - What media outlets and personalities are trusted? Assess Wikipedia Brand Awareness and Perceptions (familiarity, trust, concerns, hopes, etc.)	Help identify audience needs and develop "user challenge" narratives (like Community Liasions "user stories")	Help validate potential concepts through social channel publishing Evaluate communications resources needed to support product launch and support	Continue A/B Messaging testing on social channels to build robust understanding of audience reactions Run product messaging/positioning workshop to guide ongoing communications Brainstorm Creative Materials and Language to succinctly communicate product attributes Ensure language localization of messages (and identify problem areas in non-translatable terms) Work with ComCom list to storytell across regions	Identify pathways to audience based on known social media and press preferences Develop communications strategy needed to reach audience on key channels	Support product launch with multichannel communication strategy, which may include: - announcement/media outreach - social media promotion - supporting collateral: video, product descriptions, FAQs	Continue messaging to reinforce product needs and continue collecting user feedback
Team Practices Group	Help teams execute, keep work coordinated with cross functional stakeholders, promote general communication, and sharing metrics.