Wikimedia Product/Inclusive Product Development/Draft Playbook

Background

In fall 2021 through spring 2022, a select number of teams (Growth, Community Tech, Android, Editing, Campaigns, Platform Engineering Team, Web) in Product and Technology elected to use version 1 (V1) of the Inclusive Product Development playbook. The playbook is intended to serve as just one of the tools teams will use to check if their practices welcome members of the community with diverse experiences, ensure their processes and practices are inclusive and equitable. The Beta teams provided feedback about V1 and that input generated a draft version 2 (V2). Draft V2 underwent two rounds of feedback by 30 people resulting in V2 of the playbook below. V2 will be used by all teams at the Foundation that have a Product Manager for the next fiscal year. We will request that teams leave notes about the steps and we will conduct a survey to gain an understanding of how their processes and impact has changed based on V2 of the Inclusive Product Development Playbook.

Technology teams will be interviewed so a version of the playbook for teams without Product Managers is created.

Stages in the inclusive product development framework

V2 Playbook Contents

Instructions and Guidance

Stage ID	Stage	Instruction and Guidance	Supplementary Material(s)	Notes from Team
0	Prework	Make a copy of this Playbook for your team to capture notes while completing the steps in the "Playbook Steps" tab. You should have a fresh playbook per project.
0	Prework	Before stepping through the Playbook check in with the Inclusive Product Development Working Group to understand what resources are available for a team kick off and have at least one team member join the open #productdevelopmentframework Slack channel to share asynchronous insights and learn from other teams using the Playbook.	Playbook Slack Channel
0	Prework	Encourage all team members to read the Playbook MediaWiki page and Practical Application Deck. Teams within the context of this version of the playbook refers to your core feature/development team. Usually comprised of: Product Manager, Engineering Manager, Program Manager, Designer, Engineers, Data Analyst, QA Engineer, Community Relations Specialist.	MediaWiki Page Practical Application Deck
0	Prework	Assign who will be responsible for each step. It is recommended that the Product Manager, Engineering Manager and Program Manager lead assigning who is responsible for tasks.
0	Prework	As you go through the playbook you will notice there are different stages. Below is the explanation of each stage: Strategize: Plan for the work. Align it to the larger strategy. Do risk assessment. Articulate “why” and “for whom?” Discover: The goal of this phase is to identify the problems you'll be solving for. This includes identifying users you’ll be working with, understanding the needs and goals they have and how these experiences fit into their lives. Define: The goal of this phase is to turn research, insights, etc into an addressable and achievable scope of work with a concise problem statement. Develop: Development of experiments or solutions identified in the DELIVER phase, with refinement and further development of the most successful concepts in preparation for the DELIVER phase. This includes design iterations. Deliver: Testing and delivery of final code and/or interface. This could be either for delivering a smaller scaled test or final scaled experience to production. Own: Ongoing monitoring of positive and negative impacts resulting from the product/code delivered.
1	Strategize	If you are a team that isn't user facing (front end team), throughout the playbook be sure to substitute terms like community, for front end team. Everyone is building for someone. If you need support with scoping your who reach out to your team's working group support member, which can be found on MediaWiki.	Support Team Member
1	Strategize	When establishing target audiences, consider selecting more than one specific partner community or group so you can see how changes may impact the groups differently.
1	Strategize	When defining the goals and proposals of a grant, try to make the way the goal is achieved flexible so that there is space and capacity to pivot on a solution
2	Discover	If the feature you build will produce edits, at the stage of reaching out to the research team with your experimentation plan, consider adding edit tags.	Edit Tags
2	Discover	Avoid leading questions in research. When in doubt, review research questions with Design Researcher	Leading Questions
3	Define	If input is part of the feature, consider multiple input modes such as voice, media and text to expand the ways in which people are able to contribute or communicate, when feasible.
1-3 & 6	Strategize, Define, Discover & Own	If a strategy meeting (Shallow Dive, Risk Assessment, Wrap Up meeting, Deep Dive) will include more than 30 minutes of presentation without engagement points. Consider pre-recording the presentation portion and sharing it ahead of synchronous time to allow for asynchronous thought and feedback.
4&5	Develop & Deliver	Before wireframes become Hi-fidelity mockups, engineers should give feedback to the Designer and PM about technical limitations. Similarly, engineers should share early versions using prototyping, patch demos and other relevant tools to share with Designers and PMs early and often.
1-6	All	Things marked important in the importance column are steps that almost all teams should be able to do and are highly encouraged to complete. Steps marked optional are task that would benefit the team's process, but the working group recognize there may be considerable constraints in being able to accomplish it.
1-6	All	Throughout your development process, it is important to return to your WHY with WHO. It can be easy for your team to be pulled in many directions, remembering who you are centering in your project and why you are centering the chosen needs will help your team remain grounded and know who you may have to prioritize. As you learn more about your WHO it is normal to change the how, what or when, but the change should always center your established target users.
1-6	All	For all steps in the playbook that involve more than 2 people, consider what methods are best for collecting input from a large group in an inclusive way. Work with your Program Manager to consider the best way to plan for synchronous and asynchronous touchpoints to get input across perspectives, time zones and abilities.
1-6	All	Where possible, allow for customization. Customization should be considered when it comes to how feedback is given/collected and for the development of a feature.	How customization can lead to inclusion
1-6	All	Ensure at each stage you are accounting for at least one sprint of reactive and unplanned work
1-6	All	When engaging with target audiences, especially from growth markets, ensure definitions are clearly understood by the target audience at a minimum and other groups that may engage secondarily
1-6	All	Try to limit using jargon where possible, and where not possible, ensure your team has a glossary on MediaWiki marked for translation or consult one	Team Practices Glossary
1-6	All	Consult the Inclusive Communication Guide on technical terminology to work towards a working environment that is safe and inclusive	Inclusive Communications Guide Tech News Language Guide
1-6	All	Include representations of different genders, cultures, and backgrounds whenever using images	Inclusive Communications Guide
1-6	All	Ensure you are following our internal Accessibility Guidance at a minimum and consider the A11y checklist project	Internal Accessibility Guidance A11y Checklist Project
1-6	All	Use neutral pronouns or specific actor roles whenever possible for internal and external documentation and interfaces
1-6	All	Ensure you gather views and data from multiple sources. At no point should a team rely on a single team member to represent all people with whom they share an identity (tokenization).
1-6	All	Use a colorblind tester or simulator to diagnose any color contrast issues on mockups, features and documentation, including articles on diff, mediawiki, meta and Medium	Designing for color blindness iOS Plugin Figma Extension Firefox Extension Chrome Extension
1-6	All	At each stage, surface assumptions, especially those that could lead to unintentional exclusion. I.e., assumptions about ability (age, language, technical connectivity etc).
1-6	All	Retrospectives should be conducted at least twice a stage or once a month, whichever is greater. If a team has regular retrospectives and topics from the project are raised during that retro, that meeting will suffice. However, if there are important topics about the project not being raised during a team's regular retrospective period, a special retro is strongly encouraged.
1-6	All	Allow users to sign up for direct outreach when the team is getting feedback. If users give feedback, ensure they know how the feedback will be used and stored in accordance to guidance from the privacy team. Also inform the user that while we can collect feedback in any language, responses will more than likely be in English or machine-translated when more human translation support isn't available.
1-6	All	Instead of using terms like "marginalized", or "global south", be specific about what audience you are targeting. Consider if you are hoping to retain "established" markets or develop with "growth" markets. Be clear about what established and growth means in your context.	Defining Communities Practical Application Presentation
1-6	All	As you move throughout the playbook, and your project as a whole, your team should be learning, testing, questioning and experimenting. If your team learns that the direction of the project should change or a feature should be pulled, the team should feel empowered to take steps to escalate this desire and go through the steps to pivot or pull the project. Starting a project shouldn't be an excuse to finish it, especially if you discover it is the wrong course of action.

Playbook Steps

Stage ID	Step ID	Stage	Importance	Step	Status	Responsible	Supplementary Material(s)	Notes from Team
1	1	Strategize	Important	Review the Organizational and Product Platform strategic plans and goals			Product Platform Strategy Movement Strategy Annual Plan
1	2	Strategize	Important	Create or update DACI for decision-making for different types of decisions and ensure there is representation of all team functions			Product DACI Program Design Template
	3	Strategize	Important	Schedule project launch (kick off) meeting or a team offsite that is intended to establish strategy and goals. Ensure all functions are involved and there are ground rules for the meeting that welcome diverse opinions early enough that dissent can be seriously considered. There should be multiple channels for feedback and communication (oral and written). Asynchronous input must be valued as significantly as synchronous input.			Consensus Decision Making How to organize a remote offsite Ground Rules/Working Agreements Template Future link from Global DEI Team on planning an offsite
1	4	Strategize	Important	Establish baseline of current users and use cases, and opportunities for equity based on your context (PLEASE READ SUPPLEMENTARY MATERIALS). Include existing research (internal and external) of the problem space you want to address.			Establishing Baseline Example 1 Establishing Baseline Example 2 Establishing Baseline Example 3
1	5	Strategize	Important	Leveraging the Wikimedia Accessibility Guidelines, set explicit accessibility goals and/or guardrails at the beginning of the project			Wikimedia Accessibility Guidelines Guardrails Metric Accessibility for Product Managers
1	6	Strategize	Important	Be intentional and clear about who you will empower and engage with (have a clear why), and what that engagement will look like (target audience). If your answer is everyone consider thinking about who you will engage with first for learning and then scale. You should revisit what was learned in Step ID 4 when completing this step			Tarot cards of tech
1	7	Strategize	Important	Establish what partners (internal and external) need to be brought into each phase and give them advance notice of the goal you intend to accomplish if not sooner. Advanced notice should occur as soon as you are aware you will need the team's support, ideally at least a quarter in advance.
1	8	Strategize	Important	Identify and secure dependencies on other teams/departments, ideally through program management coordination as soon as possible. If the identified team or department can't provide a service, product or support that would enable your team to accomplish your project, then the team should descope, pivot or pull the project Note: A dependency isn't the same as a collaboration. A dependency is something that another team or department has (ex. API), that would block a project from moving forward.
1	9	Strategize	Important	Determine potential for technical extensibility beyond your team's use case, and how the use case you are building would impact (positively and negatively) other users or teams. Work with teams and users to determine if the positive impact outweighs the negative impact or ways to limit negative impact of new or modified code.			Software Extensibility
1	10	Strategize	Important	Create a community engagement strategy. Teams should review the Product Guidance on MediaWiki as a starting point for how to engage the community, where applicable. Ensure your community engagement strategy includes a chore wheel or mechanism for going through user/community feedback on a regular basis. Consider requesting a community ambassador by working with your Community Relations Specialist (CRS) to reach out the manager of Community Ambassadors. At this step, the Community Relations Specialist (CRS) is to share what the team's community engagement strategy will be for potentially selected communities (users) to relevant parties including Movement Communications. The Community Relations Specialist should also provide guidance to the team on how to ensure partner communities are not overwhelmed with the team's project plans and if adjustments to who partner communities are should be made. In cases where a team does not have a Community Relations Specialist, they should reach out to their Product Director or Group Manager for guidance on how to ensure their work and outreach efforts does not clash with other teams/departments.			Product Guidance Movement Communications Guide Chore Wheel
1	11	Strategize	Important	Establish “Who are we leaving out?” and be clear if you will include those that are being left out in this iteration or in the future based on your baseline understanding of "established" audiences and "growth" audiences. (What is your Strategy to Scale if and when it is appropriate)			Example of defining intentionally left out markets Example of Scale Strategies
1	12	Strategize	Important	When establishing OKRs be specific about who you are serving (based on previous steps), what assumptions you are making and how you will validate or challenge those assumptions (where possible)			Identifying Assumptions Example Guide for identifying assumptions OKR Definition
1	13	Strategize	Important	Get feedback from all members of the team, users, leadership and other relevant stakeholders to ensure the goals are clearly understood by all. Feedback (positive and constructive) as a reaction to the goals should be recorded and addressed before moving to the next phase. If the feedback was incorporated that should be communicated to the person(s) that gave the feedback, if it was not, the decision maker should equally share why the constructive feedback was not incorporated into the goals.			Gradients of Agreement
1	14	Strategize	Important	Clearly define what barriers and/or knowledge gaps you are aiming to address as well as what new opportunities you are planning to create for your target audience and why your team is uniquely positioned to do so. If your team is not uniquely positioned to address the barrier, consider what Subject Matter Experts you should consult or include to strengthen your unique positioning.			Knowledge Gaps
1	15	Strategize	Important	Once plans are becoming legible, conduct a Risk Assessment to identify and document risks with appropriate internal and external stakeholders			Risk Assessment
1	16	Strategize	Important	Iterate on plan based on risks identified in risk assessment and share evolved plan with appropriate stakeholders
1	17	Strategize	Important	Send an email to privacy@wikimedia.org and talk to human rights@wikimedia.org to share your plans with legal if you are planning to engage with a region you have not engaged with before and ask if there are any issues/considerations for the regions you are planning to work in.			Who to go to for what in Legal
1	18	Strategize	Important	Create list of "Stakeholders" outside of core team and define when they should be engaged
1	19	Strategize	Optional	Review your plans with another PM that is not close to this work, asking them to provide feedback on if the WHO and WHY , as well as desired impact of your OKRs is clear. Also have them provide feedback on if your equity goals could be strengthened and if your engagement plan is inclusive.			DEI (Diversity Equity and Inclusion) Definitions
2	1	Discover	Important	Hold an early team shallow dive into the problem space. Shallow Dive should include a discussion about: - concrete experimentation plans - design research/exploration plans - comparative review of how other teams/orgs/platforms approached this problem space - technical limitations and reasonable creative hacks for this portion of the code base - Performance goals and how quickly a component should be displayed Each component of the shallow dive should discuss impacts on target audiences, take into consideration the risks surfaced from the risk assessment, baseline understandings of your target audience and impacts on equity and inclusion. Each component should also address potential impact on accessibility goals and guardrails. In some cases, engineers may need to have a follow up engineering specific meeting to think through technical limitations of the project. A shallow dive is about exploration while a deep dive focuses on coming closer to solutions. This team meeting should involve everyone involved, even contractors, interns, Ambassadors, etc. who will be significantly part of the process.			Inclusive Design Toolkit Accessibility Guidelines
2	2	Discover	Important	Create project page on MediaWiki			Project Page Guidelines
2	3	Discover	Important	When project page is reasonably stable, reach out to Design Research to request translation support. Should Design Research lack capacity, tag project page for translation. Ensure target languages are prioritized at a minimum, while welcoming translation in all languages.			Translate Page
2	4	Discover	Important	Create a way to track this work that relevant parties are used to accessing. For most development teams, this usually means a Phabricator epic (e.g. 'parent' task that spawns subtasks) or project tag (or both!). There are other ways to accomplish this, though Phabricator is also accessible by the communities, making it a highly preferred option. Ensure relevant parties are aware of the tracking mechanism. For example, this could look like subscribing them to the Phabricator epic or posting the link to a project tag on the project page.			Phabricator Guidance Phabricator Task Template Example
2	5	Discover	Important	Collaborate with the Design Research, Research Team and Data Analytics to evolve and execute your experimentation and research plans. If your team doesn't have an embedded CRS, share your experimentation and research plans with the CRS manager.			Collaborating with Research Getting Research and Data support
2	6	Discover	Important	Share what has been discovered thus far through research and what you are planning to track throughout the project's lifecycle with internal and external stakeholders. Insights should be reflected on the project page, while respecting privacy guidelines and translatable			Privacy FAQs
2	7	Discover	Important	Start execution of community engagement strategy and establish your hypothesis in partnership with partner community members and target user groups that were defined in the Strategize phase. Surface possible biases and assumptions made in the hypothesis and community engagement strategy.
2	8	Discover	Optional	Consult with external organizations or groups, employee resource groups (ERGs), that focus on specific populations to increase understanding of problem space where relevant			ERGs External Orgs Example
2	9	Discover	Optional	Onboard Community Ambassadors
2	10	Discover	Optional	Partner with research agencies that are based in the market you are serving. Aim to support vendors founded and ran by local populations.			Supplier Diversity
3	1	Define	Important	Based on research, set up a deep dive meeting. The deep dive should include: - Proposed solution and more defined scope of project with consideration for different customer journeys - How adding or taking away from proposed scope will impact time of delivery and risks (Technical and political) - High Fidelity Design Proposals with variants - Live feedback from engineers about possible technical spikes - Instrumentation considerations based on Design proposals A follow up meeting led by the engineers may need to talk place to discuss maintenance strategies and who will address maintenance should communities experience degradation. This meeting is not to be confused with the Deep Dive meetings scheduled by Product Leadership. If a Deep Dive with Product Leadership is scheduled when your team is having a Deep Dive as apart of this process, feel free to welcome them in to this meeting, but Product Leadership attendance should not be a blocker to your team meeting to discuss the topics above.
3	2	Define	Important	Update Project Page with Designs and/or prototypes
	3	Define	Important	Conduct Direct outreach regarding Design proposal with Community and encourage questioning of hypothesis, assumptions and proposed solutions
3	9	Define	Important	Review relevant existing software, gadgets and APIs
3	4	Define	Important	Share Designs and Instrumentation Plans with Legal, Architecture, Global Advocacy, Security and Performance if you are creating a new schema, collecting new types of data, retaining data for longer than 90 days, possibly having an impact on user performance or building a project that introduces risks for the user, community groups or organization			Instrumentation DACI
3	5	Define	Important	Conduct first round of usability testing with 2-3 variants of design proposals, being sure to center target audiences and measuring impact on accessibility goals/guardrails. Surface possible biases in user testing.			Checking for Bias in User Testing Design Research Support
3	6	Define	Important	Share outcomes of usability testing with internal and external stakeholders
3	7	Define	Important	Determine which variant(s) to build or A/B test based on usability testing and technical research
3	8	Define	Important	Share the selected variant(s) the team intends to build or A/B test with relevant stakeholders and get feedback. Stakeholders should include (but is not limited to) other departments, target and potentially impacted audiences, design team.
3	10	Define	Important	Actively engage relevant stakeholders for how the feature will be built for extensibility for and/or with other people
3	11	Define	Important	Update measurement plan and socialize with relevant stakeholders based on DACI ensuring there is a feedback mechanism. Surface possible biases in the measurement plan. At least one person from Product Analytics should review a team's Measurement Plan and give feedback on measuring specific populations and downstream risks identified in the risk assessment before this stage is considered complete. If a team does not have an embedded analyst, a draft of the measurement plan should be created, and the Product Manager should attend Product Analytics office hours for advice on how to update their measurement plan.			Product Analytics Consultation Hours Sample Measurement Plan
3	12	Define	Important	Create measurement Phab tasks			Measurement Phab Task example
	13	Define	Important	Share measurement plan with Legal and Global Advocacy			Sample Task and Plan to share with Legal
3	14	Define	Important	If using Artificial intelligence, machine learning algorithms or other systems that automates decision making regarding to users or content, check training data for biases and share your project strategy and mechanisms for checking bias with the machine learning team in Technology. Your bias check should also evaluate the implications of scaling to other groups that are not presently a partner/target community (ex. different language)			How to remove bias in machine learning data https://github.com/EthicalML/XAI https://dalex.drwhy.ai/#examples https://fairmlbook.org/ Language Modeling on Wiki
3	15	Define	Important	Discuss code coverage and automated testing plan. Discuss how user stories could be covered by different types of tests.
3	16	Define	Important	Identify typical browser, bandwidth, platform and device usage for target audience, and develop a testing plan taking into consideration previously defined performance goals. Take extra care to consider audiences who have not been historically represented and if it makes sense to partner with other teams for appropriate coverage.
3	17	Define	Important	Identify if your team needs specialized skills or additional training for development. If special skills are needed, consider one or some of the below: - working with budget holders to enroll the appropriate team member into the necessary training - schedule paired programming with someone in the Foundation that has the desired skills - request someone that has the skills be temporarily embedded on the team - hire a contractor - reconsider scope of project
4	1	Develop	Important	Write code
4	2	Develop	Important	Write copy and QQQ messages			System Message
4	3	Develop	Important	Ask ambassadors or community relations specialist to review text for translatability. Confirm that the text will be reviewed on translatewiki. Ideally features go to localization as early as possible. In the absence support from CRS, reach out to Design Research.			Localisation file format https://meta.wikimedia.org/wiki/Writing_clearly#Writing_for_translation https://meta.wikimedia.org/wiki/Tech/News/Manual#Language
4	4	Develop	Important	Translate text of feature and FAQ page for feature which should be anchored to the project page			FAQ Project Page Example
4	5	Develop	Important	Use MediaWiki best practices and engineering architecture principles for frontend and backend development.			Architecture Principles Extensions Best Practices
4	6	Develop	Important	Adhere to technical standards for accessibility			Technical Standards
4	7	Develop	Important	Design Review of live version in accordance to design standards			Design Standards
4	8	Develop	Important	Test concept via an Alpha, Beta or Prototype with your partner communities and those potentially impacted by feature, in their preferred language. Consult Design Research for Translation Support. Where possible, consider use of hidden deployments for testing.			How to use code snippets
4	9	Develop	Important	Surface technical barriers for target users (example: Performance issues in Japanese because of Rest API). Reach out to other teams and Engineering leadership to identify solutions.			Foundational Tech Requests Board
4	10	Develop	Important	Incorporate feedback from Concept Testing
4	11	Develop	Optional	Include volunteer developers by tagging #goodfirsttask on phabricator tasks that are straightforward, not blockers, and well defined. If your team does not have instructions on how to contribute, consider reading the Supplementary Materials to ensure your team is welcoming to volunteer developers based on the guidance from the Developer Advocacy team.			Developer Advocacy Good First Bug
4	12	Develop	Optional	Tag impacted developers on other teams and request code review			Healthy Code Review
5	1	Deliver	Important	Design review on updated software
5	2	Deliver	Important	QTE Run full test of software according to test plan on Alpha or Beta. Test plan should consider bandwidth, language, color blindness, platforms, and assistive devices.			QA Accessibility Task Template
5	3	Deliver	Important	PM test software in a testing environment before it reaches production
5	4	Deliver	Important	Test instrumentation			QA Instrumentation DACI
	5	Deliver	Important	Update Project Page and write comprehensive release notes. Release notes for apps should go in the app update. Release notes for Web teams can look like Engineering specific updates as opposed to community updates.			Engineering Release Updates Community Updates
5	6	Deliver	Important	Deploy software in production or ideally enable software when using a feature flag.
5	7	Deliver	Important	Test software in production according to test plan
5	8	Deliver	Important	Test instrumentation in production
5	9	Deliver	Important	Inform relevant stakeholders about deployments and ensure there are clear instructions for users to provide feedback			Quick Surveys
5	10	Deliver	Important	Perform usability testing with specific populations, and people from different backgrounds
5	11	Deliver	Important	Create a demo video that is no longer than 5 minutes with a transcript that can be translated showing to showcase the new feature or product
6	1	Own	Important	Gather and share final round of user feedback with other teams and users
6	2	Own	Important	Internal team sync on leading indicators and/or experiment results. Team should explicitly discuss expectations and assumptions compared to reality. This session should also discuss if any risks were triggered that were discussed in the strategy phase or if new ones were surfaced.
6	3	Own	Important	Share leading indicator results and/or experiment results with all stakeholders and community while respecting privacy guidelines. It is also recommended that results are shared during the Monthly Product All meeting. Please reach out to jlayton@wikimedia.org to get your "Product Updates" on the agenda.
6	4	Own	Important	Follow up to ensure data collection ends or maintenance in accordance to policy and team needs
6	5	Own	Important	Establish a “future work” project page or Phab Epic where anyone can add insights/learnings and feature proposals for the future that may not be worked on more immediately. When establishing this space, clearly define what future means for your team and the process in which you will triage incoming requests.
	6	Own	Important	Conduct a project wrap-up meeting with stakeholders. The wrap-up meeting should evaluate who was left out and why, the impact of exclusion (intentional and unintentionally), what users the feature resonated with the most and why, and follow up plans for scaling or removing feature in the future.
6	7	Own	Important	Follow up on plans for scale or feature removal based on the project wrap-up. Ensure plans account for localization.
6	8	Own	Important	Triage post-deployment remaining tickets and clean up feature flags or incomplete code
6	9	Own	Important	Create plan for future technical/design debt and maintenance. At this step, technical documentation should be comprehensive enough that another team can pick up where the original team left off.
6	10	Own	Important	Document and elevate awareness of how goals shifted over time via the project page, and why. Also transparently share plans for the future and engage target audiences.
6	11	Own	Optional	Launch a sprint for technical and design debt
6	12	Own	Optional	Share your learnings with other teams/departments and propose changes/additions to Inclusive Product Development Playbook
6	13	Own	Optional	Surface to leadership what you weren't able to accomplish due to limitations of any kind and what work arounds you had to leverage instead			Quarterly Learning Sessions

V1 Beta Teams Test Results

In May 2022, our Beta testing teams completed a survey and turned in their copy of the playbooks. The information from the surveys and playbooks were synthesized and an external consultant shared recommendations based on this feedback.

Themes

Below are the themes shared between teams.

Time vs Deadlines

Over time it is common for teams to work toward optimization, streamlining their process in order to hit deadlines. Common as it may be, this individualized approach often lacks resilience and as a result of incremental adjustments, teams can become increasingly divergent in the ways that they get their work done. While this isn’t always a bad thing, it can certainly present challenges when attempting broad adoption of new tools and methodology.

Teams that introduced the Playbook earlier in the process seemed to have an easier time adding it into their workflow. Teams that were further along in the process had to pause, and rework their goals in order to include the playbook’s recommended steps.

The playbook helped guide team conversations, but left teams feeling unsure about how much time to invest in specific tasks, versus moving on to the next step required to meet their larger project goal. Teams questioned the rigidity of the process, wondering if all tasks must be completed before they could move to the next phase.

There was a shared belief that the approach defined in the playbook would require much more time. For instance, the time required to define and recruit a representative group of participants takes time. Translating between all relevant languages takes time (and costs money). The more you focus on people and give them time to engage, it all takes more time. This slower pacing can feel at odds with more agile ways of working, leaving people to ask what does a Prototype/MVP look like with this new approach? Not all of the 111 tasks in the playbook require the same level of investment - some tasks seem deceptively simple, only to end up occupying far more time than expected. Providing more guidance around WMF’s expectations for time spent within each stage in the process would help teams more accurately scope timelines.

Clarity & Specificity

In many cases, concerns around time came back to a lack of clarity around the expectation or specificity of the task itself. From determining which guidelines to follow or knowing who to ask if you got stuck, to knowing if you’ve completed a task to expectations, all of these issues could be clearer, if not specified on a task-by-task basis.

Many of the tasks have been distilled down to their simplest form, which puts a lot of work onto the plate of team members looking to unpack the meaning of each bullet point, without always feeling certain that they’ve unpacked it correctly.

Requesting that a team consult with a group, or adhere to a best practice without some further guidance and definition leaves a lot of room for interpretation, and continues the challenge of teams developing their own divergent approach to getting the work done.

Many teams expressed questions related to knowing how their work would be measured. Teams wanted to know how they would be reviewed, in order to learn and improve for next time. Some teams went so far as to request that the tasks be more binary or quantifiable, presumably so that they could self-assess their own performance with the playbook.

While the playbook did provide some direction for folks trying to find answers to their questions, the process of bringing someone in and getting them up to speed can feel like a significant time sink.

Recommendations

The majority of the following recommendations are designed to address the challenges outlined in the themes above. In many cases, when addressing an issue of specificity there is a direct benefit to the concerns around time requirements.

Ensure that every task in the Playbook is as specific as possible.

The more detail you can provide in the task the better, but if that isn’t possible either link to the details or a resource that makes it easier for someone to complete the task in a timely manner.

Clearly define the desired outcome for each task.

If the goal is to have every WMF project be WCAG 2.1 AAA compliant, be clear that will be how success is determined. Checklists are best suited to Yes/No questions, but if a task isn’t as simple as yes or no, help people know they’re doing the right thing by clarifying the end goal of the task.

Conduct a time audit to ensure that all tasks are achievable within a reasonable period of time.

Reviewing each of the 111 tasks in the playbook and assigning each with an estimate for the time required complete would be helpful when thinking about each task’s impact to project timelines. A time audit will be much easier after all of the previous recommendations have been addressed. Reducing ambiguity will remove significant levels of uncertainty, which in turn reduces hesitation, debate, and the need to search for answers.

Sort the binary (quantifiable) tasks and the mindset or qualitative tasks.

This isn’t always clear, even the WCAG 2.1 checklist starts with internalizing a set of four principles. Having open-ended items (like principles) alongside more quantifiable tasks introduces ambiguity and uncertainty. How do you know when to move on to the next item - if you’re not sure you’ve sufficiently or accurately internalized an ‘inclusive mindset’? A particular mindset or deeply internalized principle can be a powerful thing, and essential for guiding someone through the ambiguity of a project, but it takes time to shift your way of thinking and working, and that time rarely fits within the scope of a single project.

Clarify WMF time expectations to ensure a more equitable product design process.

If a team doesn’t know the foundation’s expectations around time, they will resort to the product priorities as they have in the past. If WMF states that a minimum of 20% of every project timeline should be dedicated to explicitly ensuring more equitable product outcomes, that prioritizes the playbook over business as usual.

Reconsider the rollout plan for the playbook.

Rather than introducing the playbook as part of the project kick-off, align the phases of the playbook to the specific project timeline or milestones. Some teams seem to have done this already, the difference would be having someone from the DEI Playbook team with them at each stage. This phased rollout would provide an opportunity for a DEI Playbook team member to be present for reflections on the previous steps, as well as contextualizing the playbook and answering questions to guide the project team into the next stage.

Provide more contextual support to the project teams.

The Wikimedia Product/Inclusive Product Development page is a useful resource but even so, the teams had questions about how best to tackle a task, or who to ask when a team got stuck on something. If the playbook included a list of support resources for each phase, including experts' names and contact, links to relevant documents, link to slack support channel, and the ability to @ the Playbook team in their document. The project team PM should also have a series of brief regularly scheduled standups with a member of the DEI playbook team, this time can be used to help answer team questions or provide feedback and suggestions for helping the team be successful in meeting the goals defined by the playbook.

SUPPORT MATERIALS

Within the recommendations above, there are hints and references to materials that don’t exist or may not exist in a way that fully addresses the need. This section includes some of the materials that will provide more support to teams as they work through the playbook. These materials range from role assignments to digital collections.

• Playbook Marketing Campaign
• Playbook v2
• Slack Support as a Community of Practice
• Examples, Case Studies & Templates
• Resource Library
• Open Office Hours

V1 Playbook Contents

STRATEGIZE

Plan for the work.  Align it to the larger strategy.  Do risk assessment.  Articulate “why” and “for whom?”

REQUIRED

---

• Involve all functions (Product Management, engineering, design, community, etc) in setting strategy, including analytics
• Welcome diverse opinions on what the plan should be, by allowing time for dissent, listening to dissent and allowing multiple channels of communication for feedback (both oral and written)
• Specify DACI for decision-making for different types of decisions
• Ask what partners need to be brought into each phase

• Understand the Organizational and Product Platform strategic plans and goals
• Explicitly discuss DEI as part of the plan
• Consider a wide variety of use cases
• Set explicit accessibility goals at the beginning of the project
• Be intentional about choosing which communities to engage with (have a clear why)
• Coordinate the choice of communities with other teams
• Ask, “Who are we leaving out?”and incorporate either outreach or checkpoints later in the process
• Be intentional and explicit in your OKRs about who, including DEI centered OKRs
• Get feedback on clarity of goals and strategic alignment of these goals with the team, users, expert insight, and stakeholders
• Share plans with other teams and leadership
• Conduct a Pre-Mortem to identify risks
• Iterate on plan based on risks identified in Pre-mortem
• Strategy and planning are constantly evolving through learnings- recognize that the plan may change.  Discuss this with the team.
• Check in with legal on any issues/considerations for the regions you are planning to work in.

SUGGESTED

---

• Review the Community Engagement Guidelines as a team
• Review your plans with another PM, asking them to review it for DEI objectives.
• Incorporate user research when creating your plans
• Plan Discovery research with risk factors from Pre-Mortem in mind
• Use data when creating your plans
• Have a team offsite when starting a new project--focused time is valuable.
• Plan for reactive work in order to ensure space for proactive work.
• Plan to use the Best Practices Playbook for every stage of the process.
• Create the space for differing opinions--ask others “what should be different and why?”
• Review your heuristics for choosing partner wikis
• Ask how you will champion specific populations as you move through the development process
• Share the context of your DEI-related goals and plans with others
• Have a team mission that explicitly states your intentions around developing in an intentional DEI-aware way.
• Talk with people in historically underrepresented communities to find out what they think about existing products, services, or brands similar to yours.

DISCOVER  

The goal of this phase is to identify the problems you’ll be solving for.  This includes identifying users you’ll be working with, understanding the needs and goals they have and how these experiences fit into their lives.

REQUIRED

---

• Identify the problems you are trying to solve--user problems and needs, product problems and inclusion problems
• Consult with user research on the problems you are solving for
• Provide context for any partner teams, including tech teams, that you are working with; help them understand the user experience and potential impact of the work.
• Ask “who else?” as you define the work; be specific.  Eg, How might this problem look different across different socio-economic groups? Who might not be able to use this product and why?
• What is the WCAG 2.1 level AA guidance in this area?
• At the end of this phase, submit your Playbook for the project to [email alias here] [1]
• Avoid leading questions in research, review research questions with Design Researcher before conducting research

SUGGESTED

---

• Consult with external organizations/DEI experts that focus on specific populations to increase learning
• Share insights through written documentation so we have institutional memory of our learnings
• Share with other functions/partners/departments (marketing etc)
• Partner with research agencies that are in the market you are serving

TEAM NOTES

---

[1]    The checklist in this Playbook will be our internal mechanism for tracking the kinds of practices, behaviors and activities our teams are using. We will be comparing what we learn via this reporting channel to the baseline research we did at the beginning of this process in order to track progress toward a more DEI-centered product development process.

DEFINE

The goal of this phase is to turn research, insights, etc into an addressable and achievable scope of work with a concise problem statement.

REQUIRED

---

• Evaluate all of your decision heuristics for biases
• Develop your hypotheses with your partner communities
• Gather feedback on definitions from partner communities, including underrepresented populations
• Use neutral pronouns or specific actor roles whenever possible in the documentation
• Are any images used representative of different genders, cultures and backgrounds?
• Are multiple input modes such as voice and text available if data input is part of the feature?
• Documentation should be easily understandable to laypeople, free of jargon
• If using AI or ML algorithms, then check training data for biases
• Work with analytics on how to measure impact, including specific populations
• Ensure you gather views and data from multiple sources instead of relying on a single team member to represent all people with whom they share an identity.  Avoid tokenization, including for user personas.
• Define code coverage and any needed unit testing plan to be executed as part of the development cycle
• Identified typical browser and device usage for targeted users, including underrepresented communities, and develop a testing plan
• Identify if your team needs specialized skills or additional training for development

SUGGESTED

---

• Examine assumptions that could lead to exclusion.  I.e., assumptions about ability (age, language, technical connectivity etc).  
• Use common language around DEI efforts
• Question your hypothesis: how will you measure underserved populations?  
• Share definitions publicly to get feedback
• Identify and question the assumptions about the feature solutions.
• Identify different Customer Journeys that reflect multiple viewpoints
• Use a colorblind tester or simulator to diagnose any color contrast issues.
• Evaluate and refine your hypothesis through Causal Loop Diagrams
• Hold a design sprint focused on bringing in different viewpoints
• Review data models for inclusivity and privacy
• Consider how legacy terminology in policies (and elsewhere) might require less “insider” synonyms and simplify language where possible [1]

TEAM NOTES

---

[1]    For example “revert” is a technical way of expressing “undo” – which might be a more layperson-friendly way of expressing the same concept. Consider when a legacy term might need to be challenged with the community for the purpose of making the platform more accessible to newcomers.

DEVELOP  

Development of experiences identified in the STRATEGIZE phase, with refinement and further development of the most successful concepts in preparation for the DELIVER phase.  This includes design iterations.

REQUIRED

---

• Look for bias in underlying tools, systems such as AI or ML. Work with the responsible teams to address the bias
• Develop multiple concepts
• Adhere to accessibility design and technical standards
• Test concepts with your partner communities, in their preferred language
• Incorporate community feedback
• Iterate on the most promising designs
• Design according to design standards
• Use MediaWiki best practices and engineering architecture principles for frontend and backend development.  (Vue etc)
• Design with accessibility goals in mind
• Conduct concept testing in the native language of the partner communities you are working with
• Conduct concept testing with your target populations
• Community liaison PR effort to socialize the work to target populations

SUGGESTED

---

• What technical barriers are there to inclusion? Reach out to other teams and Engineering leadership to identify solutions.
• Ask for volunteer developers support
• Conduct concept testing across a variety of populations and languages

---

DELIVER  

Testing and delivery of final code and/or interface.  This could be either for delivering a smaller scaled test or final scaled experience to production.

REQUIRED

---

• Perform user testing with specific populations, and people from different backgrounds
• Beta test with people from different backgrounds, genders, ages and cultures
• Conduct user testing in the native language of the partner communities you are working with
• Evaluate performance for low bandwidth users
• Test assistive devices
• Do we have standard devices that we develop and test on that should be included here?

SUGGESTED

---

• Instrument for downstream risk factors identified in the planning phase

---

OWN  

Ongoing monitoring of positive and negative impacts resulting from the product/code delivered.

REQUIRED

---

• Do retrospectives on your work
• Who was left out of previous work and why?  Is there input into future strategy?
• Plan for future technical/design debt around DEI goals
• Gather user feedback and share it with other teams and users
• Make the results of your work public
• Consider language barriers in feedback--ask community ambassadors, Community Relations Specialists, and research for help in solving this.
• Work with analytics on how to measure specific populations over time as needed
• Work with analytics on ensuring privacy through metrics.  Question if you need to keep measuring specific metrics.
• Elevate awareness of how goals shifted over time, and why
• Evaluate real results compared to expected
• Have a plan for localization as you scale
• What is community feedback?
• Track risk factors identified in STRATEGY phase
• Responsible and respectful exit from intensive community engagement

SUGGESTED

---

• Create a “chore wheel” to gather and share user feedback
• Throughout this process, have a “future work” document where you can collectively write down insights/learnings for the future.
• Establish user or community councils for feedback
• Use user feedback surveys
• Share your DEI Prod Developments efforts publicly to get feedback
• Identify unexpected uses
• Identify unexpected resultsIdentify the audience actually using (and not using) it.  Identify why.

---