Topic on Talk:WMF product development process/Archive 3

In a world of infinite resources, once the WMF decided that a specific proposal was a good idea, it would move that concept into the planning stage, and continue on through to Release and Maintenance. But, of course, we don't live in that world. So, the question: what are the factors that go into deciding what moves forward, and what is deferred?

Related to the question of who decides is estimating the cost of implementation. If two proposals have about the same level of expected benefits, but one will cost two or three times more to implement, then the WMF should probably implement the lower-cost proposal, and possibly defer the higher-cost proposal. (Note: the Concept phase now includes "technical debt", which is the eventual consequences of any system design, software architecture or software development within a codebase. That's not at all the same thing as cost: a proposal might require a large number of staff-hours to implement, and yet be relatively easy to maintain, and affect few other parts of MediaWiki code.

In short, if the WMF truly wants to be transparent, there should be a visible decision process. That doesn't mean that the community has to vote (or !vote) on most (or even any) products being developed, but it does mean that decision-making should be more than a group of developers saying "Okay, what do we want to work on next?", with regard to feature development.

What might this look like, on the chart? Perhaps a Prioritize phase:

• List of benefits
• Estimate of costs
• Synergies with other product development
• Setting priority: Very high to very low

Obviously, "benefits" typically would not be described in dollar terms (WMF is not a business). But WMF should make clear who it sees as benefiting: novice editors, editors using specialized tools, wikignomes, prospective editors, etc. That in turn encourages a discussion about the relative merits of different proposed products.

Thank you for this., @John Broughton :) I agree that we can say that the prioritization question is one that has been asked frequently and one that has potential to be included within this process, though it may also make sense to have the prioritization/decision be separate item (at least, as far as "decision making" is concerned.

As far as the bullet points you mention, I think those are a great start, and want to draw @WMoran (WMF) and @JAufrecht (WMF)'s attention to them.

I don't know that this process page is the right place to hash out prioritization (what do you think?), but it's definitely worth discussing. It's been raised several times in the past (the link is only one example), but I don't find a dedicated page here on MW. Before making one more page (there are, er, so many), I'll poke around. It would be great to have a clearly defined description of how projects are decided on and how they are prioritized.

I agree that the prioritization step may be the dominant part of this lifecycle for the Foundation, and it's the least well-understood step. I understand the "<---- Analysis & Feedback" back-arrows to encompass that, but in terms of pixel ink, they are much smaller now than they should be, and they don't have specific Task names. One big problem here is that the real work of prioritization is saying "no" or "not yet" or "not much" or "nobody else agrees with you" to most things (the visual editor backlog is at least three to five years long, and more likely decades). There are more incentives for each individual person involved in the process say or believe "yes" or "maybe", even though this leads to an unrealistic plan that fails everybody. So this is a tough step to realistically model. What are the Tasks involved? "Estimate Costs and Benefits" could be one, but "Setting Priority" is circular; set priority how? What examples are there in recent Foundation/community history of doing this work of prioritization well that we can use as models?

A known problem in organizations is the risk of rushing to Prioritize even before Concept (top management decides X is very important, team A is tasked to work on X from scratch). Even before getting in the discussion about how to prioritize publicly and how to involve our communities, it would be useful to make a conscious decision about Prioritize being part of this product development process (or not) and requiring at least a first iteration of Concept (or not).

Note that Concept identifies Community Members as stakeholders, and therefore any concept willing to become a product should be documented publicly. Having all the concepts waiting for prioritization publicly available and well organized would be helpful in many ways:

• knowing all the candidates for prioritization is a prerequisite to prioritize well and to explain/understand the decisions made
• it is also a prerequisite to involve our communities in any sensible exercise of prioritization
• beyond the WMF, other organizations (i.e. chapters, 3rd party developers) and individuals (i.e. volunteers in a hackathon) could submit their own concepts and/or prioritize existing concepts with their own resources

In any case, I agree that clarifying Concept and Prioritize is essential. We need to define a clear starting point for new projects under this new WMF product development process. Adapting all the ongoing projects to the new process will take significant effort, but we can start containing that effort by channeling new concepts through the new process.

"beyond the WMF, other organizations (i.e. chapters, 3rd party developers) and individuals (i.e. volunteers in a hackathon) could submit their own concepts and/or prioritize existing concepts with their own resources"

I agree this is very important, and I think it bears some explicit calling-out and support in how the process is defined. Wikimedia Foundation Engineering a) isn't a monolith and b) isn't the only game in town. We as members of a global movement need to be able to talk about, reason about, prioritize, and help assign some subset of our collective resources to, important issues that might not be picked up by any individual 'vertical' in WMF Engineering.

There are other organizations like the chapters which have varying resources and may have different specific interests, and WMF itself runs grants. I would love it if we had more organized collective work on non-Wikimedia uses of MediaWiki as well -- we basically ignore those use cases inside WMF engineering, while we reap the benefit of having hired a bunch of engineers out of the open source community around MediaWiki.

I'd possibly be happier with a movement-wide process that WMF enthusiastically participates in, rather than a WMF-focused one.

Agreed. That is where all of our energy should be and it is absolutely worth the effort. This is just the high-level and it will never be a perfect fully filled in explanation of the steps within each phase. This is more of a guideline to get things moving or as @Rdicerb (WMF) pointed out could be a checklist. The tasks or path for ideas from concept to plan is much better left to a specific page around prioritization, and historical success or fails seem like a great way to learn. Interested to see what you find there @Rdicerb (WMF). The whole high level table could be linked up like this to specific focus areas, that way while we evolve the parts the whole can also evolve. I like some of the elements of both the Community Wishlist Survey and the slightly different style on the Developer Summit phabricator board. @Qgil-WMF are you suggesting then to create a new column like I believe @John Broughton is suggesting or a distinct line as a decision point? @JAufrecht (WMF) did you see this specific round as part of the SPDPP and perhaps a task on the table with an analysis loop on top of plan as well?

@WMoran (WMF), I have a strong opinion about "Concept exists and has been shared with Community Members" as a requirement to enter Prioritization, but I don't have any good ideas about how Prioritization should be reflected in the draft and the diagram.

There are wishlists and surveys a-plenty. I'm primarily interested in the methodology by which projects and products are prioritized.

Like @John Broughton, I also find a couple of bullet points relevant to how the movement would prioritize developing one feature over another, sooner over another.

Things like:

• Impact, or severity of the problem (We should be prioritizing our biggest problems or opportunities)
  • Data
    • Security/Risk
    • User request
• Potential cost (people + time = money)
• Blockers (sometimes in order to accomplish one thing, another thing must be handled first. SUL is a good example - now that usernames are standard through the projects, cross-wiki notifications and global accounts can be created)

Obviously the overall themes like alignment with movement values and long-term goals also apply.

The basic way I imagine this implemented within SPDPP is that someone at the Foundation who wants to start a project picks from a few different bundles of Milestones, which are derived from past projects (successes and failures). Each bundle would start with a set of Milestones forming a pathway through conceptualization, prioritization, and planning. Each Milestone is a checklist, so all of these bundles would have checkpoints for stuff like "most recent attempt to do something similar identified and 'lessons learned' documentation recovered" and "previous normative community discussion on this topic identified, if any". One bundle might be the path for starting something off the community wishlist; another might be for an apparent greenfield (never been tried before) project, and so on. Each would have multiple "go back and try again" points. We might also want some metrics around what we think the funnel should look like: should the Foundation proceed into full funding and execution for 100% of initiated projects? 50%? 10%?

@JAufrecht (WMF) - following up on this - this section is about prioritization, and I'm trying to correlate your most recent statement with this topic. Could you clarify how the SPDPP connects to Prioritization process being discussed here? Or are you saying that Prioritization needs to have a bundle of actions as a Milestone?

I think Prioritization has to be first-level, not in a bundle of Milestones. The way Prioritization connects to SPDPP is that there should be a few places in the SPDPP process where Prioritization happens, but SPDPP so far just has that as a blank arrow coming in. SPDPP should be able to say how often and and what level of prioritization needs to happen to work well with the Foundation's existing processes, but should be pretty neutral to exactly how that happens.

@JAufrecht (WMF), the key point of this thread is to define the conditions for a Concept to become a priority and start receiving the attention and resources to define a Plan, request resources, become part of quarterly goals and annual plans...

Before being prioritized, teams should have a green field for agile experimentation without having to compromise with process requirements, strategy alignment, community consensus, project documentation... However, in order to be prioritized as a result of an informed decision, these teams will need to do some homework.

The filter proposed tries to avoid that projects are prioritized without the minimum requirements defined, out of process. If we agree on this principle, then it should be reflected in the product development process. I'm not sure whether Prioritize should be an own column between Concept and Plan. Perhaps it should be a set of criteria to enter Build?

No matter how, we need a checklist for project concepts to be considered for prioritization. I just created phab:T120070 to track this action.