Topic on Talk:Wikimedia Developer Summit/2017/Topic ideas

This is the text of the "Quality" section of WikiDev17/Topic ideas as of this writing

How can we improve the quality of our software, and pay down the technical debt we've accumulated over the years?

In developer discussions this is a recurrent topic as well, from the perspective of how difficult is to change anything from a technical point of view (from a social point of view too, but our platform + desktop + mobile web + mobile apps is a challenge in itself).

Fora: qa and wikitech-l readers and participants

I think quality and technical debt should be drivers of the Summit agenda. However, as a listener of many Wikimedia discussions, I have the impression that we frewuenrly compile problems but we lack some sense of North, of priorities, of where to start. Rob, @Greg (WMF), @BDavis (WMF), what "main topic" could be defined that would help focusing Summit participants and discussions on quality and technical debt more productively?

I'm not sure that anyone needs to be convinced that cleaning up the code base is a good idea. Where we seem to get stuck is in finding people who actually want to spend time fixing the broken windows and/or reviewing the fixes that others have proposed. There is a lot of good work that is done quietly by @Aaron Schulz, @Anomie, @Tgr, and @Legoktm to name just a few. Maybe one or more of them has a idea about a particular area that could use more discussion or just more advertisement of things that could use help?

My impression talking to non-engineering staff at the WMF is that we have a rather large communication problem between engineering and the rest of the organization that we need to tackle -- the perception is that engineering is being given lots of money and "not having anything to show for it". This impression was strengthened during rough times recently when a number of expensive projects were started and then abandoned due to poor management -- since rememdied, one hopes, but the damage has been done.

From top-down (board, C-level, public) what is needed is engineering to actually *accomplish things*. Spending a year working on "quality" in this environment -- without a corresponding push in communication/etc -- is likely to do harm (like extend our current hiring freeze, continued flatlined budgets, ongoing community disapproval).

This isn't to say I'm opposed to this topic! But we should recognize at the outset that a large part of this work isn't *finding things to do* or *finding engineering willing to work on quality*, but rather in *convincing management that resources should be spent on things that aren't products or features* and *quantifying and communicating the results of quality work*. We can't just go into our engineering workroom and come out in a year pronouncing things "much cleaner now".

This isn't to say I'm opposed to this topic! But we should recognize at the outset that a large part of this work isn't *finding things to do* or *finding engineering willing to work on quality*, but rather in *convincing management that resources should be spent on things that aren't products or features* and *quantifying and communicating the results of quality work*. We can't just go into our engineering workroom and come out in a year pronouncing things "much cleaner now".

This is a good point I think. I don't feel it is at all unique to the Wikimedia Foundation as an organization either. This is my 5th day job as a software developer and I've experienced the same issue at all of them. There is one thing that I have found to be different here however. That difference is a split of "product" vs "platform" that somehow sees these as completely separate concerns. The "product" side seems to have been in the ascendancy for the last couple of years which is perhaps making the prior split even more exaggerated.

A large portion of my time at $DAYJOB-1 where I held the title "Software Architect" was negotiating a balance with engineering and organization management over the number of new things that could be built in a given time period and the technical debt cleanup and forward looking R&D work that also needed to happen for the software products to remain healthy and viable. This is a role that I have not seen well represented in the WMF org charts or the small number of teams that I have participated in. I have no idea how that translates into topics for a development summit.

Considering the above, sounds like this would not be a very good topic for an inclusive summit. It seems that in this area everybody is aware of what needs to be done, so it is only a resource issue. These are better resolved in smaller meetings than in conferences in my experience.

This is a pretty Developer-focused (both WMF and community, so let's just say "Wikimedia") topic, not the *most* inclusive if we also consider other contributors (not the worst though, way better than "what team should hire what type of staff").

I would like to move forward on some topics in this area, however. One big thing is "unit" test coverage. We're horrible (MW Core unit test coverage is at ~10%). We also have a fairly inverted testing pyramid (we (WMF product teams, mainly) spend a lot of effort on browser/api level tests, but not unit/integration tests). That's not the makings of a solid base to build on top of.

Once the topic of test coverage comes up usually one of the immediate responses is "but who's responsible for Core?" Good question. I have some vague ideas on how to move forward here, but I also like getting feedback/brainstorming from groups of people before moving forward (just my style ;)).

Would that (something like "how do we improve unit (with real unit tests) and integration test coverage for the software we write and maintain?") be a narrower but useful focus?

Greg -- I think that would be a useful concrete subtopic, but i'd worry that we'd get lost in the details of unit testing frameworks and forget to deal with the broader question posed above: "but who pays for it?".

That's actually the point/topic I was suggesting and it's relatively easy to steer the conversation away from tooling to who pays for it/how to get there :) (we did so at the Product&Tech onsite last week).

It seems a death spiral is possible here:

1. We look at our seemingly overwhelming mountain of tech debt, and declare ourselves without sufficient resources
2. Tech debt accumulates, with fewer and fewer voluntary collaborators being involved, because life is too short to deal with the mountain of some else's tech debt
3. We get lower funding, because we don't seem to be accomplishing as much as we used to
4. Repeat

It may be that the name ("Quality") is poorly chosen, as it leads to this death spiral thinking. It may be that we should think reframe this as "Developer Experience" in a nod to @Sherah (WMF)'s earlier comment. In what ways can we make improving our code safer and more fun? How can we improve the motivation for people not paid by WMF to improve the code we deploy? Are WMF employees the only people that can possibly understand how many of our deployed systems work?

In our recent IRC conversation on this topic (Phab:E269), we talked about how it's important to use the Dev Summit as a place to discuss tech debt issues. @Legoktm pointed out "I think it is important that we don't lose the architecture part of the summit. Most of our day-to-day work/priorities are dictated by the products (in the user-facing sense) that we work on, and the summit is the one of the few places we get to work on the actual architecture/tech-debt" Let's spend some of our time figuring out how to pay down our tech debt.

As far as the "who pays for it?" topic, let's get each team working on new features to account for how they are going to clean up messes in their environment. Let's get out of the habit of hand-wringing about our current plight, imagining ourselves lying in filth (possibly self-created), whining about how someone else is not offering to clean up the environment we operate in.

Thankfully, MediaWiki isn't "filth", it's actually very good software that has served the movement well. We have a really good problem to solve: improve the software running on a website that is already in the global top 10. Not everything in it is written as we would write it from scratch today, but many of the people who built the original versions are still working with us and can teach us how they did it, and help us avoid making the mistakes they made. We can use this as an opportunity to celebrate our collective achievement; that we have something good enough to improve. Let's make sure that the product of future improvement will be something worthy of more improvement.

"How to pay down our technical debt" would be a good main topic.

I think "How do we manage our technical debt?" is a better way of putting it. To build on the "debt" metaphor we're using, many economists believe that deficit spending can be a healthy thing. With technical debt, incurring additional debt is sometimes a necessary step to get to a minimum viable product; there just needs to be a strategy to manage the load of technical debt.

@RobLa-WMF good point. I have edited the WIP list of main topics accordingly.

Well, I think there's a resourcing issue, but enough people have already talked about that. But there's also a systematic issue of that we are okay (as in, we don't block it) with people adding more technical debt, without a plan to clean it up.

• New extensions being deployed to the cluster with no clear end goal or removal plan ("this is just a small test/experiment" and then spend 3+ months getting it undeployed)
• People writing new fancy interfaces/classes that replace old stuff without finishing replacing it, even in core (my fear about MCR)
• New code being written without test coverage is always technical debt.
• Missing documentation :(
• No policy of removal on old code (see recent CH deprecation sprint + a draft deprecation policy RfC)

There are probably more things, but it's a bit late. I think those general ideas would make for a good big group discussion, where people can discuss their needs and pros/cons in a large group setting. And then we can break into smaller "working groups" (I suppose that's the term) to analyze specific problems and solutions. So for my missing test coverage example, a working group could investigate making code coverage run before the +2, and informing developers whether they're reducing it. And then identify classes which are extremely hard to test and need refactoring, and pass that onto the rest of the developers. Or finding CI infra instabilities and providing a recommendation to the releng/labs teams for resourcing. etc. I think that would a good use of time if we want to discuss the technical debt aspect of quality.

@Legoktm ... this is pretty close to what was discussed at the September onsite in SF and there seemed to be broad agreement about it, right @Greg (WMF) ? Maybe coming up with specific recommendations and coalescing around those might be one idea here?

Right @SSastry (WMF).

I think "quality" and "tech debt" are not that useful for communicating the importance of having a well-maintained codebase to non-developers (and possibly not that helpful for developers either when they need to decide what cleanup activities to prioritize). Two topics that are not proper subsets of quality but have a large overlap, and which are framed around the targeted benefits:

• Reliability: how do we make sure that our code is secure and robust by design, it is easy to identify and learn the correct usage patterns, mistakes are hard to make and easy to spot, errors are caught before deployment and in the rare cases when they aren't they are easy to debug? This would span topics like test coverage, exploratory testing, staging, logging infrastructure, documentation, API contracts, code review friendliness of the architecture (e.g. is it easy to tell which pieces of text need to be escaped?).
• DX: how do we ensure that our codebase is easy to understand and efficient to use? This includes stuff like design patterns, identifying confusing APIs, consistency of the architecture, documentation, CI, logging, debugging tools, developer environment (vagrant, docker etc), IDE integration. Also getting feedback from developers on what the DX pain points are, for which the summit is a great opportunity.