Talk:Quality Assurance/Strategy

Previous page history was archived for backup purposes at Talk:Quality Assurance/Strategy/LQT Archive 1 on 2015-07-10.

Just a note about a couple of bug triage metrics:

• A search like this shows the bugs we acted upon within a week in a certain component, but it doesn't count the reprts that were moved to another components, right? If this is true, last week we acted on 97 reports instead of 87.
• It would be great to find a query showing which users changed anything within a query. The data is there in the Change history, but Bugzilla searches/reports only will show bug reporters or assigned...

This post was posted by Qgil-WMF, but signed as Qgil.

• I cannot come up with a way to create *one* query in Bugzilla's query web interface to also cover those tickets that got moved out of the component (like "any tickets changed that were in the component X between 2013-04-01 and 2013-04-07", only an additional one "component | changed from | Skin and page rendering" (for the last bug day) which you probably also used to find out that we talk about 10 additional tickets: https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&chfieldto=2013-04-07%2023%3A59&chfieldfrom=2013-04-01%200%3A00&o1=changedfrom&f1=component&v1=Skin%20and%20page%20rendering
• Generic "who changed something" queries are pretty bad in Bugzilla. You have "changed by" in the custom/boolean queries, but it always requires either defining a field or a role (reporter, commenter, assignee etc). Again, this may work with some SQL queries on "bugs_activity.who" (I normall take a look at http://www.bugzilla.org/docs/queries.html and https://wiki.mozilla.org/Bugzilla:SQLCookBook to get ideas for those rare moments when I want to fiddle with SQL) but there's no way in the query web interface that comes to my mind. :-(

What parameters should we consider in order to evaluate a Bug Day? Some ideas:

• Stats related to bugs (which ones?).
• How many people involved compared to previous Bug Days?
• How many newcomers?
• How many regulars?
• New people signing in to the Bug Squad.

This post was posted by Qgil-WMF, but signed as Qgil.

The parameters depend on what our aims for bugdays are -- not sure how quantifiable that is. My personal aims are probably to hang around with some great people (triagers and developers), to get to know new great people, try to keep everybody around and involved in our community (they often might move on from triaging to something else), and as the actual short term actions: discussing some bug reports and "resurrecting" them while everybody learns something from each other while having fun. That's hard to put into numbers. :)

• How many people involved sounds like a good argument, but maybe also in which channels we announced a bugday and in which time(zone) it took place, assuming we want to use numbers to steadily improve our outreach.
• Number of reports acted on feels less relevant if we spend time explaining triage to people and try to make connections. If I explain a lot of basics, I triage less, still it can feel like a successful bugday.
• "How did newcomers found out about it?" which translates to "Which communication channels are important?", e.g. one newcomer from the last bugday found out because of the message on [en.wikipedia.org/wiki/Wikipedia:Village_pump|English Wikipedia's Village Pump].

The question about the aim is a good one.  :)

The ultimate goal is to get more people involved in bug triaging, isn't it. Indicators:

• Getting more people contributing to Bug Days, ideally until reaching a point when none of us is really needed to run a successful new Bug Day. How to count bug days contributors? How many did we have in the last one?
• Getting more people contributing regularly to bug triaging, ideally until reaching a point when you don't need to spend most of your time on the bugs themselves? The Bug Squad membership is one data (and we got one new person signing up!). Maybe Bugzilla stats can help finding out how many triagers do we have?

Another goal is to be able to organize Bug Days without much overhead organizing, promoting etc. I think we are doing quite well here already.

And another one is that the people interested in the specific area of the Bug Day are aware and get involved. This comes from focused promotion in the right channels and... mailing the affected Bugzilla users? I started linking to the event in the bug reports comments whenever there was something worth commenting and I think the results were moderately positive.

This post was posted by Qgil-WMF, but signed as Qgil.

I don't have much to say about parameters beyond what has been said already, but I also think it's important to follow through on reports acted on, i.e. if you request more information on a report, check back a few weeks later to see what the reporter says, and if there is no response take the appropriate action (close as WORKSFORME, etc). Or if someone promises to work on a bug, make sure he or she follows through, and if not, reset the default assignee. Maybe we should update the documentation addressing this?

Since this group probably won't be created, I think we should start having discussions in the Bug Squad Project: https://www.mediawiki.org/wiki/Project_talk:WikiProject_Bug_Squad

I wouldn't even sign the "ultimate goal is to get more people involved in bug triaging" statement. I'd say "ultimate goal is to get more people involved into MediaWiki, e.g. by starting via bug triage" instead, seeing the fluctuation in other FOSS projects' bugsquads.

"Getting more people contributing to Bug Days, ideally until reaching a point when none of us is really needed to run a successful new Bug Day." - Yes, and on the way there having more people co-organize and co-lead a bugday in order to cover more timezones. Counting bug day contributors is sometimes hard - e.g. two days ago Matma mentioned on IRC that he's sorry to not have joined the LQT triage, but that he triaged some Interface bug reports instead. :) I expect quite some up and downs in the number of people helping, depending on the project to triage, so hard to judge success. Needs time.

"Maybe Bugzilla stats can help finding out how many triagers do we have?" - Not sure how to do that. Naively assuming that RESOLVED FIXED is only used when an identifiable code commit has taken place and that WONTFIX is mostly used by developers who know the direction of their project better than triagers, you could check who has set other resolutions in Bugzilla lately, but that's extremely errorprone.

"organize Bug Days without much overhead organizing, promoting" - still need to improve where to announce and who to invite etc. Rule of thumb: If triaging a specific component, contact its developers (and probably its bug reporters, as Quim did for the last bug day), announce on #wikipedia, #mediawiki, wikitech-l etc., on the specific feedback / project talk page of the project to triage, calendars, and maybe also via microblogging if appropriate.

I agree with the sentiment  :) yet Sumana is right when she insists that we also need parameters to measure how are we doing, and therefore what can we improve. Let's check against QA/Strategy#Measuring_success:

Measuring success

Activities

• One Bug management activity every other week.
• Co-organized with different teams, focusing in different areas.
• In sync with WMF and other community priorities.

These are easy to evaluate and you are doing well so far.

Results

• Tangible progress after each activity: bugs reported / triaged, scenarios written, good perception of end user perceived quality.

This goes in the direction of selecting manageable topics: if we start with 200 reports then we probably won't even be sure that we have a good idea about them by the end of the week. If we are focusing on a goal more specific (implying say <50 bugs) then it is easier to evaluate what we had before and after. We might even have a sense of 'we went through all of them and now that corner is tidy'.

• Improvements documentation and processes enabling scalability of QA volunteer effort.

Ideally the generic docs related with bug triaging would be fine tuned based on feedback and lessons learned. The pages of each bug triage with their evaluation could still be useful for volunteers interested in those areas joining on their own in the future.

• Developer team values the exercise and is willing to repeat.

Simple to measure.

Participation

• Pool of at least 25 regular non-WMF contributors in each QA category.

There are 9 listed at Project:WikiProject_Bug_Squad/Participants. That list needs an update and be checked against reality from time to time.

• At least 10 non-WMF contributors in synchronous activities.

This is also relatively simple to measure. Like the point before, it's a goal. It doesn't mean that you failed if you didn't reach those numbers. But we should see a trend in that direction over time.

• Mix of newcomers and repeaters in each activity.
• Involvement of Wikimedia editors.
• Involvement of contributors active in other OSS projects, new to Wikimedia.

This implies knowing more about the active participants. Currently this is not easy, but at least we have traces in Bugzilla, IRC and perhaps MediaWiki if peope want to sign up or leave a trace here.

• Involvement of existing groups.

This implies looking periodically for areas affecting a clear group of stakeholders (e.g. Commons bugs) and succeeding involving them in the activity.

Organization

• Topic and dates are agreed 2 weeks before the activity starts.
• Goals are defined before an activity and evaluated right after.
• Preparing a new activity takes less effort.
• The stakeholders related with an activity are aware and engaged.
• Participants can start learning / contributing right away.
• Non-WMF contributors involved in the organization.

All this can be measured.

Evaluation checklist

For an example, see QA/Browser testing/Search features#Evaluation.

★★★☆☆

• Summary:
• Results:
• Participation: WMF? Wikimedia? Other?
  • New:
  • Repeating:
  • Mood:
• Documentation:
• Promotion:
• Developer team:
• Organizers:
• Lessons learned:

And this check list can be applied.

All this might look like extra work, but it might define the difference between proper community engagement or "Andre, Valerie and Quim going through bugs publicly with some friends".  :)

This post was posted by Qgil-WMF, but signed as Qgil.

See QA/Strategy#Measuring_success.

This post was posted by Qgil-WMF, but signed as Qgil.

Some quick'n'dirty feedback as I got to run but wanted to take a look this week:

• Community incentives: With regard to very basic reputation, "Weekly Bug Summary" on wikitech-l has a section "Top 5 bug report closers", and https://bugzilla.wikimedia.org/weekly-bug-summary.cgi?tops=10&days=7 has the "Top 10 people who resolved the most reports in the last 7 days" list and being (and staying) in that list can be intrinsic motivation, but requires making that page better visible (and fixing the bug in the lower section).
• Proposed non-existing "QA priorities" link should be QA/Priorities instead of QA_priorities.
• "Participation: At least 10 non-WMF contributors in synchronous activities." - Sounds good and challenging as a goal, but will be tough for bugdays.
• "Organization: Topic and dates are agreed 2 weeks before the activity starts." I've failed with that so far, but defining it early would be helpful for marketing. +1.
• "Contributor retention": Interpretation of term: "Contributor stays in community" might provide a better retention level than "Contributor stays in QA activity", as triage team members often become developers at some point.

I've made a few minor changes. I think the page looks good overall, but I suggest that we specify some possible ranges of numerical values or qualitative assessments for "Participants can start learning / contributing right away" and "Involvement of Wikimedia editors". I think it's good enough to go forward with and we'll learn more as we go.