User:Valeriej/OPW/Project/Paper Summaries

What Makes a Good Bug Report

This paper investigates the quality of bug reports from the perspective of developers by conducted a survey among developers and user of APACHE, ECLIPSE, and MOZILLA. Developers were asked to: 1. Complete a survey on important information in bug reports, the type of information they have used to solve bug reports, and the problems they faced with them and 2. Rate the quality of bug report from "very poor" to "very good". Reporters (people who create bug reports and are not assigned any) were asked to fill out a similar survey on what information they thought developers would rate most important in bug reports and what information have they (the reporters) provided in their bug reports.

The researchers looked for the relationships between which items the developers said were important in bug reports compared to the items reporters said were important. The researchers also looked at the relationship between which items reporters said were important compared which items reporters have previously provided in bug reports.

The survey results showed that the most widely used items across the projects surveyed are steps to reproduce, observed and expected behavior, stack traces, and test cases. Developers in two of the four projects also used screenshots while developers in the other two projects more often used code examples and stack traces. As for important information, steps to reproduce, stack traces, test cases, observed behavior and screenshots were rated highest by developers.

Items provided most by reporters are observed and expected behavior and steps to reproduce. (TODO: read reporters section again to flesh out summary) The top items reporters considered to be most helpful to developers are steps to reproduce, test cases. There is a discrepancy between the items that reporters provide and the ones they consider to be important to developers. This could be because users find it difficult to produce those items.

The researchers compared the information developers used when fixing bug reports and the information reporters provided. They found that 'steps to reproduce and observed and expected behavior align in that most reporters provide that information and most developers use that information. the other categories show a discrepancy in what information is used by developers and what information reporters provided.

Researchers also compared what information developers find helpful vs. the information reporters provide; steps to reproduce match at the top of the list which shows that developers find steps to reproduce most helpful and the majority of reporters provide that information. Developers and reporters conflict in the other categories. For example, according to the survey, developers find stack traces and test cases as very helpful (after steps to reproduce), but reporters are more likely to provide observed and expected behavior after steps to reproduce.

Finally, researchers compared what information the developers find most helpful versus what information reporters expected to be helpful. They found that, overall, reporters and developers agree on what information is most and least important. Steps to reproduce, stack traces, test cases, and observed behavior are at the top of the list for both developers and reporters as information that would be helpful for developers. One item reporters thought would not be helpful but developers rated as helpful was screenshots.

What do these comparisons mean? Reporters have a good idea of what information developers would find helpful, but they do not provide that information. One reason could be the difficulty for reporters to get the information itself, e.g., stack traces. Another reason is that a reporter may not realize how helpful that piece of information could be, e.g. screenshots.

Developers also noted what problems they have faced when addressing bug reports, and most developers reported incomplete information and errors in steps to reproduce as severe and common problems.

Notes for my project: The researchers developed a tool, Cuezilla, to help guide reporters in creating helpful reports. Their tool analyzes what type of information a reporter has put in their bug report, and if it is missing something important, e.g. a stack trace or a screenshot, the tool suggests to a user to attach the missing information. I see this proposal leading to a wizard for bug reporting, and the researcher's processes could be very useful in developing this wizard. I'm not sure if I should/could implement the sort of survey this paper describes in order to get a good understanding of what MW developers use to solve our bug reports and what information bug reporter commonly provide. This report also mentions that developers don't necessarily find duplicate reports harmful, as they can provide more information about a bug. In this wizard, users can be encouraged to comment on the initial bug report to show that the problem exists for them and allow them to provide information the bug that could be useful to developers.

Bug Fixing Practices within Free/Libre Open Source Software Development Teams

"our least effective team also had the lowest level of participation from core developers, suggesting their importance..."

How Power Users Help and Hinder Open Bug Reporting

Notes for my project: This paper has some insights in the perspective sections that I want to keep note of. One is that "...one of the most common problems was that users did not perform basic diagnostic steps before reporting a problem, despite the number of support resources..." Having a report extension to guide users through diagnostic steps could be helpful.