Quality review
 | This page is currently a draft.
|
The open nature of wikis makes quality management a serious challenge. This page gives a general overview of quality review systems and their characteristics.
Review processes by types
[edit]A review process can be:
- internal: the process is mainly handled locally by the wiki's community, and the data is stored in the wiki's pages or its database;
- external: the process is mainly handled by an external group, and the data is stored outside the wiki. Integration with the wiki is optional.
- mixed: e.g., an external group posts reviews on the wiki.
Some authority-based review processes distinguish between "experts" and others. In this case, reviewers can be classified whether they are:
- "non-experts"
- people self-identifying as experts (no credentials verification)
- people identifying as experts whose credentials have been verified.
Example review processes
[edit]- A more detailed description is available for some projects.
| Review process | Wiki | Type | Reviewers | Review location | Review tool | Visibility on page | Notes |
|---|---|---|---|---|---|---|---|
| WikiProject assessments | Wikipedia | internal | local community (thematic "WikiProjects") | wiki talk pages | optional (JS) | optional (JS) | |
| WikiProject Medicine / Google | Wikipedia | mixed | professional medical editors | wiki talk pages | no | no | Example |
| Public Policy Initiative | Wikipedia | mixed | local community and subject matter experts | Google document | no | no | |
| Article feedback pilots | Wikipedia | mixed | anyone (readers & authors) | wiki database | yes (MW ext.) | yes | |
| Encyclopedia of Life | Wikipedia | external | Individual EOL curators | external database & Wikimedia toolserver | ? | optional (JS) | |
| Rfam / Pfam | Wikipedia | external | academics from the Wellcome Trust Sanger Institute[1] | external database? & Wikimedia Toolserver | ? | optional (JS) | |
| APS Wikipedia Initiative | Wikipedia | external | technically, anyone, but filtered a posteriori[2] | external database | yes (PHP) | no | Harnessing the Power of Wikipedia for Scientific Psychology: A Call to Action |
Review systems
[edit]| Review process | Type[1] | Binary flag[2] | Metrics | Metrics values | Free-form comments allowed | Notes |
|---|---|---|---|---|---|---|
| WikiProject assessments | exclusive | no | quality, importance | quality: 7 (stub, start, C, B, GA, A, FA); importance: 4 (low, mid, high, top) | yes | |
| WikiProject Medicine / Google | cumulative system, but exclusive in practice | no | none (qualitative) | n/a | yes | checked against good article criteria |
| Public Policy Initiative | no | comprehensiveness, sourcing, neutrality, readability, formatting, illustrations | resp. 1-10, 0-6, 0-3, 0-3, 0-2, 0-2 | |||
| Article feedback pilots | cumulative | no | well-sourced, neutral, complete, readable | 1-5 (stars) | no | possibility to skip metrics (value = 0) |
| Encyclopedia of Life | exclusive | yes (trusted/untrusted) | misidentified, incorrect/misleading information, poor writing/image/sound quality, redundant/duplicate, other | 2 (boolean) | yes | user flags issues rather than assessing metrics |
| Rfam / Pfam | ||||||
| APS Wikipedia Initiative | cumulative | no | trustworthy, unbiased, complete, well-written, accurate | 5 (strongly disagree / disagree / neutral / agree / strongly agree) | yes | |
| Article feedback extended review | cumulative | TBD | TBD | TBD | yes | planning phase |
User research
[edit]In the context of quality review, there are two kinds of users: reviewers and review readers.
From a reviewer point of view
- What system, how it works, what the goals are, what they expect
- Do they want to communicate with review readers?
From a review reader point of view
- What are their goals by reading a review? (i.e. what is useful for them)
- What do they expect from a review?
- Do they want to communicate with reviewers?
People who have already created review processes might be able to explain:
- what were their goals when they created these tools & processes
- whether some design decisions were intentional (e.g. type/scale of rating)
Domain research
[edit]Sources:
- Article feedback/UX Research
- Quality assessment tools for Wikipedia readers
- Interviews
- m:Expert review & m:Talk:Expert review
- Survey: Expert barriers to Wikipedia (includes relevant questions such as: (1) whether respondents volunteer to be interviewed; (2) whether they would be willing to review wiki articles)