User:BDavis (WMF)/Projects/Review tool
Review tool is a placeholder name for a system for managing the process of having a group of users evaluate grant/scholarship/whatever proposals and collecting the evaluations to use in making final decisions.
Problem
[edit]The Wikimania Scholarships app is a one-off solution to this problem. IEG would like something similar. Internal discussions at the WMF indicate that if a more general self-serve solution could be found there would be additional programs interested in using it.
Simplifying assumptions
[edit]- For an MVP, there is no need to collect information from applicants
- For an MVP, administrators can be responsible for populating list of proposals
- For an MVP, there will be no localization of string content entered by administrators (field labels, etc)
- For an MVP, getting the IEG workflow to function with no UI to create additional campaigns or configure the proposals and reviews is acceptable
User stories
[edit]- As an administrator of the grant review system
- I want to create new grant campaigns (defining reviewers, review criteria, data to be reviewed, and data grouping)
- So that reviewers can provide feedback on the applications in the campaign.
- As a grant administrator or reviewer
- I want to mark grant applications as eligible or ineligible
- So that ineligible grant applications can be excluded from further review.
- As a grant reviewer
- I want to score a grant application on multiple dimensions (defined by the admin)
- So that the quality of the grant application can be compared to others in the same campaign.
- As a grant provider or administrator
- I want to view reports on the aggregate reviewer scores for grant applications in a campaign
- So I can share results with others outside of the review system and select applications to fund.
Hypothetical workflow
[edit]- Users can invite new users to the system
- Users can authenticate via OAuth
- Users can create a password to authenticate in the event of OAuth failure
- Review tool admin creates a new campaign and assigns 1 or more users as campaign administrators
- Campaign administrator fills in campaign details
- Define what a "proposal" looks like for this campaign by describing fields that can be populated when a proposal is entered
- Define what a "review" looks like for this campaign by describing fields that can be populated when a review is entered
- Define what the review queue looks like for this campaign (which fields from the proposals are shown as columns and what order)
- Define ... other things about the campaign
- Import/create proposals matching definition for this campaign
- Assign 1 or more users as reviewers for this campaign
- Reviewers see list of campaigns they can review proposals in
- A paginated, sortable list of proposals belonging to a campaign is available to reviewers
- A reviewer selects a proposal and sees a screen giving proposal details and a review form
- A reviewer submits review and can see their feedback along with anonymized feedback from other reviewers
- A campaign administrator can see non-anonymized feedback from all reviewers
- A campaign administrator can see summary report(s) of all reviews
- A campaign administrator can export all reviews as CSV for import into another tool
- A campaign administrator can close the campaign which stops new reviews but allows browsing reports and individual proposals and reviews.
- A campaign administrator can publish the campaign which closes the campaign and makes anonymized review feedback available to non-authenticated users.
- A campaign administrator can archive the campaign which closes the campaign and removes rights for anyone but administrators to view data.
- A campaign administrator can delete a campaign which removes data from the backing datastore in a non recoverable manner.
Data model
[edit]