Wikimania Scholarships app/Reviewer Process

Reviewer Process

note: "Scholarship Committee" = "SC"

• (pre-review) SC divides itself amount the regions so there are ideally 4 (or a minimum 3) reviewers per region
  • Each member has a minimum of three regions to review; each member is responsible for reviewing all applications within their given region. This is to ensure consistent reviewing across each region (given differences in writing patterns and Wikimedia contributions)
  • SC finalizes selection criteria and application questions
• (Phase 1): SC members view the applications for each of their chosen regions. They do a quick scan and rate all the responses with a 0 or a 1.
  • 0 = spam, 1 = application for consideration
  • In this phase, you can see who leaves what score on the application
  • There must be three 0's for the application to be thrown out; minimum of 3 checks on the applications
  • [small note] SC members are free to help filter the applications from regions outside of their assignments
• (Post Phase 1): Email is sent to all individuals in the database which tells them if they have made it to Phase 2 (yes or no).
• (Phase 2): SC members do a more thorough review of the applications. Based on the answers to the questions and the username's editing history, the reviewer rates the application based on the pre-determined selection criteria
  • In this phase, you cannot see who is leaving what scores. You can, however, see how many people have rated an application and any notes left on the application by other reviewers.
• (Post Phase 2): overall average score is calculated for each application, and the top scores for each region are filtered by score and tagged with whether or not they would accept a partial scholarship.
  • This is filtered into spreadsheets, so we can also tag (a) who has received a scholarship last year, and (b) who qualifies for a chapter scholarship[1]
  • WMF does a cost assessment to figure out how many full and partial scholarships could be given from each region, and then creates a proposal for which persons should receive the scholarship based on (a) % of allocated spaces to that region, (b) the estimated cost, and (c) diversity. It eliminates all individuals who qualify for a chapter scholarship.
  • The SC has a long IRC meeting going through the spreadsheet prepared by WMF and determining if this is the right list. A few names are sometimes switched, based on if the person has attended Wikimania on scholarship in the past, or better diversity (e.g., gender and nationality)
  • (Post Decision): Emails sent to all applicants with the final decision (Partial / Full / Chapter / None)

[1] Several chapters offer scholarships from their own annual budgets. Typically, they use the WMF system of reviewing applications, and agree to sponsor the top applicants from their areas based on their pre-determined criteria. For example, WMUK last year agreed to sponsor 10 (? I actually forget the number) individuals who were either residents of the UK or of UK nationality. The WMF Scholarship chair (Jessie) made a google spreadsheet for them with all the applicants that met their criteria, ranked by the SC's average score.

Notes

Obviously, once the reviewing process is over, most of the data filtering/mining happens outside of the review system. This may be unavoidable.

One change that would be useful is to have a look-up which would pull if someone has received a scholarship in the past. Last year, I did this by hand and wrote a note in each application where we had a scholarship recipient in the past. This is difficult to automate, but we could have look-ups by email, name, and/or username. Of course, all these manual entries leave lots of room for errors.