Google Summer of Code/2017/Lessons Learned

A program feedback survey was sent to 15 mentors and 10 students who participated in the Google Summer of Code/2017 & Outreachy/Round 14 (Phab:T171651)

7 out of 15 mentors and 2 out of 10 students filled out our survey

From mentors responses, we wanted to understand to what extent they were satisfied with student's contributions, their involvement in projects and community, and organization admins facilitation. From students responses, we wanted to understand to what extent they were satisfied with Wikimedia's coordination. Most importantly we wanted to get feedback from both students and mentors on the use of Zulip for the first time a communication/ mentoring tool.

What we learned from mentors

• 4/7 mentors were satisfied with student code contributions
• 5/7 mentors were satisfied with students overall involvement in the community and the project
• 5/7 mentors were satisfied with organization admins facilitation
• Favorite part of mentors during the program was working with enthusiastic students and receiving patches from them
• Some suggestions they've made for how we can improve coordinating the Outreach programs for Wikimedia:
  • "I think the expectation-setting in terms of how to prepare for a successful application could be better, especially in terms of encouraging as substantive a set of contributions as possible along with a well-written application. When there are many people interested in the same project (as with my project for Outreachy 15 right now), I think it's important make it clear that the standards will be high. The selection criteria for mentors set these high expectations (50+ lines of code to be a 4/5 "strong applicant") but they don't come across as clearly in the participant documentation"
  • "It seemed like the big problem for a number of projects this year (not ours, thankfully) was weak students, so it would probably be good to stress more the importance of giving coding tests, like micro-tasks, to potential students"
• Next steps
  • One of the important feedback from mentors responses -  make sure that our docs convey that microtask is a strong requirement for Outreachy applications. Recently, we have made changes to the Outreach program's documentation that covers this point. We would like to check in the future what further improvements it needs.

What we learned from students

Note: We would not be able to lead to any conclusions from the automated survey summary (listed below) from two responses we received from students.

• One student decided to apply to Wikimedia as they found the project interesting, another student was an Outreachy intern in the previous intern and wanted to continue working on the project with the mentor.
• 1 student seemed extremely satisfied, while another dissatisfied with organization administrators facilitation
• One of the student's favorite part during the program was mentor support, and others was meeting organization members at Wikimania
• One student suggested that we could improve coordinating the outreach programs by arranging meetups

What we learned about the use of Zulip as a mentoring tool

• Most mentors didn't use the tool to communicate with students and org admins. Here are comments from some of them:
  • "I did not use it actively, but I got emails from it occasionally, which was useful."
  • "I greatly prefer phabricator and I won't use it again. But maybe worth keeping it as an option."
  • "I and our student never used or logged in to Zulip, so I can't really comment on this."
  • "I only checked it when I got email notifications. That worked fine, except I think my co-mentor once didn't get an email notification."

• One student wrote: "While it is open source and ethuses the community it is not a reliable source of communication"
• Analysis and next steps:
  • Most of the mentors didn't use the tool to communicate with students and mentors. It is not surprising as they were already on Phabricator group chat before the projects were accepted. Some of the mentors were using different synchronous communication mediums to meet with their students regularly. So, this is a fair assumption that switching to Zulip, an entirely new platform would have been extra work for them. Still, some of them who were familiar with the platform emphasized that email notifications they received were useful and that we could worth keeping Zulip as an option. As an organization administrator, the tool turned out to be extremely useful in keeping a close communication with students all throughout the program, specially in matters of urgent sync up. Zulip platform has a nice feature that allows threaded discussions. It was helpful that way to have organized and separate discussions about each phase of the program. It might be worth to continue using the tool in the next round to be able to draw more concrete conclusions. One additional thing to try would be to introduce Zulip to mentors and students when the application period opens.