Wikimedia Developer Summit/2018/Lessons Learned

The Wikimedia Developer Summit 2018 feedback survey was sent out to participants on 6 February, 2018.

We will close the survey on 20 February, 2018.

Results / summary of the feedback survey are being posted here, currently this page is incomplete.

-------

Feedback survey background

rfarrand created the feedback survey. Many members of the Organizing Team made edits or suggested changes.

The feedback survey was sent to participants / wikitech-l, Engineering@ & WMFAll@ on 6 February, 2018.

A reminder about the feedback survey was sent out to all participants who had not filled out the survey (and those who choose to remain anonymous) on 13 February.

The feedback survey closed on 20 February.

The feedback survey received 47 responses

Privacy Statement for this survey

Considerations for next time

This section is based on the fill-in-the-blank/comment sections of the feedback form. Some of these comments will be contradictory. We are mostly trying to include common themes and issues that were felt by groups of people. This section is a bit more subjective than the #Data section below and suggestions may be paraphrased and combined with each other.

In this section comments at the top of each section were mentioned more often while comments at the bottom were mentioned by one person. Some comments are combined and many are shortened for easier summary. The future program committees will review full comments again in the future when they are working on specific areas of the event organization.

Questions for Participants

Please provide ideas for how we can improve the pre-event engagement of participants.

Most commented: More details / guidance / clarity on expected outcomes much earlier in the process. Discussions should also start earlier.
Multiple people: Everyone should feel some responsibility. Each person should have a task / stake in a presentation / distributing the burden instead of the session leader(s) doing most of the preparation on their own.
Multiple people liked: Small focused reading list / something to be done in limited time.
Two people: uniform pre-event engagement across the different sessions / framework of things to do while preparing the sessions.
Two people: Don’t expect pre-event engagement from participants
Be clear in Phab tasks on what conversation should be done on Phab and what should be saved for the event. / Are all attendees (not just WMF) on Phab?
Discussions should not take place on Phabricator / its not ideal for a long discussion
Have the slide decks ahead of the summit / everyone reads the slide decks ahead of the summit.

Structured debates on something like Kialo would be great. Something that clears up a debate and prevents people from re-hashing the same argument over and over. We need some technology help.
You definitely need experienced people to help pre-coordinate. Be careful that the leaders/experienced ones don't dictate the plan.
Have someone present a thesis and have a debate on it
The selection through essay writing felt rushed and unnatural. Anonymous character of it forced you to concentrate on writing something that will be liked as opposed to what you really think.
Session organizers should communicate how they wish to conduct their sessions and what they expect from participants.
I think as the outcome of all the sessions was somewhat standardized, a standard set of methods to get to these outcomes would have been helpful.
Identify 1 question that should be discussed beforehand / discussing in the phab ticket without a clear focus is not very effective
Send weekly emails to the participants about engaging / create expectations

Any suggestions on how to improve session facilitation?

Stay away from topics that provoke a great deal of impassioned debate but go nowhere
Consider dedicated facilitators for larger discussions so all participants can contribute to the discussion
Several people thought that the session facilitation went well
Have more facilitators like Joel
Don't have any session facilitators at all
Have a Gatekeeper that asks people who haven’t contributed (verbally) to the session to speak up (to avoid the pitfall of only a few strong voices talking)
A requirement that each session produce a list of recommendations for the Foundation, and that the would be a process afterward to consider each recommendation (reject, or accept, or modify)
Set up equipment day before before sessions starts to avoid AV problems etc
Establish a consistent format for all sessions up front (in communications) and during the introductions
Have list a few one or two word topics and the ones voted most popular are the ones discussed
Have a list of topics that relate to the overall session and time box each topic
Discuss the position papers themselves, not just the topics that the leaders decided on
Felt it was weird to invite "outsiders" (non-WMF employees) and not specifically ask for their ideas and feedback
It seemed passive and unclear how any outside perspective would be useful
More focused sessions and a topic leader with a concrete proposal
Keep session format consistent across sessions
Have topic "chairs" who review talk/discussion proposals to be more productive
It did not seem clear on how much moderation was expected from topic leaders
Topic leaders and facilitators should connect earlier with each other and discuss the agenda, group activity methods and their roles during the session

Any suggestions on how to improve topic leader preparation?

Create focus groups and make them by invitation only, if the overall goal is to have specific outcomes
Establish a common baseline of knowledge for every topic (what works, what hasn’t worked, what’s possible, etc)
Research the best practices in "how to organize a brainstorming session" and provide it to topic leaders to aid in better facilitation
Have a clear plan or goal or proposal to be discussed before coming to the conference, and not try to come up with something ad hoc
Needed to have a clearer idea of what people are interested in and have concrete topics for each leader to cover (something like - “your session will focus on answering these 3 questions”)
The purpose of the sessions are to select and prioritize. All attendees should be prepared for each session (read the position paper in advance) to distill the ideas into something workable while everyone is together in person
Go back to a more traditional format with one presenter and lots of discussion afterward
Don't have topic leaders; have the attendees decide what to talk about
Provide enough time in advance for the topic leaders to prepare and get more involvement of attendees; there wasn’t enough time this year due to holidays
Give topic leaders time to sync their sessions with other shared topics (and leaders)
Provide a standardized toolkit of best practices and methods

Process Questions

Ideas for improvement (related to 2018 participant selection process)

(This was the toughest section to summarize, we had many long and well thought out responses. We shortened the responses to be more readable and broke many of them into multiple parts to fit into the different sections. We are making a new proposal based on the main themes here and will make sure anyone who ends up on the program committee is able to see the full and complete responses people submitted to this section.)

A few people suggested renaming the event: Re-titled the "Wikimedia Foundation technical planning summit" / strategy planning summit, call it a strategy planning summit rather than a "developer" summit that most developers don't get to attend.

Invitation

Invitation only event: (1) in practice the event would probably have mostly the same people present as this year; (2) specific invitations could help mitigate groups opting out entirely;
The program committee to take responsibility and define topics to be covered (maybe in a bottom-up fashion). They then invite people who run the sessions for these topics and in collaboration with these people create a list of people who will be invited to attend the event. Who will get invited will depend on the outcome desired by each session and people's expertise.
Liked blind review process. Need more continuity in the participants list. Invite the topic leaders of this summit to attend the next summit, so that we can ensure that we actually build upon work. / Ensure the exchange with product managers (both from wmf and wmde).

Position statements unfair / bring the wrong people

Position statement are unfair: expectations were very ambiguous and the position statements were essentially unused both in the actual conference and in the planning of it. I am not opposed to the essay contest format in principle, but if we force people to write them, then they should definitely be used.
Someone's always going to be unhappy, but maybe try and make sure that it's not always the same person. Position papers are fine, but select for the type of person that's adept at writing position papers.
Position papers seemed like writing an essay for a college course, but with a competitive nature. So I had to both figure out what to say, and how to say it well. I failed to do either. I've never been good at English, and this is supposed to be a *developer* summit. Your quintessential developer sits in a dark corner and codes. Communication is not our thing, especially when presented as a competition, of sorts. If we don't understand the strategy, we're not welcome, or at least that's how it felt.
The summit as conceived this year was basically business meetings to determine future direction (tactical implementation of strategy). This is a very poor fit for an essay contest as there's no guarantee the right people will be in the room to discuss these things as you need stakeholders (i.e. people with decision making power) to have a fruitful discussion on tactics.
If we're going to limit the number of participants to be smaller than our core developer community, we should stop thinking of this as a "developer summit" and instead call it something like an "insiders-only summit". There seemed to be a pressure this year to enforce structure and control on the event that was unnecessary. We ended up having a few dead-end conversations rather than many more conversations where only some of which lead to a dead-end.

Current process OK

For an anonymous selection process, I don't think I can come up with a fairer solution.
Position statements and anonymous ranking is great. Limit the length of the statements for two reasons: Help scale the event to hundreds of applicants without crushing the committee with paperwork. Make it easier to read the position statements of all the participants as a participant.
The position paper process was great / I liked how you attempted to group them.
Position statement were good but the summit should provided the space to talk about them. Strategy will not be relevant at the next summit.
I think position statements worked. There are a fair number of senior people in the organization who did not submit position statements. I feel like diversity was given importance which is GREAT. It's really important to have diverse voices in these important conversations.

Position statement process

Position statement process needs to be clear, transparent, and valuable.
Make it clearer what we expect in the position statements (ie not a 5 page paper)
It would be useful to have better clarity as to how position statements are going to be matter. Also, there should be some mechanisms for non-attendees to follow / watch / participate or weigh into the discussions. There are engineers and others that can contribute very useful input and feedback and context even if they don't or cannot write position statements.
Allow everyone to vote on participants, or just those who submitted the position statements.
Similar process to before but first identify voices that should be represented. Position statements should be about what perspectives are being brought to the table, who are you representing, etc.
I think we should have a mixed process: One that gives seats to those who are making the decisions right now (within the WMF and WMDE) in addition to an open process for others to apply. Bring all Annual Plan program owners from Audiences and Technology (or their delegates). Add on to that a "tell us why you should be there" process.("I'm someone/my team is implicated heavily by the topics discussed and decisions being made and I have expertise that would be helpful but I don't have a specific position on a topic that I'd like to shepherd at the event") persona. That persona was notably absent from this year and it shows.
A submission system may be fine, but it should somehow be more technically oriented, and not a structured essay. Maybe a Q&A system, through Google Forms?
Time preparing the position statements but very little factored into actual session discussion. Some position statements were high level, and others had detail. Themes and topics were picked based on position statements, but other they were not used.
Turn the summit a drafting space for annual plan proposals. More open proposal process. People could be asked to write (possibly in groups) a summary of the proposals beforehand / select the ones that generate the most interest.

Topics

Do not ask for position papers, ask for presentation abstracts on topics people are field experts in. If the number of participants has to be reduced that much, have a scientific committee pick people based on relevance.
If you want to keep the size limited, I suppose continuing make it more C/architect-level friendly will reduce size, as the average developer sees little value proposition in attending.
Topics would be separate from position statements chosen by participants and leadership.
If you're looking for more ideological, 15+ year ideas, or more practical implementation ideas then the summit needs to change to support broader ideological discussions that might challenge some long and strongly held beliefs.
Solicit arguments for and against certain hot topics. One of the overall problems I saw in this meeting was that everyone had good ideas, but there were few rebuttals. When there were, it still didn't seem like we had made progress on down-selection.

Who should be there

The focus should be towards targeted groups. Who needs to attend and what positive outcome should come be analysed. Creating a small competition or hackathon before one month can be more helpful as there will be more contributions.
Less WMF people attending which is overall a good thing
I suggest making attendance compulsory for all product management and all direct reports of Victoria and Toby.
Having some product managers in these conversations would be nice too.
Include *not just senior technologists* but people with a varied level of expertise. Best outcomes emerge when the juniors also got to have a say in their framing, or they are just allowed to be present as listeners.
Each WMF team pre-recruits folks from their team who get to apply for the position statement and that way we ensure that there is a good mix of people from various projects/teams who land up at the summit.
Allowing the participation of only those who submitted a position statement might limit the success of the DevSummit if that means that key people (based again on scope and goal) are missing because of the process.
If you want technologists to talk only among themselves—which is perfectly reasonable—then your process is a good one. But if you'd also like discussions to influence future product developments then an additional qualifying route would be desirable.
People more involved with our development process (long term contributors, prominent volunteers, staffers working on important projects) should be given importance because these people can actually implement outcomes from these sessions.

No event

I think my one solution would be to scrap the event outright.
I strongly suggest against having a dev summit in such conditions

Other

People with Opinions (capital O intended), and they view the Dev Summit as a chance to share those and basically get validation of them. A chance to start with an unknown and bash out a plan is rare--critique of existing plans seems to fall on deaf ears.
I think the developer summit would be much better if (1) it can be as big as the community it is bringing together and (2) the topics can be controlled by the participants. (2) can be achieved with track chairs who manager general topic spaces and mentor potential speakers/facilitators.
The heart of a *Developer* Summit, where we share our ideas and think things through *technically*. Those who do understand long-term goals need the developers who understand technical challenges, best practices, etc., and together they can figure out the best approach.
Maybe the selection criteria could be made more transparent. But I guess it's ok like it was done this year.
The summit could be a place for brainstorming and kicking off new projects (worked well in the past, e.g. the developer wishlist and the Platform team proposal) More flexible schedule. People to propose projects beforehand (and then checking their activity on Phabricator) could be used for selection.
If along-side all-hands: (1) Have the whole Foundation in one place (the office?), (2) Hold exclusive dev summit sessions totalling ~9 hours over two days (the same as in 2018), (3) Not separate participants and non-participants for the remaining time.
This event seemed to eliminate the so-called "hallway track" which eliminated much of the most useful discussions from previous summits.

Please suggest topic ideas for 2019 position statements

Writing generally useful libraries and services (how, why, and are we doing this already?)
What kinds of APIs should MediaWiki and the Foundation have?
How to disentangle presentation from data in MediaWiki (without destroying Wikipedia)
Focus on the Wikimedia Platform Evolution, MediaWiki and it’s ecosystem
Tech debt: MediaWiki core, unmaintained extensions, gadgets, frontend and backend mechanisms, how to create and keep better documentation, code stewardship, sunsetting processes, future of Labs
Have specific tracks (not just discussions) for: MediaWiki, operational infrastructure, security, research and analytics, contributors and readers, developer support and community
Three things we are doing well and three things we are doing poorly
Improving our engineering culture
Several people (6+) did not want to have any position statements
Sessions to track our progress (starting from some year in the past) in Tech
Sessions on phase 2 of the strategy movement, it’s execution and the expected outcomes
Focus on the year 2030 and what the future could hold for us (not what’s going on right now)
Change up the format - product manager’s present a topic for 20 minutes, then 45-50 minutes for going into more detail, discussions and Q&A
Language or translation related topics
How will Wikipedia evolve to remain relevant in 15 years
What can the Foundation learn from 3rd-party and volunteer wiki developers with their "enterprise" use cases
How to contribute to UX research; teach how to do basic usability testing, review basic UX best practices, and ideas for future interfaces
Developer wishlist
Interview people that don’t contribute to Wikipedia right now to figure out why and what it would take for them to start contributing
Outcomes, recommendations, and action items from the 2018 sessions (focusing on the “how”)

Documentation Questions

What did you find helpful or confusing about the way we used Phabricator/wikipages/etherpads for organizing the work before and during the summit?

Wasn’t clear if (and how much) pre-event discussion was encouraged or desired
There wasn’t much discussion of topics in Phab (except by topic leaders) before the event
Felt that the discussion on Phab should have been to request answers to a small amount of questions and then go over that in the sessions
Quite a few folks said the usage was fine / good / helpful to be using our normal tools
Format was easy to find, get involved with, and were well structured
Tracking progress of the sessions and the discussions was very helpful
Input from the Phabricator pages didn’t always get to be part of the sessions
Would be great to only use a wiki instead of etherpads (one day)
Helpful to document everything but some session notes were too verbose
A recording (voice or video) would be just as good and with less effort (than taking notes in etherpad)
All three are useful as long as the scope is clear and are cross-linked
Phab wasn't as useful as it could be - many tasks weren’t filled in or had too much background reading
Better usage of Phabricator would be for detailing the action items
Lots of clicks required to find the relevant info - felt that it was too spread out
The tools were the least duplicative (from past years)
Very useful to have etherpads in advance of the conference - made taking notes easy
Too many phab tickets made it hard to navigate
It wasn’t clear how the notes would be transferred to wiki after the conference
Unsure what the final (after the conference) outcome would be - how would it feed into the 3-5 year strategy process
Have a ‘catch up on documentation’ time slot at the end of the conference
Wiki is great for providing information about scheduling, but the positions statements should have been all in Phabricator
The real-time aspect of the etherpad note taking plus the deliberative aspect of Phabricator and the wiki summarizing worked well

Other

Please provide ideas for how we can improve the event session format

Sessions were too large (too many people in each)
Sessions should be a max of 2 hours (otherwise, split into smaller sessions)
There was not enough time during breaks for informal discussions
Social gatherings were too noisy to continue informal discussions
If a person’s position statement was accepted, have them lead / talk during the sessions
Position papers didn’t always “fit” into the session topics
Add lightning talks / unconferences
Keep the fireside chat
Product managers present their FY goals and their 3-5 year plan at the beginning of each session and then do hour long slots during the day to go over those plans
Keep keynotes to less than 30 minutes
Keep the wrap-up session at the end of each day for a good recap
‘next steps for languages and cross project collaboration’ and ‘MediaWiki architecture’ programs were too big for one session
Some sessions were lacking in a clear purpose or goal and were too broad in subject
Base the participants selected on merit, not their position statements
Don’t break up the sessions over a break or for keynotes (keep them about 1 1/2 hours each)
No more than 2 keynotes a day
With fewer tracks and less participation, it felt like it resulted in less getting discussed
Create a small event of it’s own for discussions around machine learning
Better prepare the topic leaders for their session and to be in sync with other topics
Make sure all the AV equipment works
Facilitation was good

Do you have a suggestion for a specific keynote speaker for 2019? (Please provide their name, and affiliation / background)

Two survey respondents asked not to have keynotes at all
Tim Starling, Wikimedia Foundation
Don Knuth
Someone from GitHub
Something related to the events theme
Someone from Mozilla
Dries Buytaert, founder and lead developer of the Drupal CMS
Somebody technical, not politics/social justice
John Gilmore (EFF board member)
Zeynep Tufecki
StackOverflow people like Joel Spolsky or Jeff Atwood, people working in large open source projects like Drupal, Symfony, PHP, Node, developers or managers from other wikis like Wikia / Wikihow, Clay Shirky or another social computing researcher.

Do you have suggestions for specific topics you’d like the keynote speakers to cover for 2019?

Multiple people say that they should be technology based / not about social justice
Multiple people say wikimedia focused or tailored to our community specifically
Two people say: no keynotes at all
Two people say: People from other large open source organisations with similar goals / solving similar problems
We had to many keynotes
Theory and practice of operating a FLOSS project; Collaboration at scale; Promoting successful bottom up planning
Wikipedia use under authoritarian regimes
More Wikipedia related Machine Learning.
Someone who has improved the code quality and organization of the software
Someone who has dramatically reformed APIs or developer-test-prod VMs/rigs.
Knowledge equity. How do we spread knowledge to marginalized people? Can marginalized people come talk to us about it? How do they see the world, what do we need to do to meet them where they are?
The psychology behind different personalities, what drives different people to participate in wikis or other collaboration.
Software freedom in the context of the internet services era.
External speakers on tech debt and documentation.
Developer support & community
The role of an ecosystem for an open source software
Consensus-building and governance challenges of open-source software,
Social computing and the ways software influences community health and cultural norms,
An analyst / (reasonable) futurologist talking about internet and technology trends.
Boring technology.

Do you have suggestions for specific topics that you would like to hear about from WMF management at the 2019 summit?

some of technical challenges they had in 2018-2019 and how they addressed them
thoughts on some of the technical challenges ahead of us, and what we should do to overcome them.
how exactly are the funds distributed, and how can the developers influence in the allocation of resources for a small project. i.e. things that a single developer can do by herself
Integration of DevSummit outcomes with WMF Annual Plan and resource allocation. How they interact within each other and which one influence and guide the others.
A clear direction on third party operated MediaWiki and the appropriate degree of investment as a Wikimedia Foundation.
Insight into what and how we should build for interactive agents (voice, chatbots), the direction for A/V+VR and the tie-in to mobile computing and sensors/embedded.
A clear direction on our approach for channel expansion while ensuring growth in first party usage across WMF/WMDE-maintained channels.
I would like some clarity on our mission. According to T&C's cultural orientation, our technology is a means to an end. However, most people at the dev summit do not agree with this.
Specific examples of trade-offs we make, and why. How do we choose to focus on work?
How can we be more dynamic and collaborate more across teams?
A broad (still concrete) roadmap.
Implementation plans for the outcomes of the last summit.
How the Stewardship review process will be incorporated into your planning.
Less management involvement.
WMF management should prepare something, *anything* to the level they prepare a statement to the board.
How to manage tech debt
I would love to hear/talk about building the ecosystem and tech partnerships within and beyond the movement. But we shouldn't ask that the WMF management alone.
Mostly what they hope to get out of the summit.
What management did in 2018 to promote MW as a tool outside of WMF and inside WMF. Talk about how MW encapsulates the culture of WM in code.

Are there other things that we should be measuring that we are not asking about?

Post-summit impacts / effects / evaluation of outcomes / productivity
What changes have you seen since past Dev Summits? / How many sessions from previous years had any of the resolutions carried out, or blocked by lack of resources.
Interaction with other developers
Who was missing?
How can we measure success with such large generic topic areas?
Ask people who didn’t apply why they choose not to attend?
Ask me what I took away from this event -- why my attendance was important.
Did anything useful come out of the event itself or was it just because of the people attending?
Did anyone who did not attend gain anything either from reading documentation or following along?
Challenges that resulted from dividing staff (not allowing some at this event)

Data

Did you attend the Wikimedia Developer Summit 2018 in Berkeley?

Yes: 61.7%

No: 38.3%

Have you attended any previous Wikimedia Developer Summits and/or Architecture Summits organized by WMF in 2017 or earlier?

Yes: 87.2%

No: 12.8%

Questions for Participants

Omitted: How useful for you were the keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

Omitted: How useful for you were WMF management keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

How useful was each session for you?

Response options are ranked in order after giving 2 points for every "very useful" response, 1 point for every "useful" response, 0 points for every neutral responses, -1 points for every not useful response and dividing by the number of responses.

Growing the MW technical community: 17 responses
Supporting 3rd party use of MW: 16 reponses
Embracing open source software: 20 responses
Knowledge as a service : 16 responses
Research Analytics and Machines Learning: 11 responses (tied with next steps for language)
Next steps for language and cross project collaboration: 11 responses (tired with research and analytics)
Advancing the contributor experience: 15 responses
Evolving the MW Architecture: 22 responses

Please tell us how much you agree or disagree with the following statements

Attending this event was worth my time

Strongly Agree: 10

Agree: 10

Neither Agree or Disagree: 5

Disagree: 1

Strongly Disagree: 3

The opportunity to meet fellow developers was valuable to me

Strongly Agree: 17

Agree:

Neither Agree or Disagree: 2

Disagree: 2

Strongly Disagree: 1

I would like to attend this event again next year

Strongly Agree: 14

Agree: 7

Neither Agree or Disagree: 5

Disagree: 2

Strongly Disagree: 1

Process Questions

Based on your understanding of the process, do you think asking participants to write a Position Statement was fair?

Yes: 29.8% (14)

Indifferent: 8.5% (4)

No: 31.9% (15)

I don't know: 6.4% (3)

Other: 11 participants

Based on your understanding of the process, do you think the participant selection process was fair?

Yes: 21.3% (10)

Indifferent: 8.5% (4)

No: 14.9% (7)

I don't know: 46.8% (22)

Other: 4 participants

Documentation Questions

Please rank these tools in order of how helpful they were to you and your topics of interest in following the actives and outcomes of the Wikimedia Developer Summit 2018

Phabricator

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Etherpad

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Wiki

Very helpful: 14

Moderately Helpful: 0

Not Helpful: 5