Typography refresh/Retrospective

Please note: this a transcript/notes from a meeting, and thus may lack some detail or context. If you have additional comments or questions, please put them on the Talk page, rather than expanding/editing the main document.

This document describes the retrospective for the Typography Refresh Beta Feature, conducted on 2014-04-17.

Attendees

Arthur Richards (Engineer, Scrummaster)
Erik Möller (VP of Product & Engineering)
Greg Grossmeier (Release Manager)
Howie Fung (Director of Product Development)
James Forrester (Beta Features Product Manager)
Jon Robson (Engineer)
Jared Zimmerman (Director of User Experience)
Keegan P. (Community Liason)
Moiz Syed (UX Designer)
Nick W. (Community Liason)
Ryan Kaldari (Engineer)
Steven Walling (Product Manager)
Vibha Bamba (UX Designer)

Project timeline

First pioneered font changes on mobile. With additional tweaking (including e.g. spacing), ported to desktop. Started as part of first set of beta features that were released. First released in November, with 3 additional major updates before going into core:
- Release on November 22 2013. Change vector sidebar width, link color in sidebar and personal toolbar, simplify external link styling. (Jon Robson, UX Team, no PM or CL support) Gerrit change 79948
- ~January 6th. We removed the personal toolbar/sidebar link color change and the sidebar change due to community feedback on Talk:Typography refresh. Measure limit (max-width 715px) was first in place. Added thumbnail and blockquote styling. This was the first release where FOSS fonts were first in the stack: DejaVu Serif for headings, Liberation Sans for body. Gerrit change 99523 (Jon Robson, UX Team, Steven first involved, no CL support)
- ~February 9th. Removed measure, added TOC styling. adjusted body and header size and leading. Gerrit change 108155 (Kaldari, UX Team, Steven, Nick involved as volunteer CL)
- March 11 - (Lightning deploy) was adding Arimo, Liberation Sans and Linux Libertine after testing by Kaldari and Vibha.
- March 14 - The main documentation got updated from the November version.
- March 26 The actual changes made to core in Gerrit change 120978 by Kaldari and Jon. Small cleanup to pre styles and sup in Gerrit change 121419. Beta feature turned off in VectorBeta extension in Gerrit change 121244 though the VectorBeta extension remains deployed and also contains so-far-unreleased Beta Features for Winter/Fixed header and Compact Personal Toolbar
- March 27 - Removed overflow styling for pre tags, added subscript styling (same as superscript) (Kaldari)
Released and announced (blog post)
- March 31st, mediawiki.org and test wikis
- April 1st, all non-Wikipedias
- April 3rd for all Wikipedias
Post-release changes:
- Monday April 7 - Remove troublesome fonts from font stack (wmf20, wmf21) went out via SWAT deploy by Roan. Bugs filed and fix tested the previous Friday by Jon, Steven, Jared, Vibha.
- Friday April 11 - Switch back to "sans-serif" for body text by Odder in Gerrit change 124475 merged by Bartosz Dziewoński
Steven creates Trello board for staff. Previously there was just VectorBeta Bugzilla component for issue tracking, as well as wiki page. Bugs now tracked at bug 63549
Major issues/bugs on release
- Original font stack caused problems for Windows users who had fonts like Arimo, DejaVu installed (e.g. via LibreOffice). These fonts render very poorly with ClearType/anti-aliasing disabled.
- Some users of wikis in non-Latin scripts dislike old-style numerals because they align poorly with characters in those scripts (they're designed to flow with Latin text). This was due to the selection of Georgia as a serif font. eg. 1984–85 NHL season or 9223372036854775807.
- Serif declaration caused "brush stroke" style characters in some CJK languages for heading, which was unexpected, and against design norms, but didn't pose readability issues.
In progress bug fixes etc.
- Work on per-language CSS specification via MW core by Jon et al. Gerrit change 125760

Retrospective

The questions for this retrospective are:

What should we continue doing with beta features development? What went well?
What should we do in the future to improve? What should we start trying?
What should we stop doing in developing beta features? What caused stress/blockers/slowed down velocity and quality?

Note these questions are oriented towards actions to take. Our feedback should be focused on things to continue/start/stop doing in our process, not on complaints about the content of the feature itself.

What should we continue doing with beta features development? What went well?

Group meetings to review community feedback (Jon, Steven, UX Team starting Jan 6)
Links to discussion pages directing people to MW
Cleaned up Vector's code to make it more flexible for future experiments (Vector CSS ported to LESS) (The patch to update core was tiny as a result)
We got feedback from the community \o/
Working in VectorBeta made changes quicker than they would have been in core
made actual changes in response to feedback :-)
Using VectorBeta to get early feedback and testings worked fairly well
"When it's done" approach, rolling with the punches to figure out the right time to deploy
Steven recognizing this could be controversial, trimming down scope of stable, This happened a few weeks prior to going to core
Kaldari and Vibha stepping up to test FOSS fonts based on feedback from Wikitech-l
Handoff between Jon and Kaldari back and forth went pretty smoothly
Useful help from technical Wikimedia volunteers e.g. Edokter, TheDJ, Bartosz, others.
Steven reached out to Research & Data about A/B testing necessary. Someone (WHO??) worked to get data on usage levels of the beta feature which went in to the FAQ (looks like Oliver gave us the data?)
Press work to get PR for this, show design leadership etc. including connecting to our values around free fonts etc.
Steven checked in with Mobile sprint planning to make sure that we weren't using too much of their time from. When we needed to pull back a little bit we did.
Keep talking honestly about how much time this takes and how it impacts the work of other teams.

What should we start doing in the future to improve?

Get our house in order first. Lots of infighting in Wikimedia. Didn't help in a public setting.
Talk pages were not effective for discussing and identifying real issues (Flow/Bugzilla could have been used better)
- I feel lots of questions/problems got lost
Have dedicated developer time to work on beta features (rather than relying on developers' "free time"), especially for launching feature
- Get two developers — code review was slow
  - (Erik) Let's figure out how to negotiate this upfront. This project was very bottom-up, which is cool, but tends to create these kinds of resourcing bottlenecks.
Have ongoing documentation updates.
- Eg. The ToC redesign was never mentioned, anywhere.
- Be exact about actual changes. People were asking for exact before/after "diffs" for months, and still don't know.
Notifications of any kind when a BF is updated, rather than just surprise.
- What kind of notifications? To whom? How many is too many?
  - Suggested Echo spec at Beta_Features/Echo_notifications; alternative suggestion from Pau (Beta Features-specific):
- The only notification there was, up until just-prior-to-deploy, was one post on the TR talkpage.
Have a very clear decision about what is and isn't included in each Beta Feature (and name and message it accordingly)
Make the BetaFeature *exactly the same as* what will go to production for some X amount of time (probably >=2 weeks), in this case scaling back what's in BetaFeatures. No last minute additions/removals (of any importance), but bug fixes are OK.
Think about other ways of asking for feedback that includes non-logged-in users (i.e.: some little call to action that non-logged-in users see)
Felt at times it was just Jon/Kaldari and Steven dealing with mailing list threads. Burnt Jon out this is why we handed over to Kaldari.
QA and testing
- Make people more aware of testing tools available.
- Have physical Windows and Linux machines available for testing.
- Nick put this together in November [officewiki listing of browsertesting tools]; he suggested that the design team might consider putting together (over the long-term) a long list of "100+ examples pages, that we internally use as test-beds, to highlight the Extremes, the Edge Cases, and the Exemplars: From English FAs and stubs and docs, to Chinese Wikivoyage Discussion Boards, to Wikidata, to Wikisource, to ...."
All Beta Features changes in state (enabled/visible to opt-in mainly) must be scheduled in advance on deployment/roadmap calendar.
We should learn from the Mobile web team and make a multi-level "beta" features... maybe call it "Alpha" > "Beta" > "Stable in production"
Howie and Greg work together to make a final call on whether something is okay to release as default/opt-out. (see resourcing/support for graduation below)
Give announcements on internal Engineering list, including with places to give feedback and hold IRC meetings if we think it's going to be internally contentious. Also use these as opportunities to ask for help.

Resources/support for graduation

Product management
Community Liasons
UX design
Engineering
Research & Data
QA
Performance review (N/A in this case)
Security review (N/A in this case)

What should we stop doing in developing beta features?

Remove "About Beta Features" and "Leave feedback" pages linked from the main beta features preferences page, because it mostly serves to distract users with the wrong link when they want to give specific feedback – there's almost no feedback given there about Beta Features per se, and instead it's all about individual features, splitting the feedback and isolating users' ability to inform and shape development.
Scope creep! The sidebar/personal bar/measure were kind of crazy.
Having to do lots of lightning deploys right before launch
Ability to give feedback became hidden for many users when the beta feature went away. Temporarily include a section at the bottom of "recently graduated" beta features with info/discussion links but no on/off