Wikimedia Discovery/Meetings/Search retrospective 2016-10-27

Format

Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Action items from the previous retro (2016-09-20)

Deb: Create a ticket to deal with bad data in discernatron
- Done: https://phabricator.wikimedia.org/T146189)
GL should alert ops to the page (who's who)
- Done

What has happened

Using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point

Conversations
- elastic search things (ICU_folding)
- external referrers
- how to get db access for investigations
- mock design for displaying the upcoming cross-wiki search results
- CompletionSuggester with defaultsort demo available here, feedback welcome
- sometimes we fall back to a language analyzer of a (linguistically or geopolitically) related language (e.g. Russian/Ukrainian)
- Had a discussion about needing variant options for Chinese in Language Settings
- Discussed the preliminary results on running Paulscore with BM25 on zh, ja, th
Analysis
- Completed analysis of squashing Russian stress accents and folding ё to е for Russian wikis.
- Decided on the outcome of the recent BM25 A/B test, what our next steps on using BM25 will be
Coding
- Updated ElasticSearch document versioning in CirrusSearch
- Updated Completion Suggester code for searching a subpage title using the search bar
- Added position increment gap to fields where positions are stored
- Fixed when search for insource:tag finds "<tag>" but not "{{#tag:tag}}
- CirrusSearch updated to allow searches in a specific language (without translations)
- Satisfaction schema update for checkin events on if page is visible
- Update the ascii folding to preserve original emits that are duplicated tokens for non ascii characters
- Update the search api interwiki to only query when srinterwiki=1 is passed
- Reduce the number of replicas for some titlesuggest indices
- Updated the search results page because the content overflows page boundaries
- Add accent squashing to Russian/Cyrillic analyser (with tests)
- Fixed Cyrillic 'Е' and 'Ё' equivalence not found by search
- Fixed issue where the search page content overflows the page boundaries on various browsers
- Updated where users prefer pages in their language in multilingual wikis
- Enable sub-phrase completion suggester on wikitech, mediawiki.org and wikisource
- Enable document versioning support via wikimedia-extra extension in production
- Remove custom analysis chains from vagrant
- Map modifier letter apostrophes to straight or curly quotes in the French Elasticsearch analysis chain
- added inlanguage: keyword
- Works in progress but not yet in production:
  - File property search functionality merged and documented. As soon as deployment happens (next week), the reindexing will be performed.
  - Put SITENAME in search box placeholder "searchsuggest-search" like in MobileFrontend's "mobile-frontend-placeholder
  - Integrate "did you mean" data collection into search satisfaction schema
  - Improve processing of the apostrophe by the search engine in Ukrainian
  - Enable Latvian and Lithuanian analyzers
  - CirrusSearch SQL query for locating pages for reindex performs poorly
  - MySQL chooses poor query plan for link counting query
  - When showing search results for different language, exclude images from Commons

Documentation
- Documented our usage of the term 'PaulScore'
- The Search team created a new ticket to note what will be 'live' in production during the next full re-indexing: [Recurring task] CirrusSearch: what is updated during re-indexing
- Request for translation sent for Update CirrusSearch documentation
Deployment/ops/system upgrades
- Upgraded ElasticSearch and plugins to 2.3.5
- Monitor the usage of in memory data structures used by ElasticSearch
- Code deployment freeze one week due to Ops having an offsite
- Added more information in order to estimate the hardware requirements for ordering new servers for Elasticsearch
- Decommissioned deployment-elastic08.deployment-prep.eqiad.wmflabs
- No train deployments the week of Oct 17-21 due to RelEng offsite
Team/process
- Dan and Deb will split up PM duties of the Discovery Search Team in the coming weeks

What has made you mad?

Reindexing is a huge pain +1+1
Interwiki support in MW is a mess oh yes it is+1+1+1

What has made you sad?

++++Code reviews are still fairly slow (UI related ones)
Languages without spaces get the short end of the stick w.r.t. search.
- We don't have analyzers/tokenizers that work well for specific languages (yet?)
No train deployments for a couple weeks put a bit of a cramp in everyone's style
Then, once the train did return, it couldn't leave the station a few times, resulting in code freezes
Deb and Dan have the same color in etherpad :P I (deb) had it first! NOT ANY MORE! AWWW YEAH PINK
even with advances in usage of clickstream data, it still doesn't seem particularly useful for measuring large changes that affect a significant part of the results.+1
+++++bm25 deployment going back and forth due to deployment issues and other inconsistencies in our stack+1
Getting everyone sick (apparently transmitted over google hangouts :P) +1+1-100
refactoring interwiki search in cirrussearch took forever
Looks like elasticsearch cluster recovery is not going to get faster any time soon (I still have a few minor idea about low level improvements, but don't expect much)
The engineering underspend has already been accounted for and we won't be able to get those language analyser contractors that we were dreaming about-1 +1 sad √1
There are no data analysts in this retrospective-1

What has made you glad?

lots of little fixes became high priority because of the upcoming BM25 reindexing, so lots of annoying non-English things got fixed.
++lots of little annoying 'bugs' got fixed - in spite of the upcoming BM25 / reindexing work!
++++Great input on the SERP design from team & wmf design team (as of yesterday)+1
Enjoyed various research necessary to figure out how to use clickstream data to evaluate search on wiki's we can't collect discernatron data for
Lots of talk and decisions made on the little fixes and what to do about Discernatron, etc - very good communication
The comment "Interwiki support in MW is a mess oh yes it is" above.+1
discernatron lunch
Good feedback from Mikhail and Chelsy for related investigations and testing
Discovery quarterly review - we had excellent answers to all the kinds of questions that were asked. Our approach is understood and appreciated by the senior management.+1
Trey continuing to plow away at the different top 10 lang investigations, even though it's fairly tedious :)+1
Lots of good repayment of technical debt due to the interwiki search refactor.+1+1
- A gentle reminder that product owners will consider prioritising technical debt if asked about it :-)+1

Other stuff

Discovery phab project still has a workboard, which is now redundant and not being used as far as Dan knows
- Kevin thinks it may not be possible to delete a workboard without archiving the project
  - can we convert Discovery to a tag maybe?
  - we will need to note this change (not using the discovery board) to our various communities and team pages
    - Do we? The project itself would still be used, and tasks would be triaged as they come in. AFAIK nobody uses the workboard except us?
  - If so, this is not really a big deal, we can probably just add a comment in somewhere that the workboard is deprecated and get rid of all but one of the columns?
    - Yes, you can hide (not delete) most if not all of the columns

Discussion

+++++bm25 deployment going back and forth due to deployment issues and other inconsistencies in our stack+1

It was a goal and we didn't achieve it. Could we have done something different?
We didn't get the test out early enough, so it wasn't analyzed in time for us to hit the goal
Some cases where code review didn't catch things
Reindexing/restarting the cluster has caused some problems
What does "inconsistencies" mean?
- There's code that we only use occasionally and then it 95% works
- "If it's painful, do it often"
Would love to have continuous reindexing; not sure how to do that
Could practice reindexing smaller wikis
After a reindex, noticed it was missing 200k items; our tools can't detect those inconsistencies (even though it is supposed to)
- Threshold is 5% which is probably too large
Short-term plan is to improve reindexing process

++++Great input on the SERP design from team & wmf design team (as of yesterday)+1

https://phabricator.wikimedia.org/T139310#2748175 (future vision; not near-term)
Lots of details that are questionable or need further thought; there's nothing in there that seems unresolvable :-)

Action items

Deb: Create phab task to look into improving reindexing - Done: https://phabricator.wikimedia.org/T149333
Dan: Create a task to experiment with continuous reindexing Same as ^ ?
GL: Open an upstream ticket about synced flush recovery restart (IIRC Clint was the elastic developer before that was going to help us, but i'd have to look back through my emails).
Kevin: Continue CR discussions
Dan: Look into modifying the #Discovery workboard in phab