Wikimedia Discovery/Meetings/Search retrospective 2016-09-20

Format

Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Action items

from last retro (2016-08-19)

Kevin: Invite GL to search planning meeting(s)
- Done
David: Will send email to private list requesting BM25 testing; later to public list (and Discovery weekly status)
- Done
Erik: work w/PO to make sure some time gets allocated to work on cindy problems
- Done enough +1

What has happened

(using Discovery/Status updates/Next as a starting point)

Features/fixes/code updates
- Search: deployed ascii folding / stemming re-ordering changes
- Search: disable an old search schema since a new one has already been deployed
- Search: implement an updated full text search query
- Search: Refactor GeoData elasticsearch mapping using the new search field definitions from core
- Search: update wildcard queries for BM25
- Search: Centralize document id and namespace generation
- Search: Enable BM25 by default in cirrus and evaluate impact
- Search: Add a "reverse" suggestion field
- Search: turned on BM25 A/B test after re-indexing the codfw seach cluster first
- Search: added patch to properly handle acronyms in fulltext query builder
- Search: Use top_terms_blended_freq for wildcard queries
- Search: updated fulltext query builder to enhance title
- Search: turned off 'did you mean' on Wikidata
- pushed into production a change that implemented ascii-folding for French
- turned off BM25 A/B test, awaiting analysis
- Automagically fixed ÿ in Spécial:IndexPages search by implementing ascii-folding in French
- Added defaultsearchkeys to wiki search autocomplete
- Fixed bug - mwgrep and "insource:" search is missing lots of pages in its index
- Fixed bug - "You may create the page" suggestion does not appear if search contains 'AND', 'OR', 'NOT' anywhere in search even when these are not used as special syntax
Deployment, clusters, etc.
- Search: added patch to ensure timeouts always work on CirrusSearch
- Improved balance of nodes across rows for ElasticSearch eqiad cluster
- Our elasticsearch clusters now have "row aware shard allocation". This means that we can theoretically lose one row of servers in our datacenter and still serve search traffic.
- Tried to update to to ElasticSearch plugins to 2.4.0 but found out we couldn't because our analysis plugins are not compatible
- Make elasticsearch actually have shard allocation awareness
- Updated WikimediaMaintenance/addWiki.php to create cirrus indices on all available clusters
- New wdqs cluster in codfw
Discernatron
- Search: worked on updating the prose and user flow of the Discernatron introduction and instructions page
- We had a Discernatration demo with free pizza on Tuesday, Sep 13 that was a success!
Team stuff
- Several Discovery team members attended a Product and Tech on-site two day meeting in the SF office
Research/analysis
- Search: evaluated BM25 with new fulltext query and weighted sum with PaulScore and Discernatron
- Search: tested and analyzed the effect of adding ascii-folding to fr.wiki
- Search: completed the analysis for optimizing language identification for the Dutch Wikipedia (nlwiki).
- Search team wrapped up researching the ElasticSearch instabilities on the eqiad search cluster on Aug 6, 2016; nothing conclusive was found.
- Started experimenting with tracking PaulScore on Search Metrics dashboard: http://discovery-beta.wmflabs.org/metrics/#paulscore_approx
Documentation
- Updated the Help:CirrusSearch documentation to reflect the great work Cpiral did in organizing the information in a more approachable and logical way. Kept existing translation tags in place and have asked for review. This is in preparation for this small fix to the "Help" link on every Wikimedia Special:Search page.
- There's a new glossary of search-related terms available on mediawiki.org. Feel free to request additional definitions!
Community
- Search team sent out a request for comment article that was posted to various Village Pumps asking for it to be translated.
  - This was in reference to the cross-wiki search results new functionality and design articles on MediaWiki.

What has made you mad?

Cindy, Cindy, Cindy [Note: Cindy is our automated testing system]
Inability to update ElasticSearch to 2.4.0
- Required changes, which are done, but need to be pushed into the prod cluster(reindex)
- Didn't lose too much time
Understanding disk and network IO is sooo hard
- GL will use the upcoming ops offsite to learn from others
why do clothing tags seem to be 4 inches long
- use scissors +1
There is no magic bullet to make elasticsearch cluster restarts fast (the easy "delayed_unassigned_shards" does not work)
- Longstanding/ongoing issue
- GL has a few ideas that might help a bit
Why do languages have to be so weird+1+1
- (mostly unrelated side note: I'm starting to learn Swiss sign language ... I thought Swiss were speaking FSL, but no)

What has made you sad?

It's hard to make Discernatron both interesting and useful.
- We're trying, but it's difficult to take something very complex and make it interesting easy to use
- Some ratings are suspect (e.g. 80% of results scored as relevant), so we need to figure out how to deal with that
only 2 nodes on relforge cluster => downtime when doing work on the cluster
- Budget constraints when it was set up led to this situation
- Probably OK as long as we're the only ones using it; live with it for now
wdqs deployments are not as regular as some would like
- Weekly schedule is not ideal for developers--consider 2x/wk? Probably not worth it at this point
- I think once a week is fine. If we have something really urgent, we'll handle it on exception basis.
Discovery's who's who page has not changed, but the discussion was interesting
- what's this? - this was in reference to who to call when things go bad (ops)
- Existing docs seem to be good enough unless we hear otherwise
- GL should alert ops to the page
- [After the meeting, we weren't sure which page GL was talking about, and identified several candidates. He meant the last one listed below.]
Uncertainty around what the cross-wiki functionality will actually do (We have options, but not an answer yet)
- Still fairly early in the process; Q2 goal has us exploring those options and deciding a good way forward
- Design by committee is slow (we're trying to gather input from a lot of people)
Some features can't be done as a beta features because they require different indexes or elastic configurations +1
- For BM25, we had different configs in eqiad vs. codfw
- We could write code to have double indexes for limited (smaller) wikis
- Whenever we increase the indexing rate (e.g. having 2 indices), searching gets slower
Satisfaction data collection never appears to be quite right, always find out later when we want to query something we don't quite have the right data +1+1+1
- Most recent example: We didn't record which autocomplete location (header/content) you clicked on when you clicked on it, so not enough for PaulScore
- It is just really hard to predict in advance what you might want later

What has made you glad?

BM25 is making progress!
Spark projects!+1 [Note: Spark is a WMF program to encourage employee creativity]
Discernatron demo - solitare style +1+1+1+1
- Got decent turnout for the Discernatron demo luncheon +1+1
- Jan put together the new UI quickly
Trey did great at documenting CirrusSearch help and search related glossary terms (i.e.: Paulscore)+1+1
- Trey putting together the search glossary+1
Feedback from cross-wiki search results consultation - all good and they want more
Translationwiki work from community for 'showing results from' +1
- For everyone person that says "You're overloading the translators" there are just as many people that would rather waste a bit of time translating than have something untranslated. :-) +1
Kevin pre-populating the list of "what's happened" for this retro +1 +1+1+1
SPARQL workshop went well, good guest presenters too+1+1
elasticsearch row aware allocation was much simpler than I thought! Except it brings a few new problems
Ops team offsite next week! So happy to finally meet them in person!
Fixing Special:Search Help link on Wikimedia projects (and updating documentation with volunteers) +1+1
Help from Chris and Benoit on getting request for comment/translations on the Village Pumps
Relforge servers! +1+1
- Incredibly useful for early enwiki testing and commons wiki
Positive feedback at tech/product onsite about our upcoming quarterly goals (cross-wiki, advanced user features/query parser)+1
Languages are so weird!-1 +१
subphrase/subpage autocomplete search on mediawiki.org/wikitech will be great (maybe other places too :P)!
- always leave them wanting more? ;)
Lots of kool demo at last CREDIT showcase! I love the WDQS / graph integration!+1 +1+1+1
- Probably need to productize it finally then :)
New wdqs servers, yay!
- Is fail-over documented? Not yet--working on understanding how (don't use LVS); phab task already exists

Other

DevSummit is being planned. Consider what Discovery might want to get out of that event

Action items

Deb: Create a ticket to deal with bad data in discernatron (done: https://phabricator.wikimedia.org/T146189)
GL should alert ops to the page (who's who)