Jump to content

Wikimedia Discovery/Meetings/Discovery retrospective 2017-07-13

From mediawiki.org

Previous (non-offsite) retrospective: https://www.mediawiki.org/wiki/Wikimedia_Discovery/Meetings/Discovery_retrospective_2017-03-16

Previous action items

[edit]

From 2017-03-16

  • KS Follow up on other items that had votes
    • [2] ** Do other teams in Discovery feel like we (Analysis team) are supporting them enough?
    • [1] * Recent Collab Jam was insightful for me - but also confusing as annual planning was around the corner and budget was unknown and there is a possible re-org that people have been talking about for product and tech. Lots of swirling ideas. (NO ACTION NEEDED)
  • DG?: follow up on the collaboration between interactive and analysis, there are some unresolved issues there… (Related: Mikhail is doing some analytics for Maps in T170022) (NO ACTION NEEDED)
  • ErikB: Create automated A/B testing ticket, if it hasn't already been done (Chelsy is working on T131795) (WORKING ON)

From Offsite:

  • KS: Share tune-up notes with management DONE

Issues raised in team retros

[edit]

(No teams have had retros since April)

  • Still awaiting a decision on the future of maps
  • Pain getting code reviews from outside a given team
  • Product team (Discovery) owning non-product infrastructure (Logstash) feels weird and out of scope, although it makes perfect sense from a technological perspective
  • Consider dependencies between quarterly goals (e.g., language analyzers needed to wait for ES 5 update)
  • Follow up on lottery factor related to pyspark etc. (Getting David up to speed)
  • NOTE: Kevin notes that he did not do a good job of following up on action items for the team retros in April; each team should probably go back and look at the notes to see if any not-done action items are still worth doing

What notable has happened since the offsite (aside from implementing the tune-up)

[edit]
  • Sister search launched
  • OSC consultant (Daniel) started on Learn to Rank (LTR)
  • Mjolnir (meow meow) is able to forge new models
  • Explore Similar A/B test
  • Julien has departed (“left the building”)
  • Max ran away from us
  • Dan was taken from us
  • Kevin is missed

What is going well (including with the new structure)?

[edit]
  • Doesn’t seem very different from old way of working (for now)
  • One less layer of management (on Search Platform side)
  • Analytics has yet to complain about overusing the infrastructure for Mjolnir.
  • LTR models claim to have captured ~15-20% of the possible improvement, at least against the popular queries it’s training against.
  • Erik is going to Lucene/Solr Revolution next month, lots of good talks including some LTR stuff.
  • Performance reviews & peer reviews worked better than last year.
    • Last year, Tomasz was on his way out [during review season], which was a bit awkward
    • This year’s form was shorter, too.

What are some new or ongoing pain points?

[edit]
  • Figuring out new metrics that are wanted in our new teams
  • Some mysterious load issues on the elasticsearch cluster
  • WDQS could use more resources (PM, another engineer)
  • LTR going slower than hoped
  • 3 [PN] (EBj)[MP] So many information channels for status updates (SoS-Grace, Discovery Weekly-Chris, Search Platform for weekly tech mgmt meeting-Erika, Reading Monthly-Chris, others?)
  • 4 <EJB> [CX][SM][MP]Hard to run multiple AB tests in parallel, (or is it?)
  • (py)Spark is hard and mysterious
    • today wasn’t pysparks fault, just where the error was reported:P
  • Figuring out budget constraints
  • 1 [CK] Discovery status weekly email is continuing, but now added to Readers monthly email.
  • 3 [JD][PN] [dt] Asking for help for ongoing map issues is somewhat painful
  • Performance reviews were a bit difficult this year (better software, but odd for manager reviews)
    • Tune-up timing was awkward (managers changing)
  • 2 [dt] [JD] Weirdness with sister project usage (maybe)

Any tiny things we could experiment with?

[edit]
  • 4 [TJ][PN] (EBj) [dt] [SM]Facilitated cross-team conversations (i.e., ElasticSearch, machine learning, etc…)
    • There is already a weekly ‘Research lab’ meeting where ML type stuff is presented, although i (erik) have schedule conflicts
  • Technical blog posting (ala https://blog.wikimedia.org/2017/03/14/building-wikipedia-search/)

Other topics that don’t fit one of those sections

[edit]
  • Team retros going forward
    • Do we want to do them? (Yes! (EBj))
    • Who will facilitate?
  • All-Discovery retros going forward?
  • Chris is adjusting, so far without much pain, to having 5 PMs and teams to support
  • Upcoming Wikimania table of Discovery Search experts for questions (“Ask a Search Nerd”)

Discussion

[edit]

Candidate topics sorted by votes:

  • 5 <EJB>[TJ] [dt] [SM] (EBj) WMDE released a tech wishlist for search (what, if anything, do we do with this?): https://de.wikipedia.org/wiki/Wikipedia:Umfragen/Technische_W%C3%BCnsche_2017/Suchen
  • 4 <EJB> [CX][SM][MP]Hard to run multiple AB tests in parallel, (or is it?)
  • 4 [TJ][PN] (EBj) [dt] Facilitated cross-team conversations (i.e., ElasticSearch, machine learning, etc…)
  • 3 [PN] (EBj)[MP] So many information channels for status updates (SoS-Grace, Discovery Weekly-Chris, Search Platform for weekly tech mgmt meeting-Erika, Reading Monthly-Chris, others?)
  • 3 [JD][PN] [dt] Asking for help for ongoing map issues is somewhat painful
  • 2 [dt] [JD] Weirdness with sister project usage (maybe)
  • 1 [CK] Discovery status weekly email is continuing, but now added to Readers monthly email.

WMDE released a tech wishlist for search

  • Erika: Not sure what to do with it; seems like synergies. How do we/should we deal with it?
  • DT: Couple conversations about some of these. Don’t think we have worked on any of them yet. Would be great if we could take what they do to a bigger scale. As a PM, I want to stay aware of it. As soon as they have something good and tested, I would like to see about getting it implemented.
  • JD: Does anyone have more info on the point about language links on search pages
  • CK: Check with Birget
  • Erik: I think they’re talking about choosing a language in which you want to search
  • DT: We have some tickets like that in our backlog, but not prioritized
  • Erika: They have this one list just for search, so clearly it’s hugely important to them.
  • KS: Would it be worth it to get it translated?
  • Erika/Erik: Google translate seems good enough
  • DT: One thing we talked about with WMDE at the hackathon was surfacing sub-categories, which is not on this list, but we are actively working on it.

Hard to run multiple AB tests in parallel, (or is it?)

  • Erik: I added this. Maybe too technical. Maybe we can, or not. We would have to get together to think about it. Since we’re able to build models, we would like to put out a test, but it’s stalled waiting for the other test to finish.
  • JD: Related is de-duping of A/B tests. [?]
  • DT: Also, QA env that Nuria is setting up, to run multiple A/B tests would be cool, but not ready yet
  • Erik: That will require some work on our end to do things their way
  • KS: Worth some follow-up
  • JD: Probably. Technically seems OK to do multiple
  • Erik: I think with multiple buckets, yes. Sampling might become interesting
  • SM: The things we’re testing are often unrelated, so should be fine.
  • MP: don’t get users in more than one test at a time. I support what Jan said (debugging tests)
  • JD: Erik already has a patch which would help us with that. I can review it.
  • Erik: There are browser tests in that repo, but not automatically run, so might not work. Probably worth getting those working and writing new tests to make sure expected data is being collected
  • JD: Similar story on portal tests

Facilitated cross-team conversations (i.e., ElasticSearch, machine learning, etc…)

  • Erika: Also mine. The spirit is not necessarily ongoing, but sharing needs and best practices around some things that are more broadly used across the org. Pooling ideas and resources. I know there is interest, such as 4 technology mgrs interested in ElasticSearch
  • PN: I would be interested in talking with others about postgres
  • DT: Are convos wanted because you’re part of tech now? Or was the need there earlier?
  • Erika: People in tech-mgt are interested because I asked them. When I see a tech popping up across multiple teams, I would like to share best practices. So when they converge, it won’t be that painful. Or knowing places where they’ll never converge.
  • KS: Action item?
  • Erika: I’m going to call people together to discuss elasticsearch, after quarterly checkins. Probably in a couple weeks.

Action items

[edit]
  • MP: Follow up on parallel AB tests and debugging/QA
    • I can take this action item. - MP
Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Discovery/Meetings/Discovery_retrospective_2017-07-13&oldid=3944607"