Wikimedia Discovery/Meetings/Analysis retrospective 2016-11-08

Format

Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Previous action items

• Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
  • Include a statement of the perceived problem
  • Still needs to be done
• Deb: Figure out what to do with the Maps Dashboard
  •  Done - fixed and updated
• Mikhail: Think about involving GL in sys admin stuff
  • Need to meet w/GL

What has happened since the last retro (2016-09-30)

(using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point)

• Analysis of WDQS geographic breakdown of SPARQL queries
• Wikipedia.org Portal Dashboard: investigate spike in languages visited
• Wikipedia.org dashboard: minor updates
• Bug in current data retrieval script
• Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes
• Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager
• Completed analysis of the results of BM25 AB test (final report)
• Search results page: how many visitors are on mobile vs desktop -
• Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
• Investigate recent spike in pageviews on wikipedia.org portal page
• Add a PaulScore approximation to discovery.wmflabs.org (waiting on phab:T138087 to be deployed on the train)
• Compare ZRR for query features across other search engines
• Investigate what we'd need to do to ignore double quotes in search queries
• Maps Dashboard: fix and update
• Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
• Add a PaulScore approximation to discovery.wmflabs.org

What has made you mad?

• Interactions with Interactive team
  • e.g. we created a dashboard and have made many improvements on it since its deployment, but the Interactive team does not seem to use it and even cannot remember the URL at times
  • so being asked to learn a bunch of new technologies and do a bunch of additional work to make data available in a different data store so the team can do their own dashboarding is frustrating when current resource isn't being utilized

What has made you sad?

• Frustration with working with the Interactive team - trying to decide what is needed for metrics
• Lack of integration between Analysis and Interactive
  • Chelsy and Mikhail are a resource for Interactive to use, and they are being underused

What has made you glad?

• Chelsy is a super fast learner and has done terrific work on her reports+1
• Coordination with Search team on test metrics/analysis of tests - quick results
• Updating the dashboards to be much better
• M & C learning new things about R and the operations stack to get new and cool technology into our environment
  • Nice to be able to learn during work time
• Watching this after a tough election night: http://i.imgur.com/lZEeZ8z.gifv
• Kevin's involvement in Analysis; this team is better for having a TPG person+1+1

What has made you plaid? (aka whatever didn't fit in one of the other categories)

• There are several IP addresses have been doing weird things to us, e.g. pinging portal million times per day, sending the same search queries thousands of times per day. I know we are generally not blocking any of them, but wondering if we should keep a black list for ourselves, in order to remember to check them when we are doing related analysis so that we can filter out polluted data.
  • Maybe a greylist rather than blacklist, just to identify for reporting?
    • Publicizing the list wouldn't be appropriate (privacy issues)
  • Some cases would be a judgment call; might look useless to us, but maybe they're getting value
• and...this: http://i.imgur.com/sd15NVu.gifv

Discussion

Interactive team interactions

• KS: I can see 3 areas: 1) ED-level of what metrics should be tracked, 2) tech level of whether to use grafana or other, 3) inter-team interactions and social component
• They seem to be making progress on deciding what metrics to track
  • Interactive product folks (Deb/Yuri) should do some early thinking, then bring in analysts to review. Doesn't seem to be going that way so far
• For tech, there's a ticket to get data into grafana. That's frustrating because we have a dashboard system that we're really happy with, but they won't use it; want us to do additional work and learning to support this other system
  • Our dashboards can do some things that grafana can't
  • Unclear WHY
• Interactive team feels strongly about doing things their own way. That's why they chose the tech, and why they implemented their own metrics without consultations.
  • Can be frustrating to have expertise available to them that they don't take advantage of
  • Deb in semi-PM role is trying to help them with this
  • Discovery dashboard has all the vertical stuff in one place, consistent UI, etc. There are also valid reasons for having grafana.
• Was hoping that after we updated the graphs, they would work for that team. Yuri wants direct immediately control over dashboard changes, which they get with grafana.
  • Not clear that grafana will be able to produce output that will be useful for demonstrating KPIs to management and other stakeholders
• How big a project is grafana support?
  • Kind of a big project
  • Feels like a personal slight to not use our dashboards, but learning grafana would be a useful skill
• Even if the interactive team might not appreciate the maps dashboards, other people will
  • The Discovery dashboards need to track the work of the whole team
• Concern that Yuri might try to figure out KPIs and metrics in isolation
  • Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
    • Some optimism. There is a ticket out there.
    • Analysis team tries to be proactive and forward-looking; interactive team tends to be more quick with less forward planning

Action items

• Mikhail: Discuss opsy stuff w/GL (and Chelsy)
• Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
  • Include a statement of the perceived problem