Wikimedia Discovery/Meetings/Analysis retrospective 2016-11-08

Format

Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Previous action items

Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
- Include a statement of the perceived problem
- Still needs to be done
Deb: Figure out what to do with the Maps Dashboard
- Done - fixed and updated
Mikhail: Think about involving GL in sys admin stuff
- Need to meet w/GL

What has happened since the last retro (2016-09-30)

(using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point)

Analysis of WDQS geographic breakdown of SPARQL queries
Wikipedia.org Portal Dashboard: investigate spike in languages visited
Wikipedia.org dashboard: minor updates
Bug in current data retrieval script
Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes
Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager
Completed analysis of the results of BM25 AB test (final report)
Search results page: how many visitors are on mobile vs desktop -
Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
Investigate recent spike in pageviews on wikipedia.org portal page
Add a PaulScore approximation to discovery.wmflabs.org (waiting on phab:T138087 to be deployed on the train)
Compare ZRR for query features across other search engines
Investigate what we'd need to do to ignore double quotes in search queries
Maps Dashboard: fix and update
Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
Add a PaulScore approximation to discovery.wmflabs.org

What has made you mad?

Interactions with Interactive team
- e.g. we created a dashboard and have made many improvements on it since its deployment, but the Interactive team does not seem to use it and even cannot remember the URL at times
- so being asked to learn a bunch of new technologies and do a bunch of additional work to make data available in a different data store so the team can do their own dashboarding is frustrating when current resource isn't being utilized

What has made you sad?

Frustration with working with the Interactive team - trying to decide what is needed for metrics
Lack of integration between Analysis and Interactive
- Chelsy and Mikhail are a resource for Interactive to use, and they are being underused

What has made you glad?

Chelsy is a super fast learner and has done terrific work on her reports+1
Coordination with Search team on test metrics/analysis of tests - quick results
Updating the dashboards to be much better
M & C learning new things about R and the operations stack to get new and cool technology into our environment
- Nice to be able to learn during work time
Watching this after a tough election night: http://i.imgur.com/lZEeZ8z.gifv
Kevin's involvement in Analysis; this team is better for having a TPG person+1+1

What has made you plaid? (aka whatever didn't fit in one of the other categories)

There are several IP addresses have been doing weird things to us, e.g. pinging portal million times per day, sending the same search queries thousands of times per day. I know we are generally not blocking any of them, but wondering if we should keep a black list for ourselves, in order to remember to check them when we are doing related analysis so that we can filter out polluted data.
- Maybe a greylist rather than blacklist, just to identify for reporting?
  - Publicizing the list wouldn't be appropriate (privacy issues)
- Some cases would be a judgment call; might look useless to us, but maybe they're getting value
and...this: http://i.imgur.com/sd15NVu.gifv

Discussion

Interactive team interactions

KS: I can see 3 areas: 1) ED-level of what metrics should be tracked, 2) tech level of whether to use grafana or other, 3) inter-team interactions and social component
They seem to be making progress on deciding what metrics to track
- Interactive product folks (Deb/Yuri) should do some early thinking, then bring in analysts to review. Doesn't seem to be going that way so far
For tech, there's a ticket to get data into grafana. That's frustrating because we have a dashboard system that we're really happy with, but they won't use it; want us to do additional work and learning to support this other system
- Our dashboards can do some things that grafana can't
- Unclear WHY
Interactive team feels strongly about doing things their own way. That's why they chose the tech, and why they implemented their own metrics without consultations.
- Can be frustrating to have expertise available to them that they don't take advantage of
- Deb in semi-PM role is trying to help them with this
- Discovery dashboard has all the vertical stuff in one place, consistent UI, etc. There are also valid reasons for having grafana.
Was hoping that after we updated the graphs, they would work for that team. Yuri wants direct immediately control over dashboard changes, which they get with grafana.
- Not clear that grafana will be able to produce output that will be useful for demonstrating KPIs to management and other stakeholders
How big a project is grafana support?
- Kind of a big project
- Feels like a personal slight to not use our dashboards, but learning grafana would be a useful skill
Even if the interactive team might not appreciate the maps dashboards, other people will
- The Discovery dashboards need to track the work of the whole team
Concern that Yuri might try to figure out KPIs and metrics in isolation
- Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
  - Some optimism. There is a ticket out there.
  - Analysis team tries to be proactive and forward-looking; interactive team tends to be more quick with less forward planning

Action items

Mikhail: Discuss opsy stuff w/GL (and Chelsy)
Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
- Include a statement of the perceived problem