Jump to content

Discovery/Meetings/Quick surveys 2016-01-26

From mediawiki.org

Quick Survey meeting

Agenda:

  • Discuss and resolve some specific technical details that may block Discovery's use of the Quick Survey tool that Reading is developing
  • Discuss different surveys that Discovery might want to run

Attendees:

  • Dan, Deb, Erik, Jan, Julien, Kevin, Moiz, Oliver

What will QS be doing for us this quarter?

  • Last quarter's goal: After a user searches and ends up on a resulting article page, did they find what they were looking for?
  • Portal: Why are they on the page? What do they hope to accomplish?


We have not yet run any QS. We wanted to last Q, but the QS tech wasn't ready yet. What limitations in December?

  • It was a link out to a different site
  • Needed to be re-implemented to actually ask the question inline


Discussion w/Adam (technical)

  • For search satisfaction, we generate a new random session id for each user every 10 minutes.
  • When we show the survey question, we no longer have the session id
    • So no way to connect the answer to the search query that was submitted
  • Solution: Use mw_user.sessionid (or similar) plus the survey id, hashed.


Discussion

  • Need Erik and Oliver to validate that this approach will work
    • Erik: Does it remain consistent for anon users?
      • Will we capture the essence of the survey question and the page that the user was on?
      • Erik will talk to Adam, but maybe we can just try it and see if it works
      • Maybe it's OK that we only get data for logged-in users (not ideal)
    • Oliver: How long do mw session id last?
      • If longer than before, then more events would be tied to each id
      • Store mw session in new field, and keep using our random session id as we have been
      • Use old field for existing metrics, new field for new metrics
    • Dan: We'll have to modify the schema regardless
  • How is mw session generated?
    • Per browser?
    • If anonymous, you might get the cache, so wouldn't hit the app server, so wouldn't generate an id
    • Their survey may have only been for logged-in users
      • I'm sure that there are many more users that aren't logged in than are logged in. Could skew the results quite a bit
  • Clicking on the second listed result might be what the user wanted, so they could answer positively in the survey
    • Do we want to know that 50% of searches achieved satisfaction?
    • Or dig deeper, to know which item they clicked on, how many things they clicked around on before getting to the survey?
    • Oliver: We don't plan to use the ms session id for anything substantial
      • We could use timestamps to line up survey results with their actual flow, so yes
      • We want to build a model out of the quantitative data, and then use qualitative to test against it
      • Survey will only appear on some users' results, so we should get an unbiased sample
  • Are we combining the search and portal QS?
    • On portal, we can show a QS, but we can't generate a session id.
    • Eventually, we could run a QS on the page they landed on after leaving the portal
      • But we wouldn't easily have full tracking of how they got there
    • Oliver: Propose not talking about the portal survey at all yet, to avoid confusion
      • Revisit portal survey after the search survey is working (since it's a quarterly goal)
  • Internal survey doesn't offer a privacy policy option
    • There is a task and Julien proposed a patch: T119149
    • We need this before running any survey
    • Julien will follow up on this to get T119149 merged
  • Oliver: Need a task to change the schema to include the new mw session id field
    • T119153 looks related? Julien: This has already been done by QS team, so we don't need it (should decline or mark invalid--Julien?)
    • Need to create a new ticket in search sprint to add new id to seach schema (Dan created T125236)
  • Oliver: We should have Mikhail confirm that this plan makes sense (who?)
  • How long do we want to gather results?
    • Oliver: Probably a week of results should be fine. We don't want it up too long or too frequently.

When can we actually do this?

  • This isn't a quarterly goal, but is one of our 4 KPIs
  • There are some schedule issues in late Feb where we'll be short of analysis resources
Retrieved from "https://www.mediawiki.org/w/index.php?title=Discovery/Meetings/Quick_surveys_2016-01-26&oldid=2037795"