Wikimedia Release Engineering Team/DataDataData Sync Up/2019-02-12

2019-02-12

Late time

• There was no last time. This is the first meeting.

Today's Agenda

• Meeting logistics
  • Let's meet bi-weekly
• Antoine and Dan talked with Joseph from Analytics about getting help on this project
  • They want to show off their tech to other teams, and we want to store and analyze lots of stuff
    • A potential match. Let's set up a meeting or draft an email about what we need.
• Dan is documenting schemas to help tease out what kind of storage and query we'll need to discover all kinds of trends and relationships in our data
  • https://drive.google.com/drive/u/0/folders/1iNPPC4U5PjLoaFuixNQKV0kWlNtWurn9
• It would also be good to show Analytics what raw formats our data will come in
  • Jenkins: https://github.com/jenkinsci/statistics-gatherer-plugin
  • Gerrit: https://gerrit-review.googlesource.com/Documentation/cmd-stream-events.html
  • and https://gerrit-review.googlesource.com/Documentation/json.html#change
• Do we need a project/board in Phab?

Outline plan for Analytics

JR: Analytics might have thoughts on having easy access to large pool of data and querying that is flexible to changing needs

What data we have currently or are planning to collect

• Schema
• Data samples

How we might want to query that data

• Our data is highly structured (see schemas)
  • Is Hadoop or ES more appropriate for that? Would we lose structure by putting it in Hadoop?
  • How much do we have to know about how data's structure before we put it in ES?
    • Can relationships/schema be changed after data is stored?

TODOs (by next meeting)

• JR to create phab tracking task
• Dan to draft email to Analytics