Jump to content

Architecture meetings/Performance guidelines discussion 2014-05-14

From mediawiki.org

21:00-22:00 UTC, in #wikimedia-office.

We discussed the Performance guidelines draft, and secondarily the architecture guidelines and security guidelines.

Meeting summary and logs

[edit]


Meeting ended at 22:02:21 UTC.


Action items

[edit]
  • parent5446 to write "picking the right cache: a guide for MW developers"
  • sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores!
  • sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example
  • sumanah to clarify object cache vs memcached in Caching Layers section
  • sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section


Action items, by person

[edit]
  • parent5446
    • parent5446 to write "picking the right cache: a guide for MW developers"
  • sumanah
    • sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores!
    • sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example
    • sumanah to clarify object cache vs memcached in Caching Layers section
    • sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section


People present (lines said)

[edit]
  • sumanah (114)
  • RoanKattouw (34)
  • parent5446 (27)
  • ^d (22)
  • superm401 (21)
  • ebernhardson (19)
  • jeremyb (15)
  • bawolff (14)
  • bd808 (14)
  • wm-labs-meetbot (8)
  • AndyRussG (6)
  • robla (4)
  • StevenW (2)
  • csteipp (2)
  • Krenair (1)
  • marktraceur (1)
  • Krinkle (1)
  • TimStarling (0)
  • paravoid (0)
  • ori (0)
  • brion (0)


Generated by MeetBot 0.1.4 (http://wiki.debian.org/MeetBot)

Full log

[edit]
Meeting logs


21:00:09 <sumanah> #startmeeting Discussion of Performance guidelines | Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE). https://meta.wikimedia.org/wiki/IRC_office_hours
21:00:09 <wm-labs-meetbot> Meeting started Wed May 14 21:00:09 2014 UTC and is due to finish in 60 minutes.  The chair is sumanah. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:09 <wm-labs-meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:09 <wm-labs-meetbot> The meeting name has been set to 'discussion_of_performance_guidelines___channel_is_logged_and_publicly_posted__do_not_remove_this_note___https___meta_wikimedia_org_wiki_irc_office_hours'
21:00:16 <sumanah> #chair sumanah brion TimStarling paravoid ori
21:00:16 <wm-labs-meetbot> Warning: Nick not in channel: brion
21:00:16 <wm-labs-meetbot> Warning: Nick not in channel: TimStarling
21:00:16 <wm-labs-meetbot> Warning: Nick not in channel: paravoid
21:00:16 <wm-labs-meetbot> Warning: Nick not in channel: ori
21:00:16 <wm-labs-meetbot> Current chairs: TimStarling brion ori paravoid sumanah
21:00:32 <sumanah> hmmm
21:00:36 <parent5446> lol
21:00:52 <parent5446> It's the Zurich meeting all over again
21:00:56 <sumanah> ha!
21:01:04 <sumanah> it is ok since this is not a "please approve this" meeting
21:01:19 <sumanah> #link https://www.mediawiki.org/wiki/Architecture_meetings/Performance_guidelines_discussion_2014-05-14
21:01:42 <sumanah> #topic Performance guidelines
21:01:42 <marktraceur> Krinkle is here this time at least!
21:01:45 <sumanah> True!
21:01:49 <sumanah> #link https://www.mediawiki.org/wiki/Performance_guidelines
21:02:10 <sumanah> (As I said in my email to wikitech-l:
21:02:11 <sumanah> We discussed this some in Zurich but I'd love a
21:02:12 <sumanah> chance to ask some followup questions to firm everything up. I'd also
21:02:12 <sumanah> welcome the chance to explain the two similar documents I'm working on:
21:02:12 <sumanah> architecture and security guidelines.
21:02:13 <sumanah> )
21:02:36 <Krinkle> I am
21:02:37 <sumanah> #info We chatted about this during a discussion at the Zurich hackathon and it sounds like this document is about 80-90% there
21:02:51 <sumanah> #info I already have a big list of TODOs that I want to hit today and tomorrow https://www.mediawiki.org/wiki/Talk:Performance_guidelines#TODO_for_Sumana.2C_14_May_2014 before circulating this on #wikitech-l
21:03:07 <sumanah> #info The performance guidelines, along with the upcoming security guidelines https://www.mediawiki.org/wiki/Security_for_developers/Architecture , interrelate with https://www.mediawiki.org/wiki/Architecture_guidelines - the idea is for new developers to look at those 3 docs which describe how we do things
21:03:19 <sumanah> #info they are descriptive, not hypothetical/aspirational
21:04:35 <sumanah> If anyone has any questions for me - "what's the point", "why are these 2 principles separate", etc., I am happy to answer
21:05:06 <sumanah> I can also use this time to ask a few questions :-) for instance: should I list memcache as a persistence layer? maybe I should say, "maybe use memcache (and/or other caches) INSTEAD OF a persistence layer!"
21:05:57 <parent5446> Items in memcached can be removed randomly at the will of the server
21:06:07 <parent5446> So I'd recommend not listing it as persistence
21:06:19 <parent5446> Even if, in practice, WMF memcache servers tend not to delete anything unless they expire
21:06:24 <ebernhardson> redis would be the 'in-memory' persistence store
21:06:27 <parent5446> [citation needed]
21:07:02 <sumanah> So, in https://www.mediawiki.org/wiki/Performance_guidelines#Persistence_layer right now I say: "memcached cache - storage for things that persist between requests, and that you don't need to keep - you're fine with losing any one thing. Use memcached to store objects if the database COULD recreate them but it would be computationally expensive to do so."
21:07:04 <parent5446> Note, however, that Redis when used as $wgMemc should not be considered persistence
21:07:21 <jeremyb> memcache also deletes in case of server reboot and doesn't gracefully handle outage of a node (in case you have a memcache cluster with a number of nodes)
21:07:41 <parent5446> sumanah: yeah that is good, but I'd recommend not mentioned memached specifically
21:07:45 <sumanah> Why not?
21:07:47 <jeremyb> you can't shrink or grow memcache node count without pain
21:07:55 <parent5446> In reality, it's $wgMemc (or any BagOStuff object) that we should talk about
21:08:04 <parent5446> $wgMemc can be memcached
21:08:07 <parent5446> Or it can be another backend
21:08:12 <parent5446> It's abstracted
21:08:17 <jeremyb> so you always have extra memcache hot spare not in use and then can substitute in for a dead host's slot
21:08:49 <sumanah> parent5446: I am willing to, in this document, often say something like "a cache (such as memcached)" or "a unique identifier (such as a primary key)". Some people learn better by thinking of general classes, and some through concrete examples
21:09:36 <parent5446> Yeah that's fine
21:09:41 <sumanah> Thanks
21:09:53 <parent5446> I mean, in the code we make the same mistake all the time anyway
21:09:56 <ebernhardson> i suppose my worry is more about using the words memcache and persistence anywhere close to eachother :)  memcache should be treated as something more ephemeral than a persistence layer
21:09:57 <sumanah> parent5446: if you feel like taking a crack at writing "picking the right cache: a guide" then I would love that
21:10:22 <sumanah> ebernhardson: Roan said (quoting from notes): If something is very expensive to recompute ... use something closer to a store. (we use backend Varnish, which we call a cache, which is kind of closer to store on the cache-to-store spectrum.)
21:10:25 <parent5446> Yeah sure, put that an #action or whatever
21:11:11 <ebernhardson> sumanah: i imagine hes talking about the parser cache there?  I suppose we then need to declare levels of 'computationally expensive'
21:11:11 <sumanah> #action parent5446 to write "picking the right cache: a guide for MW developers"
21:11:18 <jeremyb> sumanah: right, but "store" or "persistence" can also be for canonical (permanent storage). memcache definitely isn't that
21:11:21 <ebernhardson> because what (i think) roan is talking about there is things that can take multiple seconds
21:11:51 <ebernhardson> in which case we dont just want them in an ephemeral cache, but also (in the case of parser) writen to a database
21:11:58 <sumanah> ebernhardson: jeremyb if there is an existing guide out there to our caching layers and which ones are for which purposes, that's better than https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers , please tell me so I can link to it
21:13:21 <ebernhardson> not aware of anything better
21:13:37 <bd808> sumanah: For what it's worth, I think the section you have for "persistence" looks good. Taken in context you are saying the right things.
21:13:56 <bd808> Taken out of context there is ambiguity in the wording
21:14:00 <sumanah> nod, nod
21:14:19 <RoanKattouw> sumanah: I'm here
21:14:29 <sumanah> #action sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores!
21:14:34 <sumanah> Thank you RoanKattouw
21:14:36 <sumanah> #idea I want to check whether we think it's a first-class performance goal to make anonymous browsing and logged-in browsing the same speed.
21:14:48 <RoanKattouw> I think it is
21:14:59 <parent5446> That'd be amazing if we could achieve that
21:15:02 <sumanah> RoanKattouw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140514.txt This meeting: Starts 2100.
21:15:06 <RoanKattouw> Some people have been talking about this but there are no real efforts to do that yet
21:15:18 <sumanah> OK, so, would you put this goal into the aspirational bucket rather than the descriptive bucket?
21:15:19 <bd808> So... make the app servers as fast as Varnish? Seems unlikely.
21:15:35 <parent5446> I think it means using more Varnish caching for logged-in users.
21:15:39 <RoanKattouw> Yes, exactly
21:15:41 <parent5446> Caching as much as possible
21:15:59 <ebernhardson> bd808: more like turn off user specialization so everyone can get the same varnish output (with some ESI)
21:16:07 <RoanKattouw> Which would also make things faster for logged-in users in, say, Europe
21:16:20 <RoanKattouw> And exactly, we'd need techniques like ESI
21:16:27 * bd808 nods
21:16:33 <RoanKattouw> Right now we don't use ESI at all but we would want serious ESI magic to pull this off
21:16:40 <sumanah> OK, so it sounds like this goes into the vision doc, agreed?
21:16:42 <RoanKattouw> So it's probably a ways out to actually do this
21:16:56 <RoanKattouw> sumanah: I assume the vision doc is for pie in the sky stuff?
21:17:02 <sumanah> ("vision doc" is what I am calling the "what we want to do in the future" thing that will probably be across security, perf, and architecture)
21:17:07 <RoanKattouw> Or rather, is where pie in the sky stuff goes among other things
21:17:28 <bd808> Mmmmm... mysterious sky pie
21:17:41 <sumanah> So, agreed?
21:17:44 <RoanKattouw> Right
21:17:47 <^d> The other option is slowing down anonymous users to be more like logged in users. Might be less work than the other way-round.
21:17:49 <^d> </troll>
21:17:54 * RoanKattouw agrees
21:17:56 <sumanah> ouch :-)
21:17:59 <sumanah> ok
21:18:00 <RoanKattouw> (with sumanah , not ^d )
21:18:13 <^d> Aw, I was hoping you agreed with me there :p
21:18:22 <sumanah> #agreed that the goal of making anonymous browsing and logged-in browsing the same speed goes into the future architecture vision document
21:18:48 <sumanah> btw RoanKattouw, Tyler agreed to write a "so you need to pick the right cache for your app needs" doc :D
21:18:54 <RoanKattouw> Sweet
21:19:27 <sumanah> So, another minitopic
21:19:29 <sumanah> I have in my notes:
21:19:30 <sumanah> "Maybe going too much into details: in context of frontend caching & Varnish - do not serve different content from the same URL. Diff content, diff URL. Must also be true for anon users! URLs must be deterministic - with proviso that not if you are logged in! Then wrinkles - cookies submitted ..... caching layers..... should strive to cache logged in requests..... longstanding problem."
21:19:54 <RoanKattouw> This sounds like a "how to not break things in the face of caching" thing?
21:20:09 <sumanah> The fact that something here is called a "longstanding problem" makes my ears perk up .... what bit here is the problem? That is, what bit should be going in the vision doc?
21:20:27 <sumanah> I presume it's "we want to someday cache logged-in users". Am I right?
21:20:32 <RoanKattouw> Yes I think so
21:20:39 <RoanKattouw> Which is what we just agreed to put in the vision doc already
21:20:55 <ebernhardson> i could be off base, but i think the 'longstanding problem' is that we vary by query string instead of the main url.  its not an end of the world type thing though
21:21:03 <sumanah> OK! Just wanted to confirm and make sure I'm understanding right
21:21:12 <bd808> It's very easy to break the ability of Varnish to cache content.
21:21:20 <sumanah> (with this stuff I am very aware that I might think "that is the same thing" when actually there is an important distinction I am missing)
21:21:43 <bd808> This might be the biggest WTF for new developers in our stack.
21:22:01 <sumanah> bd808: How do developers tend to do it, when they do break things accidentally?
21:22:03 <RoanKattouw> sumanah: They're not exactly the same thing if you wanna be pedantic (one is a goal, the other is an implementation), but to most people said implementation is the de facto way to achieve said goal
21:22:07 <sumanah> nod, nod
21:22:40 <bd808> sumanah: One way is by adding a new cookie. That happened just a couple weeks ago from ... somebody.
21:22:49 <sumanah> Yup. Matt, GettingStarted.
21:22:57 <sumanah> It's a "bad example" in that section.
21:23:05 <ebernhardson> but not all new cookies, just cookies that match a particular regex that no developer would know exists
21:23:07 <sumanah> But are there others I should mention explicitly?
21:23:19 <sumanah> nod, nod. ebernhardson is that regex available somewhere that I should link to?
21:23:28 <RoanKattouw> We tend to have things relatively well tied up such that you're unlikely to break Varnish's ability to cache content (except for the GettingStarted thing, but that had never happened before), it's much more likely that Varnish's success in caching content will break what you're trying to do
21:23:57 <sumanah> heh
21:24:00 <ebernhardson> sumanah: not sure, someone in ops(or maybe roan ;) would know
21:24:16 <ebernhardson> the varnish config is so far out of an average developers perview
21:24:27 <robla> RoanKattouw: doesn't that just lead to the developer "fixing" the bug by breaking caching?
21:24:30 <sumanah> this sounds like a lint/unit test thing
21:24:37 <StevenW> I'm not really sure that the GettingStarted issue is really a good generalized performance example
21:24:44 <bd808> RoanKattouw: Agreed. Scholarships needed soe work when first put behind the misc-varnish cluster to actually function.
21:24:52 <bd808> s/soe/some/
21:24:53 <sumanah> StevenW: ah - Matt suggested it to me as one of the examples for that particular section
21:24:58 <RoanKattouw> robla: Depends on the caching layer. If the caching layer is internal to MW, that is likely; if it's Varnish, it's unlikely
21:25:18 <RoanKattouw> We simply don't give you a lot of power to break Varnish
21:25:39 <StevenW> It didn't cause a performance problem just because of adding a new cookie. It caused it because if you just happen to use a new cookie name that matches "Token" our regex-based system for cookie matching causes Varnish to go haywire
21:25:42 <RoanKattouw> sumanah: Re the cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue
21:25:54 <RoanKattouw> And yes what Steven said
21:26:01 * ^d noms on some cookies
21:26:15 <RoanKattouw> It was an unfortunate combination of GS naming their cookie badly and the Varnish config being extremely naïve
21:26:16 <robla> fair enough....rather flukey example then
21:26:42 <sumanah> #link https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers - I have 3 examples right now listed for the caching layers stuff. I think in context as 1 of those 3 it's probably fine?
21:27:18 <csteipp> Slightly OT, but on the topic of breaking the cache-- can we mention that mediawiki has pretty sane default in OutputPage, and if disable it and add your own caching headers, please don't cache if the user is logged in. That broke lots of things last fall.
21:27:31 <sumanah> #info <csteipp> Slightly OT, but on the topic of breaking the cache-- can we mention that mediawiki has pretty sane default in OutputPage, and if disable it and add your own caching headers, please don't cache if the user is logged in. That broke lots of things last fall.
21:27:54 <sumanah> superm401:  http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140514.txt This meeting: Starts 2100.
21:27:54 <ebernhardson> on the other hand, when i hear about us having any kind of big enough perforamnce problem that its discussed on the lists, its always some outlandish hard to predict thing like a regex inside varnish matching a new cookie
21:28:09 <parent5446> > doing your own caching headers
21:28:18 <parent5446> ^probably something that shouldn't be done
21:28:19 <sumanah> would it actually make sense to write a Jenkins test looking for things that are gonna break that regex?
21:28:40 <sumanah> RoanKattouw: ok, I will chase those down somehow, thanks
21:28:42 <ebernhardson> sumanah: no, because its not the specific regex, its more the class of problem
21:29:07 <sumanah> #action sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example
21:29:13 <^d> You could add a jenkins test for any time someone calls WebResponse::setcookie() in a patch. That'd at least call attention to it.
21:29:18 <superm401> sumanah, that would be a hard test to write, since cookies are usually set conditionally.
21:29:21 <bawolff> [sorry just got here]: memcached and object cache are listed both as layers, but aren't they the same thing?
21:29:22 <^d> But ebernhardson is right, it's hard to check for programmatically.
21:29:32 <superm401> ^d's idea is doable, though.
21:29:34 <RoanKattouw> It's theoretically impossible, even
21:29:36 <sumanah> bawolff: happy to fix that, or to ask you to fix it :)
21:29:45 <AndyRussG> sumanah: I'd be happy for to see detailed info, or links to detailed info about about performance, cookies and caching. Haven't looked carefully at that section yet (I came here out of interest on query tuning) but there is some new-cookie-setting happening in the Campaigns extension, which I'm working on.
21:29:54 <superm401> Kind of like how jenkins-job-builder warns you if the XML changes.  Not an indication of a problem, just something to look at.
21:29:59 <^d> RoanKattouw: $foo = 'bar'; $setcookie = 'setcookie'; $setcookie( $foo );
21:30:03 <^d> Hard to check for :)
21:30:08 <sumanah> bawolff: I'd like to clarify that Caching Layers section (the description) and break it out onto wikitech.wikimedia.org at some point, and welcome help on it
21:30:16 <bawolff> ok
21:30:23 <RoanKattouw> ^d: Exactly. Also, halting problem etc. Strictly speaking checking for this is literally impossible without running the code
21:30:23 * bawolff is still reading through the document
21:30:33 <csteipp> Yes to ebernhardson-- it's the idea that special cookie imply you are logged in (ones that contain the strings "Session", "Token", or "forceHTTPS"). If varnish thinks you're logged in it won't use the cache.
21:30:45 <parent5446> $cookie = 'setcookie'; $varname = 'cookie'; $$varname( $cookie );
21:30:52 <sumanah> AndyRussG: https://www.mediawiki.org/wiki/Performance_guidelines and https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code may come in handy for you. I don't think we have anything more comprehensive than those drafts
21:31:25 <AndyRussG> sumanah: OK, thanks!
21:31:26 <^d> RoanKattouw: Indeed. But see what I said above. You could naively check for introduced calls to WebResponse::setcookie() and call attention to them at least.
21:31:43 <RoanKattouw> Yup
21:31:45 <sumanah> AndyRussG: if you want more specifics on specific topics, please feel free to ask on the talkpage or create redlinks
21:32:15 <sumanah> Would someone like to be responsible for following up on this automated test idea? I can #action it and we can move on with talking about the guidelines
21:32:17 <AndyRussG> sumanah: OK, will do, thanks
21:33:43 <sumanah> ok, no one has volunteered, so I'll ask for #help
21:34:09 <sumanah> #help would be cool for someone to write an automated test to " naively check for introduced calls to WebResponse::setcookie() and call attention to them at least." to notify us of potential upcoming cookie issues in new code
21:34:27 <sumanah> OK, now 2 quick-ish questions for y'all
21:34:27 <sumanah> #link https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths This looks good to me and I want to put it into the "how to think about performance" section (the second one, not the intro bullet points)
21:34:59 <sumanah> #info "memcached and object cache are listed both as layers, but aren't they the same thing?" - bawolff
21:35:22 <sumanah> #action sumanah to clarify object cache vs memcached in Caching Layers section
21:35:25 <RoanKattouw> bawolff is right, memcached is a possible implementation of the object cache
21:35:42 <RoanKattouw> Other implementations being APC, database, etc.
21:35:56 <RoanKattouw> (or nothing)
21:36:36 <sumanah> Yes. But this document is talking specifically about WM.
21:36:41 <sumanah> And we use memcached.
21:36:51 <sumanah> And we're not planning on changing that.
21:36:57 <RoanKattouw> OK
21:37:01 <sumanah> I'll clarify it. :)
21:37:06 <RoanKattouw> Yes in that case the object cache and memcached are the same
21:37:22 <RoanKattouw> This conflation is pretty rampant in MW core as well, due to WMF-centricness
21:37:40 <sumanah> nod.
21:37:48 <sumanah> Any thoughts on the "critical paths" comment?
21:37:58 <sumanah> It looks good to me - anyone else?
21:38:24 <RoanKattouw> Looks fine
21:38:45 <sumanah> cool, moving on
21:38:45 <sumanah> #agreed https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths looks good to add to "how to think about perf"
21:38:47 <AndyRussG> Important issue--we had to think about it in a bit of a different context
21:38:53 <AndyRussG> ^ re: critical paths
21:39:04 <jeremyb> sumanah: https://wikitech.wikimedia.org/wiki/Incident_documentation/20120607-LastModifiedExtension
21:39:15 <AndyRussG> when adding some hooks for notifications for the education program extension
21:40:10 <superm401> sumanah, however, MediaWiki-Vagrant uses Redis
21:40:14 <sumanah> I would love to ask people whether we should have something in the Performance Guidelines about logging - about what you should log, or about logging sampling, etc.  I know I need to improve the "how to profile" re logging and add stuff from https://www.mediawiki.org/wiki/User:BDavis_%28WMF%29/Projects/Structured_logging#Current_logging but does any logging info/advice belong in the dev guidelines?
21:40:42 <parent5446> Logging does not really fit in with performance guidelines, but remembering to profile definitely does
21:41:00 <parent5446> Logging is more of just general coding practices
21:41:11 <sumanah> parent5446: maybe you don't understand me :)
21:41:14 <superm401> Logging could help with tracking down performance in some cases, but like parent5446 said (used correctly) it could help with a lot of things.
21:41:24 <parent5446> Oh
21:41:28 <sumanah> parent5446: I'm asking whether there are ways that a choice to log or not log data could slow down the site
21:41:47 <superm401> Oh, I didn't get that either.
21:41:54 <sumanah> Sorry
21:42:04 <bd808> I can't think of a way that not logging would slow things down.
21:42:11 <superm401> I think the logging impact is pretty small, since it uses UDP.
21:42:17 <superm401> Rather than writing directly to a disk.
21:42:31 <^d> Well, logging every single page view 5 times probably isn't a great idea for performance.
21:42:41 <superm401> Yeah, if you go crazy, it could cause problems.
21:42:42 <jeremyb> sumanah: re "not planning on changing that" we already moved off memcache for some things. e.g. user sessions -> redis.
21:42:48 <^d> But generally, people should error on the side of logging more and not worrying about performance of the logging system itself.
21:42:54 <^d> *err
21:43:23 <sumanah> #link https://wikitech.wikimedia.org/wiki/Incident_documentation/20120607-LastModifiedExtension "LastModified extension was causing every page view to an attempted POST request to http://en.wikipedia.org/w/api.php. "
21:44:00 <bawolff> Good guideline - don't do anything on every page view unless you are very sure its ok
21:44:09 <superm401> Yeah, I think there have been a few outages/slowdowns due to API usage.
21:44:23 <sumanah> Hold on - jeremyb superm401 - are you saying that we are, in some cases, choosing to implement the object cache in Redis?
21:44:33 <bawolff> yes we do that
21:44:42 <sumanah> Because my "we're using memcache and not changing that" was in response to <RoanKattouw> bawolff is right, memcached is a possible implementation of the object cache
21:44:55 <bawolff> or can do that. People on tool labs use redis for object cache all the time
21:45:01 <bawolff> and we use it for sessions
21:45:05 <bawolff> on wmf
21:45:09 <superm401> sumanah, yes, but only for MediaWiki-Vagrant, Tool Labs, etc.
21:45:17 <superm401> sumanah, memcached is the object cache in production.
21:45:21 <jeremyb> sumanah: right, i know what it was in response to. I don't know exactly what "object cache" means. but memcache maybe isn't forever
21:45:35 <superm401> But Redis is used for other things (job queue, sessions, GettingStarted) in WMF production.
21:45:37 <sumanah> (btw welcome Krenair to the Wikimedia Foundation since I just saw the welcome email :-) thanks for working with us!)
21:45:47 <Krenair> :)
21:45:51 <parent5446> Object cache is an interface, a layer; memcached and redis are implementations
21:45:59 <jeremyb> we interrupt this programming to welcome Krenair :)
21:46:07 <bd808> I think in this context "object cache" means "ephemeral storage for serialized php objects"
21:46:23 <superm401> Welcome, Krenair.
21:46:27 <ebernhardson> bd808: well, doesnt it mean the includes/objectcache/ directory?
21:46:28 <bawolff> Welcome and congratulations!
21:46:34 <ebernhardson> bd808: because anything else would just be odd :)
21:46:47 <jeremyb> sumanah: re logging or not logging having a perf impact see the link i hilighted you with before
21:46:52 <sumanah> nod.
21:47:00 <bawolff> bd808: Any time you need to store a Bag O' stuff, we use the bagOStuff class (object cache) :)
21:47:04 <sumanah> I would absolutely love for experts to give me a glossary to supersede https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers , or to edit that section to clarify it. Anyone volunteering?
21:47:12 <^d> ebernhardson: Pfft, you newbies and your subdirectories of ./includes/. IN MY DAY we put everyting in ./includes/ and WE LIKED IT
21:47:19 <ebernhardson> :P
21:47:48 * bd808 is still baffled by the lack or real or fake namespacing
21:48:03 <ebernhardson> bd808: php5.3 has a *huge* perforamnce hit for namespaced code
21:48:15 <bd808> MW_Cache_BagOStuff
21:48:16 <ebernhardson> bd808: its really not a great idea until 5.4 or hhvm(e.g. 3 years ago for everyone else :P)
21:48:26 <^d> bd808: Well, that's why MWNamespace exists.
21:48:29 <^d> Used to be Namespace.
21:48:32 <^d> But then 5.3 happened.
21:48:42 <sumanah> #info in production, we implement the object cache (ephemeral storage for serialized PHP objects) in memcached. But in MediaWiki-Vagrant installs, on Tool Labs, and in some other contexts, we use Redis for that instead. The object cache is an interface or layer that we can implement in various ways.
21:48:47 <bawolff> Use long complicated class names. Its fun
21:48:51 <parent5446> Hence PEAR pseudo-namespaces using underscores
21:49:07 <sumanah> #link https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency I think this looks good and am trying to figure out where to put it
21:49:11 <parent5446> RevDel_* (although that was just removed)
21:49:11 <superm401> sumanah, tweaked it a little re object cached.
21:49:16 <jeremyb> sumanah: we also cache opcodes in APC. which means that we can reduce the opcode cache hit rate if too many copies of mediawiki are in use on a single box at once. that's why we limit it to 2 slots in prod at a time. (excluding testwiki. and maybe testwikidata? but test2wiki is always on one of the 2 slots)
21:49:16 <sumanah> Thank you superm401!
21:49:20 <^d> parent5446: Yeah, because PEAR is a beacon of good design practices ;-)
21:49:59 <^d> jeremyb: Just test, not testwikidata. But otherwise yes, you're right. We exhaust the APC cache otherwise.
21:50:25 <sumanah> I think latency stuff might not really be anywhere right now except in the discussion of edge caching and of general user expectation.
21:50:47 <ebernhardson> getting off on a tangent, but PEAR is also pretty much dead.  not even phpunit plans to continue releasing there
21:50:53 <sumanah> jeremyb: The thing you just said: does it imply that developers should do or not do anything in particular? :)
21:50:59 <^d> ebernhardson: PEAR2 or bust!
21:51:08 <sumanah> http://fabien.potencier.org/article/72/the-rise-of-composer-and-the-fall-of-pear - I need to read this thoroughly
21:51:25 <^d> sumanah: No, most developers shouldn't be bothering with that. It's mostly a WMF deployment/operational concern.
21:51:36 <^d> Developers shouldn't be worrying about "am I going to exhaust the bytecode cache"
21:51:37 <sumanah> I thought so.
21:53:22 <jeremyb> sumanah: it means that if we can't have enwiki on one mediawiki version, commons on another and dewiki on a 3rd. all 800+ wikis can only be one of 2 distinct versions. one impact for developers is that they should make liberal use of feature flags so that individual wikis can vary on those in InitialiseSettings.php even though the mediawiki version is the same
21:53:35 <jeremyb> sumanah: and that applies to extensions too not just core
21:54:07 <sumanah> I feel like Brion's point on latency might be important enough to merit another top-level arch principle - "users should experience consistent speed of service despite variable network latency" or something like that
21:54:08 <^d> Feature flags should exist for about half a dozen reasons, that being one of the least imho.
21:54:53 <jeremyb> ^d: sure, plenty of reasons :)
21:55:02 <sumanah> Yeah, it sounds like this does not need to go into the performance guidelines, in my opinion. General HOWTO, yes.
21:55:17 * sumanah hopes people will illuminate her thinking on the latency topic
21:55:32 <jeremyb> s/means that if we/means that we/
21:55:33 <sumanah> BTW since we have 5 min left, any questions about security guidelines or architecture guidelines?
21:56:36 <sumanah> RoanKattouw: Krinkle robla - would love your thoughts on my ponderings in the last ~20 lines of backscroll re https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency  :)
21:58:05 <superm401> sumanah, not sure what Brion means by "consistent speed of service despite variable network latency".
21:58:09 <superm401> Is that in a mailing list somewhere?
21:58:20 <superm401> Seems hard to make very slow connection seem equally good as a fast connection.
21:58:40 <sumanah> superm401: well, I'm the one coming up with that phrasing, which is why it might not make sense. :)
21:59:16 <superm401> There are things you can do to make loading delay seem less annoying.
21:59:21 * sumanah has generally been doing the work of initially summarizing and synthesising developers' thoughts into one- or two-sentence principles
21:59:23 <bawolff> I think acceptable speed of service regardless of network latency should be the goal
21:59:32 <superm401> E.g. load the center (main text) first, and the rest after.
21:59:36 * sumanah nods
21:59:39 <superm401> Though done wrong, this results in annoying flashes and reflows.
21:59:42 <sumanah> yeah
21:59:57 <bawolff> cough...fundraising banners
22:00:11 <sumanah> bawolff: you saw https://www.mediawiki.org/wiki/Performance_guidelines#Preserving_positioning ?
22:00:14 <robla> I think a lot of the latency stuff belongs in general performance principles
22:00:51 <bawolff> ah, its already been said
22:00:52 <robla> (just a guess, though...would need a more careful reading to figure it out)
22:01:37 <sumanah> #action sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section
22:02:02 <jeremyb> sumanah: edited parser cache
22:02:15 <sumanah> Well, it's been an hour so I'll say g'night. Thanks for your feedback all. I hope the end document helps a lot of people understand how to write efficient code, and helps code reviewers have something to point at when giving thumbs-up and thumbs-down.
22:02:21 <sumanah> #endmeeting
Retrieved from "https://www.mediawiki.org/w/index.php?title=Architecture_meetings/Performance_guidelines_discussion_2014-05-14&oldid=1038104"