Talk:Requests for comment/Reducing image quality for mobile
Add topicImpact on infrastructure
[edit]Have in mind that depending on how you'll implement this, you may significantly increase imagescaler/Swift/upload CDN requests and storage requirements for Swift. I'm not saying to not do it, but I'd like to see some risk assessment & load estimates in the RFC itself. I'd also very much prefer for this to be blocked on the implementation of the simplify thumbnail cache RFC, as it may alleviate half of my concerns (but not all). Faidon Liambotis (WMF) (talk) 01:23, 19 March 2014 (UTC)
- Added some basic guestimations and reasoning. Please provide some basic swift stats (like how many images have actually been scaled down, how many different versions of a single image do we currently have on average, and what is their size vs original) to have better statistics. Having varnish-based thumbnail cache would obviously be a great improvement, but I think we should treat them as two independent projects, unless we know for sure that estimated image-size increase cannot be handled by our current infrastructure. --Yurik (talk) 04:44, 19 March 2014 (UTC)
- Your guesstimations are very wrong, your numbers are off by about 300-600x. We currently have 24.148.337 originals taking up 40.9TB. These, have 302.195.630 corresponding thumbnails, taking up 19.8TB and growing. They are consuming space in the Swift backends with replica count 3 (i.e. 60TB), as well as on the SSD disks of backend caches (eqiad, esams, ulsfo), in the pagecache of Swift backends & Varnish backends, and in memory of Varnish frontends (yes, it's wasteful, that's why we have been discussing it over at the simplification RFC). Finally, note that a lot of the scalability problems arise from the count of files that we keep, not from their aggregate size, a dimension that you haven't considered at all -- just imagine how different it is to handle e.g. 10 files of 2G each vs. 10 million files of 2K each. If this proposal is to double the amount of thumbnails that we keep, I'm afraid that it's going to need serious ops & platform work with many months of work needed to make significant improvements to the architecture and it would definitely need to be blocked on the simplified cache RFC. —The preceding unsigned comment was added by Faidon Liambotis (WMF) (talk • contribs) .
- Thanks for the numbers, let me try to go through them. 25 million images turn into 300 million thumbnails. That's 12 per 1. There are automatic gallery size (shown on category pages in commons), and all image pages have these options on the file page: 800×600, 320×240, 640×480, 1024×768, 1280×960, 2048×1536 (assuming the image is larger than those options). Any dumb crawler that simply follows all URLs would trigger image scaling. Also, when we generate HTML, the srcset attribute automatically adds 1.5x and 2x options, tripling thumbnail count (does not happen for the gallery or file page, only for articles.
- So if we assume that 12 consists of ~5 preset options on the file/category pages plus 1 original, what remains are 2 article usages (x3 because of srcset). Mobile javascript would only replace those 2 usages (removing srcset attribute and ignoring pages in file namespace), so we end up with 50 million extra pictures 5-10KB each - 250-500GB (twice the size of my original calc). In other words, we are looking at about half a terabyte growth in disk space and ~15% growth in the number of files. Hope my calculations make sense and I did not miss anything major. --Yurik (talk) 16:00, 21 March 2014 (UTC)
- Your guesstimations are very wrong, your numbers are off by about 300-600x. We currently have 24.148.337 originals taking up 40.9TB. These, have 302.195.630 corresponding thumbnails, taking up 19.8TB and growing. They are consuming space in the Swift backends with replica count 3 (i.e. 60TB), as well as on the SSD disks of backend caches (eqiad, esams, ulsfo), in the pagecache of Swift backends & Varnish backends, and in memory of Varnish frontends (yes, it's wasteful, that's why we have been discussing it over at the simplification RFC). Finally, note that a lot of the scalability problems arise from the count of files that we keep, not from their aggregate size, a dimension that you haven't considered at all -- just imagine how different it is to handle e.g. 10 files of 2G each vs. 10 million files of 2K each. If this proposal is to double the amount of thumbnails that we keep, I'm afraid that it's going to need serious ops & platform work with many months of work needed to make significant improvements to the architecture and it would definitely need to be blocked on the simplified cache RFC. —The preceding unsigned comment was added by Faidon Liambotis (WMF) (talk • contribs) .
Authors shouldn't have to worry about compression settings
[edit]The idea to have authors specify a quality setting, unless I'm fundamentally misunderstanding the intent, seems very misguided. We should write our software to take performance and efficiency considerations into account, not shift that responsibility to authors. If indeed images should be delivered at a higher compression factor / lower quality in certain use cases, let's identify those use cases, and go as far as we can without introducing markup to specify compression settings before even thinking about doing so.--Eloquence (talk) 02:26, 19 March 2014 (UTC)
- This feature is not for authors, but for internal/advanced use - when we are serving image to a mobile device on mobile network, the goal is to automatically reduce image quality to reduce wait time. Additionally, this might also benefit users who pay for their data plan per MB, as it would allow us to create low/high quality mobile settings. --Yurik (talk) 04:44, 19 March 2014 (UTC)
- @Yurik: If it's not for authors, I fail to see why it should be added to the markup (and your proposal contradicts that statement: "[t]his parameter might be used by various template authors").--Eloquence (talk) 06:35, 19 March 2014 (UTC)
- Sorry, wasn't clear. This is MOSTLY for internal use, but there might be advanced authors out there who might decide to reduce image quality in addition to reducing pixel size for some obscure template. After all, if we provide rotation and scaling, why limit the toolset? And since the generated image URL is exposed to the world, why not do a well documented parameter as well? In any case, I am ok to not include it if the community is against it/there is a technical reason not to. --Yurik (talk) 07:16, 19 March 2014 (UTC)
- Image compression quality is a technical concern rather than an editorial one. It doesn't belong in the markup.--Eloquence (talk) 03:00, 20 March 2014 (UTC)
- Yeah, I'm struggling to understand why we would want to reduce image quality (very strange to say aloud...) manually rather than programmatically. I agree with your posts here. --MZMcBride (talk) 03:20, 20 March 2014 (UTC)
- Image compression quality is a technical concern rather than an editorial one. It doesn't belong in the markup.--Eloquence (talk) 03:00, 20 March 2014 (UTC)
Ok, for now the core patch 119661 does not let users specify quality reduction via image link param, but only via modifying the URL itself. --Yurik (talk) 17:41, 24 March 2014 (UTC)
File insertion syntax
[edit]File insertion syntax is already an abomination. It really shouldn't be extended any further. --MZMcBride (talk) 02:04, 20 March 2014 (UTC)
- The image with a different quality must have a different URL -- the image is not scaled by the initial request during HTML rendering, but rather it is processed on 404 by extracting needed parameters from the URL. The URL syntax is what seemed the most straightforward to me. How do you suggest we pass that information to the image processor? --Yurik (talk) 17:13, 20 March 2014 (UTC)
+1 on MZ (and Erik in #Authors shouldn't have to worry about compression settings). --Nemo 14:45, 26 March 2014 (UTC)
- @Nemo bis: , are you proposing an alternative to changing URL for the image? --Yurik (talk) 17:13, 26 March 2014 (UTC)
- I have no opinion about that. MZ, Erik and I all commented on the image markup, i.e. what's written in wikitext. --Nemo 18:06, 26 March 2014 (UTC)
- Ok, so is @MZMcBride: ok with the URL change, or is there another option? This is NOT related to the above discussion about image link parameter, which has been removed. --Yurik (talk) 22:25, 27 March 2014 (UTC)
Bikeshed
[edit]"Reducing image quality" feels like a strange name to me. Perhaps the name of this RFC could be "Compressing images for mobile" or something like that? Just a suggestion. :-) --MZMcBride (talk) 03:22, 20 March 2014 (UTC)
- Renamed. --Yurik (talk) 19:51, 20 March 2014 (UTC)
Fundamentals
[edit]I'm unable to comment on this proposal, because it doesn't provide any background on how you reached your proposal. All we're provided as use case and background is one line: "Many mobile devices with low bandwidth or small screen/slow processor could benefit from showing JPEG images with reduced quality". All the main points seem to be given for granted: they shouldn't.
- Why JPEG? We have lots of PNG thumbnails, are we sure those are not taking more bandwidth?
- Why for mobile? If we can have quality degradations without noticeable problems, why not tweak the thumbnailing settings for everyone in core? Kiwix also compresses images a lot, for instance, and the quality is still rather good: worth exploring. (About 30 GB in addition to the 10 GB of text in the first and last full ZIM release.)
- Why quality? Isn't it easier to change default thumbnail size on the mobile site, from 220px to something else we already have thumbs for, like 120px? --Nemo 14:45, 26 March 2014 (UTC)
- Thanks @Nemo bis: , I expanded the Rational section, hopefully that answers your question. --Yurik (talk) 17:12, 26 March 2014 (UTC)
- Thank you. I summarise that with "because we can [easily]". That's not particularly convincing, I must say, even though it may still be a good idea. On the alleged lack of alternatives:
- changing default thumb size is something we've done before without noise;
- as for PNG we don't even know how much traffic they produce, maybe it's substantial and we can make vipscaler compress them lossily or otherwise tweak them;
- for JPEG we have bugzilla:51451 and [1] claims a couple options can have a big impact. --Nemo 18:06, 26 March 2014 (UTC)
- @Nemo bis: , I am not against vips, but it seams the biggest benefit of it is execution efficiency - it runs much faster and consumes less memory. These are great qualities, but they are orthogonal to this RFC - I will be very happy if our scaler switches to a more efficient one, but getting it should be a separate issue. I ran stats against one day:
- PNG: 2,199,455 / 18,749 MB / 8.7 KB/file
- JPEG: 1,708,366 / 28,548 MB / 17 KB/file
- So even though JPEGs are only 43% by count, they are 60% of total traffic, and more than twice the size. Targeting JPEGs seems to give more bang for the buck, without introducing a new backend scaler.
- Method: ran
zgrep '/commons/thumb/.*image/png' sampled-1000.tsv.log-20140325.gz
, counted with -c, summed withcut -d$'\t' -f7 | awk '{s+=$1} END {printf "%.0f\n", s/1024/1024}'
- --Yurik (talk) 22:21, 27 March 2014 (UTC)
- Thank you. I summarise that with "because we can [easily]". That's not particularly convincing, I must say, even though it may still be a good idea. On the alleged lack of alternatives:
16 April discussion
[edit]From yesterday's RfC chat:
- 21:03:31 <sumanah> #topic Reducing image quality for mobile
- 21:03:42 <TimStarling> the patch seems quite different to what yurik and I discussed at the architecture summit
- 21:03:50 <sumanah> ( but brion TimStarling - I may ask some follow-up questions at the end about a few other RfCs and pending things)
- 21:03:54 <sumanah> #link https://www.mediawiki.org/wiki/Requests_for_comment/Reducing_image_quality_for_mobile
- 21:04:15 <sumanah> #info I asked Yuri what he wanted: 1) an ok from ops to increase thumbnail storage by 2-3% and number of files by 15%, 2) from core/tim/etc to proceed with the proposed patch <yurik> assuming my proposed path is satisfactory to everyone's involved
- 21:04:19 <TimStarling> I thought that you should have only quality classes exposed, not expose an API allowing any integer percentage quality
- 21:04:59 <yurik> TimStarling, it would be fairly easy to change from a number to a string constant
- 21:05:11 <TimStarling> you suggest 30% but probably every mobile app will choose something different
- 21:05:12 <yurik> if this is a requirement of course
- 21:06:37 <yurik> TimStarling, this is similar to the problem we face with the thumbnail dimension - every wiki varying images by a few pixels. I propose a somewhat different solution here - an extension that does filtering/rounding of these numbers during the rendering
- 21:07:04 <sumanah> thedj: dfoy_ - http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140416.txt for the logs up till now
- 21:07:21 <TimStarling> I don't see any filtering or rounding in the patch
- 21:07:41 <yurik> example: user requested 240x250 image - the ext would say 250x250 already exists, or it is a multiple of 50, hence render it as a link to 250x250, with width=240
- 21:08:00 * aude waves
- 21:08:11 <aude> yurik: is this something your extension would do? rather than core?
- 21:08:12 <sumanah> Hi :)
- 21:08:12 <yurik> separate patch - as an extension - to address all such rounding requirements for both image size & quality
- 21:08:16 <TimStarling> yeah, you can read my thoughts on that on the relevant RFC
- 21:08:47 <yurik> aude, not ours, a new extension whose job is only to "standardize" on thumbnail generation
- 21:08:54 <AaronSchulz> gah
- 21:08:59 <aude> but not core?
- 21:09:11 <sumanah> AaronSchulz: I presume you think that's the wrong approach :)
- 21:09:28 * brion just added comment on the patch agreeing with idea to use quality classes rather than expsoing full integer range
- 21:09:33 <sumanah> #link https://gerrit.wikimedia.org/r/#/c/119661/ Gerrit changeset, "Allow mobile to reduce image quality"
- 21:09:34 <yurik> no, i think core should be more flexible - depending on the site
- 21:09:46 * aude prefers we allow any size, but not keep cached so long if it's not requested
- 21:09:58 <aude> if that's feasible
- 21:10:09 <TimStarling> me too
- 21:11:37 <thedj> can i ask what the primary purpose is ?
- 21:11:46 <thedj> reduce time to load ?
- 21:12:01 <sumanah> thedj: honest question: does the RfC address that? do you think the RfC should be clearer about the problem being solved?
- 21:12:03 <yurik> reducing quality? to lower bandwidth consumption
- 21:12:36 <thedj> yurik: so download time and download cost ?
- 21:13:30 <yurik> both
- 21:13:36 <thedj> Do we have some metrics/ideas to give us indications of how much benefit that would translate into ?
- 21:13:43 <yurik> especially when the bandwidth is donated
- 21:14:22 <yurik> thedj, 30-40%
- 21:14:55 <thedj> ah k. so it's to a large degree from the zero perspective that we want to do this.
- 21:15:01 <yurik> correct
- 21:15:36 <brion> i could see it being handy for hi-dpi devices as well, we could serve the double-size images with a medium quality setting to trade-off brandwidth and visual quality
- 21:15:38 <sumanah> BTW, for those who haven't looked, we now have a few more comments on the changeset https://gerrit.wikimedia.org/r/119661 in the last few minutes
- 21:15:50 <brion> but definitely the incentive is where we’re pushing donated bandwidth :)
- 21:15:55 <sumanah> (there's our brion always looking out for responsive design & gadget stuff :) )
- 21:16:19 <bawolff> My comment was just that it shouldn't touch the -quality setting on pngs, and a nitpick on the commit message
- 21:17:11 <gwicke> once we move to HTMl storage, is the idea to implement this as a DOM post-processing step?
- 21:17:39 <yurik> TimStarling, brion, please take a look at the https://www.mediawiki.org/wiki/Requests_for_comment/Reducing_image_quality_for_mobile#Possible_approaches
- 21:18:15 <yurik> it discusses the 3 paths to do this, with 1 path doing everything internally without exposing it via URL
- 21:18:43 <brion> *nod* i was assuming the first pass implementation once the qualitys etting was available...
- 21:18:52 <TimStarling> probably option 2
- 21:18:54 <brion> … was to do it as a dom postprocess step in mf+zero
- 21:19:09 <TimStarling> that's not on the list
- 21:19:24 <yurik> that's #3 i think
- 21:19:26 <brion> agh, i confused that with the js one
- 21:19:59 <yurik> tim, you think it is better to let varnish do automagical image url rewrite?
- 21:20:19 * AaronSchulz prefers js if possible
- 21:20:31 <TimStarling> how would it work with JS?
- 21:20:38 <yurik> because we won't have as much info in varnish, plus we would have to put too much biz-logic in varnish (ops won't like it)
- 21:20:43 <gwicke> one issue I see with Varnish is transparent downstream caches
- 21:20:52 <yurik> yes, that too
- 21:20:55 <TimStarling> a DOM ready event?
- 21:20:55 <gwicke> the third option (JS) avoids that
- 21:21:03 <yurik> JS would rewrite the URL
- 21:21:10 <brion> hmm
- 21:21:24 <brion> my main concern with that is rewriting urls in JS without often loading the original url is tricky
- 21:21:35 <TimStarling> I am wondering what the CPU requirements of option 3 are
- 21:21:37 <AaronSchulz> gwicke: related to downstream caches is handling purges
- 21:21:51 <TimStarling> and whether there will be flicker, browser incompatibilities, etc.
- 21:21:58 <gwicke> AaronSchulz, *nod*
- 21:22:02 <AaronSchulz> I guess if it's the very frontend cache it's fine
- 21:22:09 <TimStarling> we can't really waste the CPU of phones the same way we can desktop browsers
- 21:22:17 <gwicke> we'd have to send s-maxage-0
- 21:22:19 <yurik> workflow: zero ext changes src= to low quality, JS changes it back to highres if device/network is good
- 21:22:21 <gwicke> =0
- 21:22:34 <brion> :\
- 21:22:45 <yurik> how expensive is a JS image tag search?
- 21:22:55 <gwicke> it's pretty cheap I believe
- 21:23:07 <brion> replacing them may be slow if it’s a big page with lots of images though
- 21:23:19 <gwicke> one querySelectorAll call
- 21:23:24 <brion> and you’ve got the issue of loading the original images and then the new ones....
- 21:23:25 <TimStarling> image loading will start as soon as the img tag is created, right?
- 21:23:27 <yurik> percentage wise i still think it won't be much
- 21:23:35 <AaronSchulz> TimStarling: I think so :/
- 21:23:45 <gwicke> yeah, I think that's the bigger issue
- 21:23:57 <gwicke> we have a similar issue with the thumb size pref
- 21:23:58 <yurik> that's the big question - can the low->high quality img tag replacement be done before browser starts loadnig them?
- 21:24:07 <TimStarling> what about what brion said, why is that not an option?
- 21:24:19 <TimStarling> <brion> … was to do it as a dom postprocess step in mf+zero
- 21:24:22 <gwicke> if we can find a way to suppress the original thumb load before resizing / quality downgrading, then that would be awesome
- 21:24:46 <yurik> TimStarling, we would have to do it anyway, but there will be users who would want high-end images
- 21:25:03 <brion> i think we’re trying to avoid having php-time cacheable differences on zero….. it’s all very scary
- 21:25:35 <brion> in general, trying to scale for estimated network bandwidth is just a tricky tricky business
- 21:26:45 <sumanah> tfinc: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140416.txt for chat so far
- 21:27:18 <yurik> there is another question - i am pretty sure there are many mobile users out there who don't have zero and who might want low bandwidth too
- 21:27:24 <TimStarling> what about having a separate new service to do DOM rewriting?
- 21:27:49 <yurik> so we really should have a mobile setting "auto/always high/always low"
- 21:28:01 <TimStarling> yurik: those users can put up with what we give them
- 21:28:05 <gwicke> TimStarling, that's doable for low volume
- 21:28:26 <gwicke> which zero is afaik
- 21:28:57 <gwicke> what are the peak request rates on zero in pages / s ?
- 21:29:00 <yurik> well, not those who are still on 2G, or who is paying high price for their internet.
- 21:29:24 <TimStarling> it's out of scope
- 21:29:31 <aude> yurik: then i'd want no images, if concerned about bandwidth (imho)
- 21:29:40 <aude> maybe my mobile browser allows that
- 21:29:45 <sumanah> Those who have questions for Max, he's here now
- 21:29:54 <TimStarling> the problem is complicated enough when it is just Zero
- 21:30:09 <dr0ptp4kt> fwiw, MobileFrontend already has an Images on/off toggle
- 21:30:13 <sumanah> (OK, maybe today's meeting WON'T be a short one after all.)
- 21:31:17 <TimStarling> dr0ptp4kt: does it work?
- 21:31:26 <TimStarling> or do the images start loading and then get aborted?
- 21:31:34 <dr0ptp4kt> TimStarling: it is completely rewritten html
- 21:31:40 <dr0ptp4kt> it works
- 21:31:40 <MaxSem> it works via DOM rewriting on PHP side
- 21:31:53 <gwicke> there are ways to parse html without loading images, using https://developer.mozilla.org/en-US/docs/Web/API/DOMParser for example
- 21:32:27 <gwicke> or XMLHttpRequest
- 21:32:27 <dr0ptp4kt> one caveat is supporting devices that don't support javascript, or rather "advanced javascript" as determined by rl
- 21:32:35 <TimStarling> gwicke: well, that's the kind of thing that I would expect to use a lot of client-side CPU
- 21:32:48 <gwicke> not really- it's using the normal html parser
- 21:32:59 <gwicke> it does rely on JS support though
- 21:33:05 <gwicke> and a non-sucky browser
- 21:33:13 <dr0ptp4kt> HA!
- 21:33:20 <gwicke> ;)
- 21:33:26 <MaxSem> I don't think that many devices we want to support will work well with this
- 21:33:53 <gwicke> are you using XMLHttpRequest currently?
- 21:34:06 <MaxSem> libxml2
- 21:34:14 <MaxSem> be its name foreveer cursed
- 21:34:22 <TimStarling> can someone give me a quick overview of how HTML delivery in MF works and what the plans for it are?
- 21:34:42 <dr0ptp4kt> we use xhr opportunistically. so it's usually to upgrade the experience, like avoid server roundtrips for newer phones
- 21:34:58 <dr0ptp4kt> er, bigger roundtrips
- 21:35:15 <MaxSem> шеэы ыешдд мукн кщгпр щт увпуы
- 21:35:18 <gwicke> I see, so you are hesitant to require it
- 21:35:25 <TimStarling> preferably in a latin script
- 21:35:28 <yurik> MaxSem, +2
- 21:35:34 <MaxSem> it's still quite buggy so is used only in alpha
- 21:35:54 <dr0ptp4kt> sumana, would you please wire up a translation bot now? :)
- 21:36:02 <MaxSem> plans are to fix it
- 21:36:06 <MaxSem> ...eventually
- 21:36:11 <MaxSem> ...maybe
- 21:36:35 <gwicke> I don't see an issue with DOM post-processing on the server and storing that HTML back
- 21:36:36 <dr0ptp4kt> yeah, the xhr for w0 is more like getting runtime config to do things ahead of caches being purged (e.g., add zero-rated support for an additional language)
- 21:36:58 <sumanah> dr0ptp4kt: I think here it would just emit those cartoon profanity things, like $%#%@
- 21:37:17 <gwicke> as long as there are only a few variants and the transforms build on a known DOM spec that should work well
- 21:37:52 <yurik> gwicke, zero already does a DOM post-parse rewrite to replace all external URL links with special warning URLs
- 21:37:59 <MaxSem> I would reeeeeally love to avoid doing it in PHP again
- 21:38:21 <gwicke> it's fairly easy in JS
- 21:38:27 <gwicke> you can use jquery etc
- 21:38:37 <MaxSem> wouldn't be lethal for zero which already does HTML transformations, but still sucks
- 21:38:50 <yurik> gwicke, assuming flip phone has it :(
- 21:38:59 <gwicke> yurik, I mean on the server
- 21:39:38 <yurik> do we have a framework for node.js extensions?
- 21:39:57 <gwicke> yurik, we have HTTP..
- 21:40:08 <gwicke> set up a service, make requests to it
- 21:40:24 <sumanah> So we're about 2/3 through the hour and I'm not sure what to #info :)
- 21:40:46 <yurik> gwicke, you mean PHP becomes a proxy to another service on internal network?
- 21:41:07 <yurik> in any case, this is an optimization for the future, outside of the scope imho
- 21:41:19 <TimStarling> sumanah: three of us wrote comments on the gerrit change
- 21:41:36 <gwicke> yurik, you can go through PHP if you want; depends on whether it adds info that would be hard to get otherwise
- 21:42:04 <AaronSchulz> do we actually need the wikitext syntax addition too?
- 21:42:14 <MaxSem> definitely not
- 21:42:16 * AaronSchulz leans toward not adding it
- 21:42:17 <brion> i think we don’t need the wikitext addition no
- 21:42:28 <brion> keep it opaque to that layer
- 21:42:32 <AaronSchulz> right
- 21:42:33 <TimStarling> #info comments were provided on the image quality gerrit patch
- 21:42:33 <brion> it’s a presentation-layer decision
- 21:42:34 <sumanah> !link https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Reducing_image_quality_for_mobile#File_insertion_syntax
- 21:42:50 <sumanah> er
- 21:42:53 <gwicke> -1 on the extra syntax
- 21:42:56 <sumanah> #link https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Reducing_image_quality_for_mobile#File_insertion_syntax on wikitext addition
- 21:43:03 <sumanah> can I say #agreed ? :)
- 21:43:07 <bawolff> +1 on not adding extra options to file syntax
- 21:43:27 <yurik> bawolff, how do you mean?
- 21:43:27 <TimStarling> #info image scaler backend relatively uncontroversial -- HTML/URL manipulation to access that API is more complex
- 21:43:38 <yurik> we need to distinguish low-quality URLs from the highs
- 21:43:41 <bawolff> yurik: I'm agreeing with everyone
- 21:44:03 <yurik> good position :)
- 21:44:05 <bawolff> yurik: as in not adding [[File:foo.jpg|quality=20]] ui
- 21:44:15 <yurik> gotcha
- 21:44:42 <TimStarling> #info gwicke predictably favours Node.JS service
- 21:44:48 <AaronSchulz> lol
- 21:44:53 <gwicke> hehe
- 21:45:06 <gwicke> that's in response to MaxSem's lament about libxml2 ;)
- 21:45:34 <MaxSem> having a service for that would be even more cruffty
- 21:45:39 <yurik> ok, i will change the URL syntax to image.jpg/100px-qlow-image.jpg this way we can later change it to some other magic keywords
- 21:45:45 <sumanah> So it's sounding like people think this is a relatively uncontroversial idea overall and we're just talking about implementation, right?
- 21:46:08 * tfinc reads the backscroll
- 21:46:20 <yurik> any objections to that URL format?
- 21:46:32 <bawolff> yurik: Maybe re-order those parameters. Easier to regex out qlow-100px from the actual name of the file
- 21:46:37 <sumanah> "this" being the RfC as a whole
- 21:46:53 <MaxSem> +1
- 21:46:54 <bawolff> since we're going to be presumably keeping 100px-image.jpg for the normal quality image
- 21:47:15 <gwicke> are we sure that we need a different URL?
- 21:47:21 <MaxSem> yes
- 21:47:29 <MaxSem> varnish rewrites are evil
- 21:48:16 <gwicke> do we already have info about zero ip ranges in varnish?
- 21:48:27 <yurik> ok, all settled, will implement the first step (core patch), and start implementing JS magic
- 21:48:43 <MaxSem> gwicke, for all that is holy, don't
- 21:48:44 <yurik> gwicke, yes, varnish detects zero based on ip
- 21:48:52 <sumanah> #info <yurik> ok, all settled, will implement the first step (core patch), and start implementing JS magic
- 21:49:15 <MaxSem> especially since now only mobile varnishes know about zero
- 21:49:19 <gwicke> hmm, then it might not actually be that hard to use that for image request rewriting
- 21:50:31 <yurik> #info required modifications: use string instead of integer "qlow-100px-image.jpg", make it JPG only (no png)
- 21:50:31 <gwicke> I'd be against adding that info if it wasn't there already; but since it's already there it seems that the extra complexity would be fairly limited
- 21:50:46 <TimStarling> varnish doesn't have a lot of string handling built in, but you can use inline C, I did it once...
- 21:51:17 <MaxSem> regexping it would actually be possiblee
- 21:51:34 <MaxSem> but still this would SUCK
- 21:52:00 <sumanah> I have a few min of "what's up next week + other RfC news you should be aware of" to say before the end of the hour.
- 21:52:04 <sumanah> Any closing statements?
- 21:52:39 <TimStarling> modules/varnish/templates/vcl/wikimedia.vcl.erb was my own little bit of varnish URL manipulation
- 21:53:30 <yurik> if we are done, would love to get +2 for https://gerrit.wikimedia.org/r/#/c/109853/
- 21:54:34 <TimStarling> #info Tim skeptical about client-side JS rewrite: potential for CPU usage, flicker, image load aborts, browser incompatibilities, etc.
- 21:55:21 <gwicke> avoiding a double-load is hard afaik
- 21:55:41 <TimStarling> which is an argument for doing it on the server side
- 21:55:49 <AaronSchulz> yeah it may not be possible to use JS
- 21:55:50 <gwicke> or in Varnish
- 21:56:04 <AaronSchulz> so it's 1-2
- 21:56:18 <TimStarling> we have so many powerful tools on the server side now, we shouldn't be so keen to offload processing
- 21:57:00 <gwicke> for normal desktop page views the thumb size pref is pretty much the only one that can't be easily handled in CSS
- 21:57:53 <gwicke> so if we can find a way to do this in Varnish it might be possible to implement those prefs purely in CSS
Comment on Javascript scaling
[edit]When working on a mobile site with several others and we tried the JS dynamic size option. The primary issue we encountered was flicker, or browser incompatibility/bugginess on older devices, mostly android 2.2 devices using carrier or manufacturer installed browsers. It didn't handle it well, and occasionally there were several stages of flicker, although we did not determine exactly why. We were just a few people, and after we saw the issues we just decided to use a low quality iamge. Perhaps a better implementation could be created here (I'm sure something better could be, but how much better I'm not sure), but fair warning. NativeForeigner (talk) 23:37, 17 April 2014 (UTC)
IRC discussion 11 June
[edit]From Architecture meetings/RFC review 2014-06-11:
- Reducing image quality for mobile (sumanah, 21:17:01)
- LINK: https://www.mediawiki.org/wiki/Requests_for_comment/Reducing_image_quality_for_mobile (sumanah, 21:17:04)
- the patchset has been merged into core! yay! (sumanah, 21:17:11)
- LINK: http://lists.wikimedia.org/pipermail/mobile-l/2014-May/007176.html Adam Baso said: "The next step we think is to rewrite the <img> tags on a popular page (e.g., http://en.m.wikipedia.org/wiki/Cats) when Wikipedia Zero is in force in order to give this a run in production." (sumanah, 21:17:18)
- LINK: http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007275.html Yuri asked: do we use it via JavaScript rewrite or Varnish-based rewrite? (sumanah, 21:17:49)
- <brion> my recommendation is to do this either varnish-side or php-side with varnish-side cache splitting ... JS replacements are not going to play well with the browser pre-downloading things (sumanah, 21:21:10)
- LINK: http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007334.html (sumanah, 21:21:28)
- LINK: http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007333.html on gzipping (sumanah, 21:21:34)
- LINK: http://en.m.wikipedia.org/wiki/Cats ? (sumanah, 21:24:41)
- ACTION: dr0ptp4kt Yuri Astrakhan + Adam Baso: try the Cats page next week, then the week after that expand to a full language, then the week after that all languages. Do qualitative testing of image quality + measure bandwidth savings (sumanah, 21:28:24)
- AGREED: PHP-side modification for low-qual images on zero, splitting cache in Varnish. JS not suitable (brion, 21:30:39)
- IDEA: maybe someday we could also change original/regular images, not just thumbnails, if the user wants to save bandwidth there, too, but let's see how thumbs go first. - Adam (sumanah, 21:32:25)
- Yuri is comfortable with this action plan (sumanah, 21:35:09)
- ACTION: dr0ptp4kt to check in with Brandon Black and either have him as Ops liaison for this project or find someone else :-) (sumanah, 21:36:21)
Please see Architecture meetings/RFC review 2014-06-11 for the full log. Sharihareswara (WMF) (talk) 19:54, 12 June 2014 (UTC)