Talk:Reading/Web/Lazy loading of images on Japanese Wikipedia

This has been clarified.

@ABaso (WMF) is there a reason you did not sample from Before and Post groups randomly so end up with the same number of data points in each sample size prior to computing the 95th percentiles? Also, did you do a hypothesis test to see if the Before and Post differences are statistically significant?

The analysis we've done so far wasn't very scientific. The numbers reported are simply a count of events before and after roughly the time of the change. I used about 3 weeks before and 1 week after. This preliminary analysis was just to check if data was going down or up.

Following up on this, per our previous face to face, the NavigationTiming data were already sampled. In the updated analysis, however, as recommended, the Kolomogorov-Smirnov significance test was applied to these two groups and the null hypothesis rejected.

Special pages will not be impacted by this change and namespaces outside the main namespace will be less impacted. The current data analysis shows impact on our servers and paints a picture from an operations perspective but not from a reader's perspective.

It would be good to filter page views to only pages in the main namespace and compare bytes shipped before and after to get an accurate picture of how this impacts readers.

This has been clarified.

@ABaso (WMF) can you specify in section "Analysis of weeks worth of data" what exact dates Before and Post refer to?

@ABaso (WMF), why is this plot showing data from May? Shouldn't it show data only for the span of time starting one week before the rollout until one week after? If not, can you explain what the drop around May 25 is?

JR to update the graph to hide the drop as it is confusing and not related to this experiment.

I'll also work out what caused that drop because we should make sure we've documented that :)

Graph has now been updated. The drop on May 25 of fully load time on Japanese Wikipedia is a bit of a mystery to me. Looking at sprint Mand sprint N I can't see anything obvious but something clearly happened. Maybe @Ori Livneh (WMF) can remember some kind of backend change that may have gone live then?