Topic on Talk:Wikipedia.org updated page layout

I could not find any evidence that people were more likely to use the drop-down menu than the old list. I suppose that the links hidden in the new menu are called "secondary", is it correct? In this case the figures show that users were less likely to use the new layout:

• 12 (or 1.1%) clicked on a link in the current list
• 3 (or 0.2%) clicked on a link in the drop-down menu

This means that the test shows that the new layout is 4-5 times worse than the older one, as people were less likely to find the language they were looking for.

Another flaw is that we do not know what language the user was looking for. It would be much more meaningful to do the following test:

• pick users whose preferred language is not in the top-10 but whose accept language includes one of top-10
• check how many of them would pick their preferred language in the old list and how many will pick the accept language in the top-10
• same for the new drop-down menu.

As an example, many users from Ukraine would be looking for Ukrainian Wikipedia, but they would not mind reading Russian Wikipedia if there is no Ukrainian Wikipedia. Are they more likely to find Ukrainian Wikipedia in the new menu than in the old list or are they more likely not to find it and choose Russian Wikipedia which is more prominent?

The "top 10" ("primary') links were dynamic, using the user's preferred languages to populate the links around the globe, and filling the rest with the remaining of the default top 10. This was implemented as a result of a previous A/B test where we tested dynamic link population through language detection. So a user from Ukraine who has Ukrainian and Russian as preferences in their browser would see links to those respective Wikipedias as the first two links in the top 10, making the secondary links (which is what they were always called) unnecessary for their goal. I hope this addresses the confusion.

@MPopov (WMF): No, this does not address, as there is no confusion. I will try to explain the problem by presenting a use case.

Let us think of a user who is looking for a Wikipedia which is not in the top 10. This may happen for a variety of reasons:

• User does not know how to set preferred languages. From my experience this is the main reason: experienced computer users who know how to set preferred languages will most likely find the wiki directly, without using wikipedia.org. In this case the user will have default languages, e.g. a user from Ukraine may have English and/or Russian by default.
• User uses a public computer (e.g. school, library or internet café) and cannot change preferred languages
• User is explicitly looking for a Wikipedia not in their preferred language, e.g. a user from Ukraine is looking for Belorussian Wikipedia

Now, we want to know what will this user do. There are three options:

1. User will find the wiki they are looking for, either in the new drop-down menu or in the standard list (in our case a user from Ukraine finds Ukrainian Wikipedia)
2. User will not find that wiki but will choose one from the top-10 (e.g. a user from Ukraine goes to Russian Wikipedia which is more prominent)
3. User will leave the page without clicking anywhere.

What I want to know is whether case 1 is more likely or less likely with the new layout. Unfortunately we cannot detect users whose preferred language is not set correctly, thus we can make a test on users who have their preferred language set to a language other than one in top-10. We can do the following test to check this:

• Both Control and Test users get default (not dynamic) top-10.
• Control users get the old list of Wikipedias
• Test users get the new drop-down menu

We can split users into two subgroups:

• Users whose most preferred language is in the top-10. For these users we want to know the % of users who click on a Wikipedia not in their most preferred language (i.e. they were explicitly looking for it); we can suppose the % of such users is the same in both subgroups.
• Users whose most preferred language is in not the top-10. For these users we want to know the % of users who would click on their most preferred Wikipedia with the old or the new layout.

So far the figures are not satisfactory (by the way, it is very difficult to find the right figures in the report, for instance, I could not understand what 1.1% in "12 (1.1%)" refers to, as I could not find any figure in the range between 1043 and 1143 in the report):

• Engaged with page. 60% (1802 out of 3019, Test) vs. 56% (1444 out of 2560, Control) is 99% statistically significant, thus indeed users were more likely to engage
• Used secondary link. 0.1% (3 out of 3019, Test) vs. 0.5% (12 out of 2560, Control) is also 99% statistically significant, which means that users were much less likely to use secondary links.

This can mean that users who were looking for a Wikipedia among secondary links were 4.7 times less likely to find it. This would be a significant problem for Wikipedias not in top-10: they already receive less traffic by not being in the top-10, and they would receive even less traffic if users would be unable to find them (and this will more severely affect minority languages). However, this can also mean that users explicitly looking for a Wikipedia among secondary links were just underrepresented among the Test group. That's why I think that a different test is needed to measure this.

The point about the user not setting the preferred language is an especially hard issue for us. A thing that really helps in this regard is that the browser sets the language automatically based on the language the operating system has set. In my personal experience with Russian users (particularly the elderly who are new to computers), the OS has been in Russian. I guess what I'm trying to say is that inexperienced computer users may not know how to manually change the settings of the browser, but the browser installers have also been made to account for this.

I apologize for the lack of clarity in the report, and going forward I will try to be better at referencing figures when I reference the numbers within them.

Thank you for your input, @NickK. We'll consider your suggestions.

@MPopov (WMF): One of Ukrainian users made an experiment in a rather small Ukrainian-speaking village, and he found out that most computers had their OS and browser in Russian, much less frequently in English. Now, these users will not see Ukrainian Wikipedia neither in the top-10 nor in the list, thus they may think that Ukrainian Wikipedia does not exist. In addition, users with the same problem may also not find Ukrainian Wikipedia in the interwiki list due to compact interwikis and their wrong language settings, effectively thinking that there is no Wikipedia in Ukrainian.

This is just an example, but the problem is much wider: we have minorities like Catalan speakers in Spain or Welsh speakers in the UK whose default language is usually wrong (Spanish and English respectively), most of India who have English as a default language, most of Sub-Saharan Africa who have English or French as a default language etc.

So far I find a very disturbing statistics that users are more than 4 times less likely to select these wikis (i.e. a Ukrainian speaking with Russian as a default language is 4 times less likely to find Ukrainian Wikipedia with a new layout than with an old one), which is in my opinion something we cannot proceed with.

Thus I would like to ask you to measure impact on these users before implementing these changes: we are not that in a hurry but we have a risk of seriously harming a lot of users. Thank you

Hi,

Screen Shot displaying of preferred browser lang and new lang dropdown expanded

Screen Shot displaying setting browser preferred settings for languages

Please take a look at this link with your own browser settings to see how various browsers will always display the user's preferred browser language settings in the top 10 links displayed around the globe. This recent improvement has gone a long way to solve the issue (that has always existed) of the user having to search for their preferred language wiki link in the very long list that is currently still displayed on the Wikipedia.org portal page.

I've included a links and images of my own browser set to Russian, Ukranian, Esperanto and English as well as how it would appear on the wikipedia.org page with those settings.

Also, please realize that the statistics that we talk about in our research may seem small (for example, research might find that 1% of users are more likely to click into the search box or a particular set of links) but compared to the average page views on the portal, it's actually a quite large number of users that are finding their links / information faster and easier.

Let's do the math for a moment. For instance, on June 1, 2016, we recorded an average of just under 14 Million pageviews to the wikipedia.org site. If we take 1% of that 14 Million visits, that means that in one day, 140,000 users were more likely to take an action on the page. We feel that those numbers are completely statistically important and positive.

I realize that this new language link drop down (that is the subject of this particular talk page) won't fix everything for everyone but it's a step in the right direction to help out many people that use the site to find information.

Hi @DTankersley (WMF):

I am doing another math.

• 14 Million pageviews are going to wikipedia.org (source: your figures).
• 0.3% of users of Wikipedia visit Ukrainian Wikipedia (source)
• As a result, 42 thousand pageviews from wikipedia.org are likely to come to Ukrainian Wikipedia.
• Many Ukrainian users do not know how to set preferred languages and have only default language (Russian and/or English), and not Ukrainian (software in Ukrainian is not that widespread). According to statistics, only 21.3% of Ukrainian users have Ukrainian is a primary language in their browser.
• As a result, 33.1 thousand pageviews are coming from Ukrainian users who do not have Ukrainian as a preferred language. For these users, Ukrainian will be in a secondary language and will now be hidden in a drop-down menu.
• With the new layout users are 4.72 times less likely to choose a secondary language (source + my comment above), and this figure is statistically significant.
• As a result, instead of 42 thousand pageviews only 15.9 thousand will come to Ukrainian Wikipedia. Other users will most likely choose another Wikipedia, such as English or Russian.

This will represent -26.1 thousand pageviews per day for Ukrainian Wikipedia. Or -0.8M pageviews per month. This is a lot, and it is statistically significant. While it might be a move in the right direction for some, this will definitely be a move in the wrong direction for Ukrainian Wikipedia.

As I have mentioned above, this concerns not only Ukrainian Wikipedia but also other languages in which software is not that widely available, like Catalan or Hindi, for example.

Your theoretical potential (42 thousand as the maximum potential visitors from the portal to the UK wiki) sets an artificial limit, decreasing any percentages derived from that potential. You're statistics for primary language being set to Ukrainian also depends on visitors to that particular web page (refreshing the page a few times increased the counter.). That makes those statistics less reliable.

I understand your valid concerns. That users who do not have their preferred language set in their browser will have to find their language in a dropdown menu as opposed to a list of languages by size.

Language detection work (for the primary search box on the portal and elsewhere across Wikimedia), a potential update to the design of the dropdown, and a discussion on sorting are all things that may improve the interface - and potentially outright negate the need for an interface - for users whose language we are unable to detect.

The portal team measured the impact of this change in an A/B test. The results were an incremental improvement to visitor action. We have a task to investigate tracking portal to wiki traffic, but are do not have it in our plans to investigate this quarter. We'd much rather implement this incremental change now than to wait for an unknown point in the future.

I could have provided more figures for Ukrainian (for instance, this study gave similar results), but this is not just about Ukrainian. This is about all countries where most software is not in national languages (actually most of Global South), this is about languages not supported by any browser (and there are over 100 of them, including, say, Cebuano, which despite having 3rd biggest Wikipedia is not supported by IE, Firefox or Chrome).

What is disappointing for me is that we have two proven facts.

1. With this change users are 1.07 times more likely to use primary links or search. This is good news for large wikis already in top-10 that will get a bit more additional visitors.
2. With this change users are 4.7 times less likely to use secondary links. This is very bad news for small and medium-sized wikis not in top-10 as they are very likely to lose visitors.

Still, getting some 140K page views per day seems to become more important than multilingualism. Wikipedia was always proud to have more languages than any other leading website, now these languages are hidden and we rely on browsers that are much less multilingual.

I agree that tidying up the main page is a good idea, but is it possible to come up with a solution where people are at least as likely to use secondary links? We need a win-win solution, now we have only a win-lose, unfortunately.

I'm not sure that there is a way to teach everyone how to change their browser settings, unfortunately.

Please remember that we're not taking away the secondary language links, they will simply be in a dropdown format.

Yes, you are not taking them away, but your study shows that users are 4.7 times less likely to use them. It is a lot, and it might mean much less pageviews for Wikipedias outside top-10.

Wikipedia is available in 291 languages, while browsers do not offer that much: Firefox offers to choose among ~190 languages, IE and Chrome have just 120. For instance, Crimean Tatar (crh) is not available in any of these three browsers, thus all Crimean Tatar speakers (even if they make their best effort to change their browser settings) will have to use the dropdown menu. Multilingualism has always been an advantage of Wikipedia, and it should not be hidden from readers.

I completely agree with @NickK. One may think of the decrease in view not essential, but for small wikis, struggling for their readers, it is significant.

Hi @Ата and @NickK,

Wikipedia.org portal more languages button

Do you think this particular treatment would be better - with the 'more languages' dropdown directly underneath the globe and in the same area as the top ten links?

We don't want to make it more difficult for anyone to find their preferred language wiki, especially if it's smaller ones like the Crimean wiki or the Cebuano wiki.

@DTankersley (WMF): I don't know. I have suggested a way to test this above in this thread. You can even try an A/B/C test (with "A" for current layout, "B" with "more languages" as a button and "C" with more languages as a globe) to find out whether it will have an effect or not. Personally I do not have a necessary level of UI expertise to confirm if it will have any impact.

I think @NickK lays a strong argument for location-based language detection on the portal.

This idea has been suggested before, but not with the breadth of languages mentioned above (Ukrainian, Catalan, Welsh, Crimean Tatar etc). This 'minority language' use-case, where users browse the web in a more popular language than their own, might be quite high. Off the top of my head, I can image that Irish, Scottish & Welsh user might just click on English Wikipedia and Swiss German users might just click on German wikipedia.

Surfacing smaller languages was the original goal of localizing the top-ten links, and further supplementing this effort with location-based languages might help smaller wikis gain more readers.

There is a precedence for location-based language detection as well. The ULS on translatewiki.net offers 'suggested languages' based on location. Because I'm in Poland, I see languages like Lithuanian, Ukrainian & Belarusian, even though my browser is set to just english:

Indeed, can we not simply ensure that a Ukrainian IP would be guaranteed to see the Ukrainian Wikipedia prominently, no matter what their OS/browser settings?

From my own travels, I can confirm that many users, including experienced editors(!), use computers or devices without setting the preferences to the language they actually contribute in. And this is the case, as User:NickK pointed out, not just in Ukraine, but in other eastern Europe countries, in India, and in southeast Asia.

It might be easy for Ukraine, as Ukraine has only one national language. Unfortunately it will not work in, say, South Africa which has 11 languages, 10 of which have Wikipedias, effectively leaving no space for anything else beyond national languages. This might be very disappointing for, say, a German speaker living there: they might have set German as their first language and German is in top-10 by any approach, but they will fail to see it.

Thus can we either expand the circle beyond 10 languages or add something like "languages popular in your area" as a separate line?

Hi @NickK and @Ijon, thanks again for your feedback. We probably won't expand the listing of languages (by browser setting or by top viewed) simply because there really isn't that much space around the globe to display more than 10 language links. However, we can have some sort of widget or link or line of regional languages in addition to the language links around the globe. This would enable us to detect what the visitor's preferred language is -and- be able to provide them with language links that are typical to where they physically are in the world. Hopefully it would be the best of both worlds!

We haven't fully vetted / flushed out the idea of using the translatewiki type of detecting which country your browser is accessing the internet in and then displaying the top X amount of languages that are used in that country/region. I say this because even though we've talked about it a lot and created some rough mocks, we haven't done any A/B testing on how well it would work and if visitors would like it.

This will be in our future work - to take a closer look and to run some tests on the interaction and we hope that you'll be able to provide us guidance at that time!

Thanks, @DTankersley (WMF). Could you offer something more concrete than "future work" and "at that time"? When would the team do that A/B testing, to introduce more data into this decision-making?

Hi @Ijon - I don't have anything more concrete that 'future work' at this time, unfortunately.

We are taking a bit of a hiatus from working on the portal to focus on other priorities within the Discovery department. We are planning to do maintenance to the wikipedia.org portal page such as stats updates and additional of translations as well as bug fixes for the next couple of quarters, but no larger scale work is planned to be done right now. Please view the page here that describes the work we've done so far and the work we want to continue to do in the future, mostly likely restarting in March 2017.

I don't think that March 2017 is really a good date for this. Wikipedias in smaller languages were very visible, now they are hardly visible (unless user has already pre-selected these languages: A/B test shows that users are much less likely to use the drop-down menu). This has negative impact on smaller Wikipedias, in particular in Global South, which are less likely to get visitors. Having this last for 7 months is, in my opinon, a really bad idea.

@DTankersley (WMF): could you clarify who made this decision at WMF? It is not immediately obvious that this should be a decision made by the engineers implementing it.

And it seems to me that @NickK brought up some concrete arguments (and numbers) arguing the new state is decidedly inferior to previous status quo, from the small and medium Wikipedia point of view, and that these argument have not yet been refuted, and yet there does not seem to be any change planned by the team.

@Ijon If you mean the decision to halt new work on wikipedia.org for a few quarters, that was Tomasz Finc and myself, and later Katie Horn. This was announced on the Discovery mailing list. In order to deliver on the annual plan, it's necessary for us to halt some work in order to pick up new work such as improving the search page. If you have a question about that, I'd be happy to talk to you about it, but we shouldn't do that on this page as it's specifically about the portal; feel free to email me, reply on that mailing list thread, or contact me elsewhere on-wiki.

Thank you, @Deskana (WMF), for this clear response. That is almost the decision I meant, although I was specifically asking who made the decision (if an explicit decision was made) to continue to consider this feature Y Done and better than the previous status of the page, in the face of the concerns raised by @NickK. I do understand that the team's original plan dictates moving on, having spent the time originally allocated for this feature's implementation; but I think there is reason to consider the feature flawed, and therefore to at least consider altering timelines.

As I wrote in the second paragraph of my previous comment, I don't see that NickK's arguments have been fully engaged with, and considering that he makes a (to my mind) convincing argument that the new status is worse than the status it replaces, it seems to me to warrant a(n additional) PM/manager decision, that would result in one of the following: 1. NickK is wrong, and here's why.; 2. NickK is right, but here's what we can do to mitigate the damage to smaller languages (e.g. use geo-location to suggest languages, as suggested upthread); 3. NickK is right, and we'll roll back this new layout and go back to the drawing board to find a design that does not cause this collateral damage to smaller languages; or conceivably 4. NickK is right, but due to This Even More Crucial Thing, we will neither roll it back nor fix it until March 2017.

What I would like to hear is that that further decision has indeed been made, and that it gave due consideration to these concerns.

(the bold here is to ease reading only, and should not be read to convey emotion :))

@Ijon Fair questions. The short answer is that it is inconclusive whether @NickK is correct or not that a decrease in page views to the Ukrainian Wikipedia is observed as a result of the changes. Discovery Analysis will do some further analysis to make a proper determination. If it is determined that there is a decrease in page views, we can consider reverting the changes to wikipedia.org.

Now, the longer answer. Regarding the quantitative analysis, NickK's reasoning regarding decreasing page views to the Ukrainian Wikipedia sounds compelling. However, it is not possible to reason about nonlinear systems in the manner he did, especially if one is combining statistics and data points from different sources and contexts. Experimental controls generally need to be identical in order for the data to be cross-comparable. We'd need to do a more scientific analysis to know if the change we made to wikipedia.org has negatively impacted page views to the Ukrainian Wikipedia (or other projects). Our previous statistical analysis during the A/B test suggests it is unlikely that this is the case, and the page views dashboard does not seem to show a decrease, but we do not know definitively. I've filed T143853 for us to perform that analysis. If the analysis shows that there was a statistically significant decrease in page views to the Ukrainian Wikipedia, we can consider potential solutions to that problem, one of which is reversion of the changes in question to wikipedia.org.

Hopefully this answers your questions! If not, please let me know, and I'd be happy to clarify.

Thank you, @Deskana (WMF):, this answers my question.

I agree that more scientific analysis is indeed needed. I suggested a way to do A/B tests, now that this new version is live one can study clickthrough rate from the page:

• One can compare % of users who clicked on wikis on the large list to % of users who clicked on wikis from the drop-down menu.
• One can take as an example wikis that cannot be set as the main language (not supported by any browser), e.g. Cebuano or Crimean Tatar.

I am looking forward to seeing more detailed and more scientific analysis.

Hi @NickK,

Based on our conversation earlier in the year, we created a new dashboard page that can be used to monitor the usage (clickthroughs) to any of the individual language wikipedia's from the wikipedia.org portal page.

You can get to the new page by clicking here and then you'll need to click on 'languages' and then 'languages visited' to see the data I'm referring to. (Note: we just released this new page into production this morning and the phabricator ticket for this work is here.)

Once you're on the new languages visited page, you can select a particular sorting mechanism from the 'sort languages' dropdown. For instance, in the dashboard image attached to this comment, I used Tatar, Catalan, Croatian, Cebuano, Ukrainian and Turkish.

Portal-dashboard-language clicks may-aug2016

Pageviews-on-lang-portals jul2015-aug2016

By hovering with your mouse, you can view any date that we have collected in the dashboard data to view how many clicks occured on those particular language linked wiki's from the portal or you can also view how many users went to those language portals from the wikipedia.org portal page. Please be aware that these are numbers gained from our eventlogging schema and aren't the actual total clicks (or users) but are a representative view of the actions taken on the portal.

We created this new page specifically to address the concerns that you and others have had about the new portal page layout with respect to adding the languages into a dropdown (other than the browser preferred language and the top ten viewed languages). We wanted the community to know that we do take the portal page layout change very seriously and we've created tools in which to monitor any drops in usage or any other weirdness.

The other image attached to this comment is from this page that shows the amount of pageviews to a particular language wikipedia - you can see that pageviews to specific language portals are fairly seasonal (I used the same languages as my other screenshot).

We plan on watching the community's actions via these two dashboards and - as @Deskana (WMF) already pointed out - we have an analysis task to review the clicks to various languages from the wikipedia.org portal and if they changed after the new layout was pushed into production.

@DTankersley (WMF): Excellent statistics, thanks! I will look into it.

Thank you, @Deskana (WMF). This is the sort of answer (and action) I was hoping for. I look forward to the results of that further analysis.

@Ijon I'm glad it was helpful. We'll update T143853 when we have more info.