Talk:Wikipedia.org Portal A/B testing

I like minimalism and making things cleaner, so I like the overall idea, as I said to Daisy, it will make the page smaller and hide information that we don't use all the time.

I suggest that the modal when opened should "push" all the down portion of the page instead of floating above the contet (wich will also make it hidden), and I also think it would be better if the height of the modal wont be fixed (hidding the overflow), so the best away for me shoud be that when we click on the button:

All te content appears, push the rest of the page downwards as far as it needs (don't creating another scroll bar inside it). And don't auto-close when we click outside or (worse) simply take the cursor out of the area (sites that do this like facebook tends to irritate XD)

: )

Hi Ogat, thanks for your comment! We actually did that, to push down the data on the page when the language dropdown is expanded on a mobile device.

We'll take a look at the desktop version of the drop down, which is currently in a modal with a scroll, to see if we can do this in a better way.

It seems like putting the label for the dropdown hiding the other languages in English is not going to be maximally helpful for speakers of languages other than English. Customizing it with a translation in the user's browser accept language, or just using an icon (if a suitable one can be devised—a few ideas exist on Commons) would be more universal

Thanks for the suggestion, Trey, I'll discuss with @MSyed_(WMF)!

A way to solve that would be making the OS/Browser prefered language always appear with the main languages around the globe, even if it has less than 1mi articles!

Yup, we did that exact thing in our A/B test and then pushed it to production. :)

It combines several tests so it would be nice to explicitly specify what do we measure there and compared to what (e.g. baseline as it is now or some implementation of one of the previous tests by then may be in production?).

Thanks - I've updated the text for this test!

The current "why" is " The Portal Team is curious as to how many of the visitors to the Portal page do not have English set as their preferred browser language." That doesn't seem right to me, since we can already answer it based on logging. I suspect the real "why" is more like "The portal team would like to know if users are more likely to click on a language link if languages relevant to them are in a more prominent position."

Agree, the rationale and test descriptions do not match. Ideally, we need to describe what decision would the test influence (i.e. "do we want to arrange languages differently depending on user's language?") and then say how we decide if to take that decision or not (e.g. "if we see that rearranging improves clickthrough rates then we conclude rearranging is good thing and we'll add it to portal").

We'd like to know if re-arranging the links around the globe increases click-through for the top 10 links. The idea is that if the user does not have English as their preferred browser language, we'll put their preferred language into a more visible area for them to click on (if they can't read English). The thought that we'd really like to prove is that if the browser lists a language that is not part of our top 10 language links - that user will click through more frequently to their preferred language and not have to use the language by article links to get to a page that they can read easily.

I've updated this test's WHY section.

The second half looks good now (yay!), but I still don't like/understand the first half as it sits. How about "The Portal Team is curious whether Portal visitors who do not have English set as their preferred browser language would be more likely to take an action (click through to a wiki site) if their preferred language link is in a more prominent and visible position." ?

I think that's what it says now in the first half? I might have updated it after you posted this comment? :(

The text still says the test will tell us "how many" users don't have English, but that's information we already have access to. My proposed rewording is similar at a glance, but different.

Updated now, thanks!

Detecting the setting of the browser of the visitor is not enough. I think we can do better. Look at https://translate.google.com/, they are detecting the language with the text which was typed.

Hello, we actually have another ticket for this type of work: https://phabricator.wikimedia.org/T129627 To be able to tell what language the user is typing in and to "automagically" switch to search in that language. Stay tuned!

I notice that in the images for tests 2, 4, and 5, the blue/red/green Wikimedia logo is shown in all blue. Why is this? I would think it would hurt branding. Is this even allowed, according to the identity guidelines?

Thanks, Yair - we'll update the mocks. It's a good catch, but mocks are just that - an idea of what we want to do but not necessarily the final product. :)

Thanks Yair for bringing this up.

I confirmed with Heather Walls, our Creative Principal in the Communications team who leads all our branding efforts and she is happy with the one-color WMF logo as long as it is black or grey. Infact, the foundation has been using the one-color WMF logo in places like the Transparency Report (https://transparency.wikimedia.org/) and the Annual Report (https://annual.wikimedia.org/2014/ and https://15.wikipedia.org/) for a while now.

I have updated the mocks and changed the blue color to grey. @DTankersley (WMF) will update this page soon with the updated mocks.

If I understand test 3, it actually recycles the "languages inside dropdown" feature that was in test 2. If so, it should say that explicitly, and should probably also say that that aspect would be subject to the results of the second test.

Actually we won't do that in test 3, we will revert back to the languages by article count display as it currently is on the portal page. Test 4 will be a combination of test 2 and 3. I've updated the wording with this information.

Ok. I confused the new dropdown at the right edge with the earlier proposed new dropdown for the list of languages by article count. Both are a list of languages/projects, but in a different context and with different presentation. Probably fine now. Thanks.

I think this is the great description. I've listed a number of notes for improvement but in general I think it would be great to have this. Also maybe link to it from https://meta.wikimedia.org/wiki/Discovery/Testing?

Done, thanks!

For the second test (collapsing the long list of languages inside a dropdown), the current "why" is "The Portal team wants to find out if the listing of languages by article count could be displayed in a more streamlined way." I think the "why" should end with something like "...without significantly increasing the bounce rate."

My thinking is: Of course things can be streamlined. But can they be streamlined without negative consequences.

For this one, it is also unclear to me what we are going to measure. It is specifically not clickthrough as it says "this test is not meant to affect the click-through rate" so what do we intend to affect?

The test might be to ensure that it does not affect the click-through rate. I'm not sure it's reasonable to expect absolutely no drop-off, but if we saw 10% higher bounce rate, I would personally consider the test a failure.

Sounds good, but it would be nice to state so upfront. In general, it is a good practice to write down test criteria and what we'd do if we find it one way or another before the test.

Thanks for the comments - this test was to de-clutter the page a bit, without affecting the clickthrough rate in a negative manner. I'll update the information for this test.