Discovery/Status updates/2018-11-12

This is the weekly update for the week starting 2018-11-12

Discussions

David and Trey have resolved the problems with 32-bit Chinese characters (like 𨨏—[1]), which were returning irrelevant results, and showing lots of unicode replacement characters (�) in the results. The highlighter fix was deployed [2] first so there aren't any more � characters in the results. The re-indexing [3] to improve the relevance of results is now also done for Chinese-language wikis.

Letters are encoded internally by computers as numbers—for example, “A” is 65 and “a” is 97.[4] Years ago, programs and even websites would use different encodings[5] to represent text, often leading to unreadable gibberish on screen. Unicode[6] was intended to be a single encoding for most of the world’s writing systems. The most-used parts of it fit into a 16-bit representation,[7] which can handle about 65 thousand characters. But that's not enough for the very large number of rare and historical Chinese, Japanese, and Korean (CJK) characters, which are represented in 16-bit Unicode using “surrogate pairs”.[8] 1,024 Unicode characters are set aside to be “high surrogates”—the first half of a 32-bit character—and 1,024 characters are set aside to be “low surrogates”—the second half. By themselves, the surrogates aren’t valid and don’t represent anything, but in pairs they can represent over a million additional characters. Since these characters are usually rare, software can sometimes treat them incorrectly split them up, which can result in you seeing the Unicode replacement character �,[9] which is used when something has gone wrong processing Unicode text. (When the character is fine, but you don’t have a font to show it, you sometimes get little squares instead. Since the most common source of these squares for English speakers is unrepresented CJK characters, a slang term for the squares is “tofu”.[10])

--