Topic on Talk:JADE/Flow

@Halfak (WMF) and I identified a design assumption in IRC last week, which has been implicit in a few of the other threads here, especially in "Judgments, Endorsements, and Preference" below, but is worth surfacing as its own issue. I'd like to open this for more comments.

In the current design of JADE, there can only be one "correct" judgment per wiki entity. Most entities will simply have one editor making one judgment, but in the cases that multiple editors supply a judgment, the second, third, etc. editors will be overwriting each other's judgments as the "correct" answer. This follows the BOLD,_revert,_discuss_cycle on English Wikipedia. @Halfak (WMF) has pointed out that this is the best process for the problem domain, that we want to encourage consensus.

An alternative workflow would be something like the Wikidata model, in which multiple judgments on a wiki entity default to having equal "rank", and it would be optional to go through a consensus process. Any editor could mark one or more of the judgments as "preferred", with or without discussion, or the editors could leave all judgments as "normal" rank and we would consider the set of judgments to be unresolved and equally valid.

I think there are two questions here: what is the preferable workflow for the various use cases in which editors add JADE judgments, and what are the best requirements on the data in order to support algorithms and reasoning using the judgments?

Actually “…this is the best process for the problem domain, that we want to encourage consensus.” is wrong. It focus on the strongest contender of the content, or in other word, “the single user that gives a rat ass about the consensus” to quote an user at nowiki. I believe it is important to say that this approach does NOT gives the consensus, it is the meaning of the user that does not give in and continue fighting. I'm not sure that creates a viable solution, and it definitely does not create a welcoming and including environment.

I think this discussion of the "Wikidata" model misses that it has BRD as well. In this case, "rank" is part of the data model itself. There's still one true representation that is identified via a consensus process. But with "rank" a more accurate reflection of a messy reality is represented in the data model itself. If we wanted to represent messiness like this, it might be better captured in JADE's schema itself -- depending on the type of judgement.

E.g. if a user decides that one judgement should be ranked "preferred" while another should be ranked "normal", they are essentially marking one as best and another as "safe to ignore". You might show up and disagree with how they have ranked and revert them. Preferably, you'd then discuss your differences in ranking.

I think when it comes down to it, I want to know why having ranks makes any sense. What information does it capture that we need to capture here? In what case, do we want one judgement to be "preferred" when another is simply "normal"?

BTW, I'm Halfak (WMF) accidentally made this edit through my volunteer account.

Is there a compelling user story that would have a ranking/preference system? How likely is this to be encountered in the wild?

I think the simple BRD approach is fine, with the one "true" judgement that can be revised further by others. If I could think of an imperfect analogy, it'd be kind of like File History on Commons. The latest upload is always displayed as the canon version, but previous revisions are still visible/available. Does that make sense?

Good Q. Here's the thread where I originally brought up the proposal for "preference": Tzw0uv2bucrdprm4 It discusses a use-case.

I think that generally, we should allow people to disagree with consensus -- much like what happens in a straw poll.

I can buy this argument. I was thinking it would be more simple to a user to "make a judgment, ignore what else is there as you wish" rather than "make a judgment, yours will be the new authoritative judgment and will overrule the others". But I know I haven't had my fill of wiki Kool-Aid yet ;-)

In that case, I still think limiting it to one preferred judgement is what we'd want to go for. People are welcome to discuss it, but splitting it off into rankings and separate judgements seems unnecessary IMHO.

There are several articles that discuss rating, and points out the need for a cost function. One article is Jøsang, Audun, Roslan Ismail, and Colin Boyd. “A Survey of Trust and Reputation Systems for Online Service Provision.” Decision Support Systems 43, no. 2 (March 2007): 618–44. https://doi.org/10.1016/j.dss.2005.05.019. Note that the article is mostly about trust and reputation given by open rating system, but it is very similar to trust and reputation for rating articles.

The conclusion is pretty simple; if the cost goes towards zero, then the system becomes easy to game, and then users will try to game the system. To avoid that you need some kind of small cost, typically that you have done something positively. For example that you must contribute (non-reverted) content to be able to rate (non-reverted) content, still you may be able to rate according to prepared reasons as they are hard to use for vandalism.