Topic on Talk:JADE

I've been reviewing some of the pages of ORES' mistakes (e.g. it:Progetto:Patrolling/ORES and some of our review of anomalies in training data (e.g. Phab:T171497). I think we'll need a nice pattern by which multiple judgments can be submitted about a single wiki entity (revision, page, user, etc.), a conversation can take place and consensus recorded. In order to manage all of this in the JADE/Schema, we've split up the notion of a "judgement" from an "endorsement" and we've borrowed the concept of "preference" from Wikidata.

Let me illustrate with an example. Let's say that I want to judge the quality of en:Special:Diff/327645238. I'd create a new "judgement" with damaging=true and file an "endorsement" of the judgement with my comment "Removing cleanup tags without fixing the problems." Because (in this example) I was the first person to submit a judgement/endorsement, that judgement would be marked "preferred".

At a later date, Awight is reviewing some anomalies in our training set and comes across this judgement. He reviews the diff and comes to a different conclusion. So he creates a new "judgement" damaging=false and files an "endorsement" of that judgement with the comment "Good removal of cleanup tags". He's feeling pretty confident, so he moves the "preference" to his non-damaging judgement and starts a new discussion thread tied to this wiki entity titled "Cleanup was performed in past edits" where he points out that the cleanup had already occurred before the edit in question.

I get a notification that an entity I judged has a new discussion thread. After reviewing the edit again and seeing Awight's reasoning, I still disagree, but I don't feel strongly enough to dispute the new preferred judgement. I could change my endorsement and edit my comment to reflect Awight's, but I don't want to. So I leave my endorsement on damaging=true and leave the "preference" alone (so it's still pointing to damaging=false)

Does this make sense? Should it be possible to have two different "preferred" judgments or should we limit it to just one that captures consensus? Does it make sense to allow individuals to keep their endorsements where they like and set the preference bit separately? What do you think?

I'd like to see the judgment and comment more tightly coupled, for a few reasons:

• It's meaningful to distinguish between a comment left during judgment, a comment added later, and a comment left during judgment but edited later.
• We should be encouraging tool authors to always provide the free-form input field during judgment, because we think it's a best practice and results in more thoughtful and higher-quality labeling. The easiest way to do this is to make it a field on "create judgment".
• What is an endorsement without a judgment, if not a discussion thread? I'm interested in keeping discussion threads out of our model and keeping that in pure Flow-land.

Good question about multiple preferred rankings. I would follow Wikidata's lead, but their documentation is deliciously ambiguous about this. Looking at how their data is used in practice, https://en.wikipedia.org/wiki/Template:Wikidata#Configuration_flags, I'd guess most tools will query our data using an equivalent of "rank=best". If a preferred judgment exists, only return that. If several rank=normal judgments are present, return them all... What if we leave it to consensus to set zero, one, or multiple preferred judgments? We're already in a plural paradigm in which we're going to return a machine judgment along with a human judgment, might as well allow multiple human judgments e.g. in the case of "zero preferred judgments but multiple unranked human judgments".

"What is an endorsement without a judgment" such a thing should be impossible. I agree that discussions should stay in flow land. A comment with an endorsement serves another purpose. It supports an endorsement. It's a special kind of statement. In Wikipedia, !votes take the form where endorsements have comments that directly support the endorsement, but comments are made orthogonality. Given, discussion threads will spawn from Endorsements, but I believe this is an anti-pattern since these discussions get messy quickly and much work is done to re-summarize and recover.

In other venues, we decided to drop endorsements for now. Instead, free-form discussion can happen on the talk page. Disagreements about judgments can also follow the bold-revert-delete workflow, where the judgment is simply replaced by a new, proposed judgment.