Topic on Talk:JADE/Implementations

I drew a picture of what I think our event flow should look like. See this article about managing event-based systems for reference.

JADE event flow. Events are generated by human actions. These events are the primary truth and go directly towards being stored and being applied to JADE's own state store. An "executor" turns events into actions and a "translator" converts events to a new format.

Missing from this is MediaWiki integration that will include the minting of RecentChanges rows and a minimal UI for suppression actions. The MediaWiki integration will likely also include the ability to submit judgments/endorsements.

I've been thinking about this. It's going to be hard to emit fully-formed events without direct interaction with the state store. The most difficult part is generating IDs for events that create something. I was thinking that we could get around this by having the JADE system emit a proto-event that corresponds to strict protocol of "everything but IDs" and that the system *only* uses these proto-events to update its own internal state.

JADE state validator. The process of generating events is expanded to include a flow of proto-events and responses from a database that then results in emitting fully-formed events to the event log.

I made this diagram to capture what I'm thinking. The cool thing about this is that we can have the proto-event actually correspond to a state update action that might fail for good reasons while updating state. If that happens we can then rollback state and not emit the event at all the the event log because it, in effect, didn't happen.

I just got talking to Aotto and Awight.

We identified a hole in this diagram where an event affects the state but doesn't get written to the log. We could move the event log into postgres (same as state) but then we wouldn't be absolutely sure an event made it out to the kafka bus. We talked about a few options.

1. Find a Change-data-capture system to connect postgres to kafka. Wrap writing in a transaction that would confirm that the event makes it to kafka if the transaction is committed. 2. Don't worry about it. Just expect that the event is written 99.9% of the time. 3. Send all events to Kafka and read valid state from Kafka. Expect that state is up to date 99.9% of the time while validating new events.

I like 1 because it could potentially get us all the things we want. I'm not sure how a CDC system can enforce the guarantees we want, but it *would* make me happy if it could.

I've been looking for a way out of the distributed transaction rabbithole, please take a look and see if this might be a solution:

The biggest sleight of hand is that `create_judgment` no longer requires a validity check on the page being judged, nor does it return an ID. The API sends an asynchronous event to create the judgment, and we simply return the <wiki>:Jade:<article> URL where we assume the judgment will eventually be created.