Lexora

The problem

A product where one user action has to trigger several downstream effects, and none of them are allowed to quietly not happen. The naive version, doing the side effects inline with the request, loses work every time something fails halfway through.

The system

A Rust backend built around an event-driven core. A state change and the events it produces are committed together through a transactional outbox, and background workers pick those events up and do the slow work outside the request. A React front end sits on top, and the whole thing runs on Cloud Run.

My role

Backend architecture and implementation of the event-driven core, the outbox and worker design, and the production operations story around it.

The hard part

Delivery guarantees. Between writing to the database and telling the rest of the system about it there is a window where a crash loses the event. Closing that window without reaching for a distributed transaction is essentially the whole design.

Architecture decisions

1. Transactional outbox instead of dual writes. The event is committed in the same transaction as the state change, so there is never a moment where one exists without the other.
2. At-least-once delivery with idempotent consumers. Duplicates are much cheaper to handle than losses, so workers are written to tolerate seeing the same event twice.
3. Workers off the request path. Slow or failing downstream work shows up as degraded throughput to catch up on, not as an error returned to the person who clicked the button.

Where it stands

In production. Red Sentra continues to support and extend it.