Machine Learning

In general? Because deep RL is a skyscraper made of sticks and glue. Nothing, and I mean nothing, is actually guaranteed to work or has any kind of theoretical foundation at all. There are guarantees for toy problems, but everything past that is the wild West. In practice it's janky in a way no other field of ML is.

The standard way of learning value functions is to use the Temporal Difference update. Except we've known since the 90's that this doesn't really work -- sometimes the solutions diverge, and there's no known way of ensuring the neural net weights won't all go to infinity. In practice this means that frequently authors will do multiple runs, and only report the runs where the weights don't explode. Even if your weights don't explode, in general policy class are not expressive enough to learn the optimal max-entropy policy, and even if they are, the loss isn't convex. It's possible to learn the right value function and not be able to recover the optimal policy.

And even granted that those don't cause issues, you have to have an exploration strategy. Exploration is far and away the hardest problem in machine learning. You have to reason about the expected value of places where you don't have data to make estimates. And even when you do have estimates, none of your data is iid. It's basically impossible to do any kind of normal statistics to solve exploration. If you look into the literature on exploration and online learning, you'll instead find some incredibly unusual math, most frequently involving an algorithm called Mirror Descent that does gradient descent in non-Euclidean geometry. But even that's really only usable for toy problems right now. The only viable strategy for real problems is trial and error.