It's about who controls it.

Because if one side controls it and does something with it they're not legally entitled to do do, they still get away with it if it's not worth it for the other side, even being in the right, to take it to Court.

So if you have in your hands the files for the installer for a game which doesn't do phone-home validations and you've actually explicity rented it for a limited time yet carried on using it past that point, the other side would have to take you to Court to stop it, and in the absence of a streamlined and very cheap Judicial process to do so for that kind of thing (like there is for things like loans) it's generally not worth it for them because it costs more than they gain.

Similarly, if they control if and when you can install and run a game (which is possible even if you have the physical media - all it takes is for the game to have phone-home DRM that checks if you're authorized) and they stop you from using it, even if you're in the right with zero doubts (so, there was no clear upfront information that you were paying for a unilaterally revokable license and you live in a jurisdiction where Justice isn't a joke, like Germany, so the unilaterally imposition of further contractual conditions after the sale - i.e. EULAs - aren't at all valid), it's generally not worth it to take it to Court to force the other side to restore you access to the game or compensate you (a few euros) for it because it costs more than you get back from it.

Holding physical media is highly correlated with controlling it, but (as per my example above) in a World of always on Internet access were phone-home DRM is easy to have, it's no guarantee at all of actually controlling it - you might have the bytes in the stablest imaginable storage medium in your hands and you still don't control it because you can't actually access the game without external authorization - and that in practice means that you can get shafted even though you're entirely in the right as long as the monetary amount you being shafted out of isn't too large.

Converselly, whilst digital distribution is highly correlated with not controlling it, as stores like GOG show it's perfectly possibly to sell games digitally in such a way that you get control of it (if you want - you have to actually download GOGs offline installers) by which point the cost of trying to control what you do with it is on the other side and the same monetary logic applies - it's not worth it for them as long as the monetary amount they would gain from doing it isn't too large.

Curiously, this also means that for PC games GOG can actually be a better way of getting de facto ownership of a game copy than physical media (if you make sure you download the offline installer) because GOG enforces the rule that their games have no DRM, whilst plenty of PC games in physical media have phone-home DRM.