In the average case, it's generally not an issue, but in a game loop, i could see it being a source of issues.

It's really easy to make a mistake and accidentally make a copy of a reference, or obfuscate any conversions...which, I can imagine in the game engine world being quite a massive footgun.