Running them overspec is obviously going to generate more heat; but as far as why the 23s are within reason vs the 17’s not…
You are likely dumping the same amount of power into a much smaller volume and a lot less surface area to dissipate heat. Is there a good reason you are running them overspeced; do they not produce enough torque?
You said you ran 23’s in the same robot, but the robot seems to be form-fitting do your nema17’s, does that mean they ran outside of the containers and stuck out the sides? If so that would provide more area to cool. Even if not, it still has more surface area to cool. Each side has ~54% of the surface area on the nema 17, assuming the same depth of motor.
If you absolutely need to run these in this configuration and can’t fit aircooling, you might want to consider coolant lines; though I imagine that increases the complexity of your model significantly and increases the BOM.
Really all I can contribute based on your statements is that you have FAR less surface area for similar power drain.